---
name: video-production
description: >
  A community guide to making videos with open-source tools and Opper
  media: promos, explainers, brand films, motion graphics, animated shorts
  and social clips. Covers the method (brief, cue file, audio
  first), how to get every medium out of Opper (images, video clips, voice,
  music, sound effects, transcription, and vision / listening models as
  reviewers), assembling it with open-source tools (a code renderer and
  FFmpeg), mixing, and a measured review loop. Use it whenever the user wants
  to produce a video, animate stills, score a clip, add narration, sound or
  captions, or asks how to make video "with Opper", even if they only say
  "make a video".
category: guide
metadata:
  version: "2026-10-06"
---

> **A guide, not an Opper API skill.** It shares how we make videos with open-source tools (HyperFrames, GSAP, FFmpeg) and Opper for the media; the method and tool choices are suggestions. For Opper itself, start at [`opper`](https://skills.opper.ai/); request fields live in [`opper-multimodal`](https://skills.opper.ai/opper-multimodal/SKILL.md) and `https://api.opper.ai/v3/openapi.yaml`.

# Video production with open-source tools and Opper

**Generate what must be organic, code what must be exact, measure before you call it done.** Opper gives you every medium and the review senses through one key. Open-source tools on your machine do the rest: the picture is an HTML page animated with GSAP and rendered frame by frame by HyperFrames; FFmpeg trims, mixes, masters, muxes and measures.

## How the work is done

A film here is three layers, each with its own tool:

1. **Media from Opper.** Stills, video clips, voice, music, sound effects and transcripts are generated by small scripts that call Opper and save files. Nothing is placed or timed yet.
2. **The picture is a web page.** Layout, type, logos, UI, diagrams and every animation are HTML, CSS and SVG, moved by one GSAP timeline. Generated clips and stills sit on the page as ordinary `<video>` and `<img>` elements. HyperFrames turns the page into video by seeking the timeline to each frame and capturing it.
3. **Sound and delivery are FFmpeg.** It prepares the clips, mixes and masters the audio, joins picture and sound into the final file, and measures the result.

One cue file (`cues.json`) ties the layers together: the page and the mix both read their times from it, so a retime is one edit. One command per film runs the chain (prepare, mix, render, mux, check) and writes a new, numbered export.

### What the tools can do

| Tool | Capabilities | Limits and habits |
|---|---|---|
| **[HyperFrames](https://github.com/heygen-com/hyperframes)** (open source, HeyGen) | Renders an HTML composition to MP4 or WebM, deterministically, frame by frame in headless Chrome. Timed clips, images and nested compositions via `data-start` / `data-duration`. A linter for common mistakes, a preview studio to scrub the timeline, key-frame snapshots, beat detection from a music file, and website capture for product footage | No audio mixing. 3D only through Three.js driven by the timeline. Strict linter. A 0.x release: pin the version |
| **[GSAP](https://gsap.com)** | The animation engine inside the page: tweens, eases, timelines, SVG and filter animation. A paused timeline is what HyperFrames seeks | Every value must be a function of time: no timers, `Date.now()` or unseeded randomness. One owner per property |
| **[FFmpeg](https://ffmpeg.org)** | Trims, rescales and re-times generated clips to the film's size and frame rate; filters and EQs sound; loudness normalisation to a target (for example -14 LUFS) and limiting; joins the silent render with the mix; encodes for delivery; pulls frames, contact sheets and loudness readings for review | A command line tool: keep the recipes in the film's scripts, not in your head |
| **Python: numpy, soundfile, [librosa](https://librosa.org)** | The mix as code: each effect placed at its cue time with a gain, music ducked or cut with gain curves. librosa measures tempo, beats, onsets and loudness per second, which is how scores are chosen and effects are checked | Decode and encode through FFmpeg |
| **[Playwright](https://playwright.dev)** | Pixel-exact captures of your real site and app, for explainers that must show the true product | Capture public or demo data only |

Other renderers fit other teams: [Remotion](https://www.remotion.dev) for React (check its company licence), [Motion Canvas](https://motioncanvas.io) and [Manim](https://www.manim.community) for diagram and maths explainers, [Blender](https://www.blender.org) for real 3D. The rules below hold for any of them.

## 1. Ask, then plan

Open with a short round of questions unless the user said to just go: purpose and viewer, length and aspect ratio, the look (hand-drawn, flat vector, paper cut, photoreal, motion graphics), how each shot should move (code animation or pose cuts, generated clips, or a mix per shot), narration / music / sound design, allowed brand assets and claims, and whether real people appear (and have agreed). Offer two or three concrete directions with a recommendation.

**Animate only when the style or the request calls for it.** Cut-out poses swapped on the beat, code animation and held stills are styles in their own right: cheaper, exact, and often what a paper or explainer piece wants. Recommend per shot; never default to generating clips for everything.

Then write:
- **`BRIEF.md`**: viewer, the one takeaway, length, look, claims you may make. For an explainer, a **`FACTS.md`** with a source for every on-screen claim, and a list of words to avoid.
- **A shot table**: time, on screen, purpose of the beat, carried object, audio cue.
- **One `cues.json`** that picture *and* sound both read: tempo, every cut, every hit, every narration line. Retiming is then one edit.
- **A still animatic** (the approved stills on the real cue times, with the real score) before any clip spends. Say plainly that it is stills only, or put one real clip in it: held stills look far worse than the film will, and a viewer judges the film by it.

Choose the medium by what must be true on screen:

| On screen | Make it with |
|---|---|
| Text, logos, UI, code, numbers, diagrams, anything a viewer reads | Code (never a generator: it garbles letters) |
| Actions that must line up with sound (contacts, hits, clicks) | Code animation. Generated clips invent their own timing and props, and the sound can only guess |
| People, places, weather, organic performance | Generated stills, animated with image-to-video |
| Product truth (your site, your app, a real API response) | Real captures (Playwright) and real calls |

A stylised look (flat vector, line art, paper cut) beats generic realism: it hides model artefacts and cuts well with code.

## 2. Opper, medium by medium

One key (`Authorization: Bearer $OPPER_API_KEY`), base `https://api.opper.ai`. **For request fields, response shapes, async jobs and model discovery, load [`opper-multimodal`](https://skills.opper.ai/opper-multimodal/SKILL.md);** this section keeps only what matters for making a video. Every surface has a model list; read it, never hardcode ids from memory, and test one call before a batch.

| Medium | Call | Models |
|---|---|---|
| Image / edit | `POST /v3/images` | `GET /v3/images/models` |
| Video clip (async) | `POST /v3/videos`, poll `status_url` | `GET /v3/videos/models` |
| Voice | `POST /v3/audio/speech` | `GET /v3/audio/voices?model=<model>` for its voices |
| Music, sound effects | `POST /v3/audio/generations` | `GET /v3/audio/models?type=music` / `?type=sound` |
| Transcription (captions) | `POST /v3/audio/transcriptions` | `GET /v3/audio/models?type=stt` |
| Text, vision, listening | `POST /v3/compat/chat/completions` (OpenAI shape) | `GET /v3/models` |

Every response carries `usage.cost`. Log it with the prompt, model and id in a `manifest.json` per folder, skip files that already exist, and never overwrite a named export. Send `store: false` for throwaway variants. Provider-specific fields go in `parameters` and pass through unchanged.

What we recommend in 2026-09 (check the lists; newer may exist): `openai/gpt-image-2.5-sunburst` and `openai/gpt-image-2.5-flare` first for stills and edits (then `gemini/gemini-3-pro-image-preview`), `gemini-omni-1.1-flash`, `wan3.0` (or `wan3.0-prime`), `seedance-2.5` and `kling-3.0-pro-i2v` for clips (bare names are pooled across providers; prefix one, like `fal/kling-3.0-pro-i2v`, to pin it), `elevenlabs/eleven_v4` for voice, `elevenlabs/music_v2_5` for scores, `elevenlabs/eleven_text_to_sound_v2` for effects, `elevenlabs/scribe_v2` for captions, `gemini/gemini-3.1-pro-preview` to look and listen.

### Images

```json
{"model": "...", "prompt": "...", "image": "data:image/png;base64,...", "reference_images": ["data:..."], "quality": "high", "store": false}
```
- `image` = the picture to edit; `reference_images` = identity or style to keep.
- **Fix a still by editing it, not by regenerating.** "Same drawing, identical in every line; the only change: ...". Regenerating from references snaps back to their composition.
- **Colours as hex or light, never as objects.** "A peach sphere" came back as a fruit with a stem. Write "a round ball of light, coral #F9B58C core, cyan #8CECF2 glow".
- Probe the size you got. Resolution can hide behind another knob (`quality: "high"` gave 2K on Gemini image models). Send `size` on edits too: an edit without it came back square. An edit can also shift the picture (a 1024x1824 edit of a 1080x1920 frame moved a sofa 72 px): register edited frames to the original before animating them.
- Photo to drawing: edit the real photo ("redraw this exact photograph as pencil line art, same composition, same people") so a code wipe from photo to drawing registers.

### Video clips

```json
{"model": "...", "prompt": "motion, then: keep the exact art style, no new people, no text", "image": "data:...", "seconds": 5, "store": false}
```
- Animate a still you already approved. Append each submission id to `submitted.jsonl` **before** polling, and resume from it after a crash instead of paying again.
- Plan to use the first good 1 to 2.5 s; clips drift after that. Cut on the beat.
- Watch for what sampled frames hide: a cut to a new angle mid-clip, invented people, props turning into other objects, a light that tints every shirt. Check a 4 to 5 fps strip and crop the key prop at full size.
- A clip that continues a still must continue its camera move, or the handover pops.
- **Clip audio:** when the score carries the film, set clip audio off. When you want the clip's own sound, name only what the scene makes, in order: `Diegetic sound only: oil sizzling, the pan scraping the burner, a spoon tap, the stove knob click, then room tone.` Never write instrument or genre words, or even "no music": with "No music" Seedance still added music, and Scenario reports such words can trip audio moderation. With this wording Kling gave clean foley with no music or voice, the events in order.
- Clips come back a little longer than requested (about 0.04 s): read the real duration before laying out the timeline.
- **Handing a generated subject to code** (an object flying out of the clip): measure its last position from pixels, edit the last frame to remove it, check the two register, then animate a code-drawn copy from exactly that spot.

### Voice

```json
{"model": "...", "input": "One full sentence.", "voice": "<voice id>", "format": "mp3"}
```
- **One file per line**, placed on the cue file. You control the gaps and a bad line is a cheap retake.
- Write warm, full sentences; punctuation is the direction. Agents default to clipped fragments and piles of numbers.
- Pick the voice by ear: `GET /v3/audio/voices?model=<model>` (filter by `language`, `gender`) lists each voice with its accent and, on ElevenLabs, a `preview_url`. Shortlist two or three, then render the same real line in each before choosing.
- The audio comes back in `audio.b64_json`.

### Music

```json
{"model": "...", "sections": [
  {"duration_ms": 8000, "prompt": "120 BPM, C major, playful, sound from the first beat", "styles": ["pizzicato", "marimba", "glockenspiel", "instrumental"]},
  {"duration_ms": 8000, "prompt": "the same instruments, fuller, ending on a resolved chord", "styles": ["pizzicato", "marimba", "glockenspiel", "instrumental"]}]}
```
- Without a timed plan, send `prompt` + `duration_seconds`. With `sections`, the sections set the length and carry the prompt; some models refuse a top-level `prompt` or `instrumental` alongside them (put "instrumental" in `styles`).
- Ask for structure: a round tempo (120 BPM gives a clean grid), key, instruments, "sound from the first beat", "resolved final chord, then ring-out". Never name artists.
- **One instrument palette in every section; vary intensity, not instruments.** Changing instruments between sections gives an abrupt genre switch. This was the single biggest music improvement.
- Describe each section in `prompt` (and `styles`); words to be sung go only in `lyrics`. Put section boundaries on the phrase grid (at 120 BPM, multiples of 2 or 4 s); other boundaries get moved.
- **Do breakdowns, drops and silences in the mix,** not in the prompt. Models often ignore "drums out here".
- Generate 3 or 4, measure all (beats, loudness per second), listen to the survivors, and cue the picture to the measured track.
- Send a `seed` with your `sections` and note it next to each take, so the one you choose can be rendered again. A changed section can still change the whole track, so measure any retake again. On ElevenLabs a seed works only with sections. Sound effects take no seed: generate a few takes and pick one.
- Timbre is the weak point: generated orchestral strings and piano often sound synthetic. Light, plucked and electronic palettes hold up better.

### Sound effects

```json
{"model": "...", "prompt": "a small wooden domino tipping onto wood, one click, close, dry", "duration_seconds": 0.6}
```
- **One action per request, described as the material on screen.** Add `loop: true` for ambience and hums.
- Place each effect on the **measured** action frame (motion or brightness peak), not a guess.
- Normalise every effect to about -1 dBFS peak before gains; some come back nearly silent. Place each effect by its measured onset, not its file start (a lever's clack began 0.375 s into its file).
- **A diegetic bed under a song needs a level target,** about -18 LUFS under a -14 LUFS song, ducking the song a few dB under the key effects. Check the lift at every hit: a bed at -26 LUFS was "in the mix" and inaudible.

### Transcription and captions

Transcribe the **final mix** for word timings. Pass brand names as key terms (`parameters: {"keyterms": ["Opper"]}`; without it "Opper" became "OPA"). Take the words and punctuation from the script, and only the timings from the transcript.

### Looking and listening

Send frames (`image_url`) or the mix (`input_audio`) to a chat model and ask for a blunt score. Useful as a second opinion on "is this good". Never for "when" (timestamps are seconds off), and not as proof: a listening model rated a mix 10/10 with "every effect audible" while measurement showed effects 10 to 20 dB under the music, and vision critics invented objects. Measurement and a fresh critic decide.

### Films made mostly of generated footage

Cinematic brand films and multi-style montages (each scene in a different art style) have their own method: canon
blocks and a hero frame, identity sheets instead of real photos, last frames made by editing the first, clip prompts
written as timed scripts and read before any spend, a model per shot from a shoot-out, events measured from pixels and
retimed onto beats, motivated transitions that reveal the next scene from its own first frame, a character animated
over a code set (keyed off green), a code assist for the key moment, songs with lyrics and karaoke subtitles. **Read [references/generated-films.md](references/generated-films.md)**
before planning one.

## 3. Assembly

Any code renderer works if **every frame is a pure function of time**: frame *n* renders at *n / fps* after any seek. No free-running timers, `Date.now()` or unseeded randomness.

- **Bake every change.** If you precompute state per frame and skip "unchanged" values, use a tiny threshold. A 0.02 threshold froze a slow zoom, which then jumped 40 px twice.
- **Pre-roll every clip** (decode it about 0.2 s early, hidden) and switch hard cuts a hair early, or the cut frame is blank.
- One owner per animated property; move with transforms; bind every font to a local file; frame 0 is a finished picture (it is the thumbnail).
- Draw the logo from the official SVG paths, in brand colours; generate the wall, code the sign.
- Motion that reads as crafted: gravity arcs, anticipation, squash and stretch, follow-through; nothing passes through anything; energy never increases after a bounce.
- Match source frame rates, or motion judders (24 fps clips stretched to 30 repeat every fifth frame).

## 4. Sound

- **Audio first:** lock and measure the score, write beats and hits into `cues.json`, then animate to them.
- Duck the music 8 to 12 dB around each action and 10 to 12 LU under narration.
- **Measure effect lift:** compare the mix with a music-only render in the 80 ms after each cue (1.5 to 8 kHz band). Under a few dB is inaudible, whatever a listener model says.
- Master with two-pass loudness normalisation to -14 LUFS, then a limiter at about -1.5 dBTP. Always export a music-only stem.

## 5. Review loop

1. **Checks on every render:** loudness and true peak; frames that differ sharply from both neighbours (glitches, a layer one frame early); near-frozen time; single-colour frames; cut and hit sync within 1 to 2 frames.
2. **Look yourself:** contact sheets at 2 to 4 fps plus every frame of each transition.
3. **A fresh critic each round:** a new agent with only the render, the brief and the facts, returning ranked defects and SHIP or ONE MORE PASS. It catches what the builder cannot: blank cut frames, a fruit for an orb, a highlight on the wrong code line, an invented failover shown as "a real call". **On a film the owner already approved, its findings are proposals, not fixes:** show them and let the owner choose. A critic checks its own rubric, not whether the film got better; applying every fix to an approved brand film (cutting a scale shot for pace, a colour wash, a code overlay patching the story beat) passed the critic and made the film worse.
4. **A human watches last.** They catch what nobody measured (a camera that "jumps", tools whose sounds do not match). Measured is not judged; say which is which.

## 6. Delivery and honesty

- Export H.264 High, yuv420p, constant frame rate, AAC 48 kHz, `+faststart`, trimmed to the timeline. A captioned and a clean version, plus an `.srt`. Vertical needs its own layout, not a crop.
- On-screen claims come only from the facts file. Label illustrative data as illustrative; never put a mock next to the word "real".
- Real people need consent; keep their photos out of shared repos. Only say "made with Opper" for what went through Opper.