--- name: code-video description: "Make any video from code (Canvas, SVG, HTML/CSS, Three.js, Remotion, Manim, p5.js), with sound: turn the request into a checkable spec, render every frame as a pure function of time, capture to MP4, review with contact sheets and verify the file. Includes craft guides for explainers, stylized looks, simulations, characters, 3D scenes, motion graphics and promos, sound and narration, and music sync, plus helper scripts. Use for any request to make, animate or render a video, animation, explainer, promo or motion graphic in code." --- # Code video core A code video is a program `frame(t) → image`, sampled at `t = i / fps` and encoded. Almost every failure in this kind of work is one of four things: the film is never finished, the picture depends on wall-clock time, nobody looked at the frames, or the file does not match the brief. This skill prevents those four. ## 0. Read the craft guides you need This skill is one folder; every path in it is relative to the folder. Besides this file it holds craft guides in `crafts/` and helper scripts in `scripts/` (audio packages in `requirements.txt`). Read every guide that matches the request before writing `SPEC.md`; most videos need two or three. | Guide | Read when the request has… | |---|---| | `crafts/explainer-diagrams.md` | anything to teach or explain: math, science, history, processes, data | | `crafts/stylized-looks.md` | a named style, medium or artist (ink wash, pixel art, cut paper, vaporwave, watercolour, …) | | `crafts/simulations-and-generative.md` | particles, flocks, cellular automata, physics, fields, shaders, generative patterns | | `crafts/character-and-creature-motion.md` | people, animals or creatures that move or act | | `crafts/threejs-scenes-and-cameras.md` | a 3D scene, product or camera move | | `crafts/motion-graphics-and-promos.md` | kinetic type, UI, logos, product or brand promos | | `crafts/sound-design-and-narration.md` | any audio: narration, music, effects, ambience | | `crafts/music-and-beat-sync.md` | music-led films, beat-cut picture, supplied tracks or lyrics | A request that names a style always needs `crafts/stylized-looks.md`, and a film with any audio always needs `crafts/sound-design-and-narration.md`. ## 1. Finish the film - The deliverable is a playable video that meets the brief. Code, stills, a plan or a preview are intermediate results. Keep working until the final file exists and has passed section 6. - Decide composition, palette, easing and implementation details yourself and note the choices in `SPEC.md`. Ask the user only about the few open choices that would change the whole film (for example language, aspect ratio, length, or whether there is a voice), and only when you can ask; otherwise choose a sensible default and say which. - Budget the time. Get a rough full-length render early (even ugly), then improve it. A finished, slightly plain film beats an unfinished ambitious one. - Timebox sub-problems. If one problem (recalling an exact pattern or constant, tuning one effect, chasing one bug) has taken about 30 minutes without visible progress, change approach: simplify the effect, derive the value from its definition, use the reference data in the craft guides, or fall back to a design that does not need it. Do not spend hours reconstructing data from memory. ## 2. Turn the brief into a spec Before writing code, write `SPEC.md` in the project folder: - **Format:** width × height, fps, duration (target and allowed range), frame count = round(duration × fps), loop or not, and audio: required, forbidden, or left open. When it is open, decide from the brief whether sound would serve the film (section 10). Write the sound requirements (type, voice language, fixed lines, sync points) into the spec too. - **Exact copy:** every on-screen string the brief fixes, verbatim, in order. Nothing else may claim facts, prices or specifications. - **Facts and numbers:** every value the picture must show or obey (dates, physical constants, data values, counts). Note the source the brief gives. - **Assets:** each supplied file and how it must be used (unaltered logo, screenshot fidelity, photo kept photographic). - **Must-haves:** one line per requirement in the brief ("camera orbits counterclockwise ~90°", "dominoes never interpenetrate"). Each line becomes a check in section 6. Choose the stack for the content, not out of habit: | Content | Good default | |---|---| | 2D diagrams, math, particles, pixel art, painterly textures | Canvas 2D (browser or `@napi-rs/canvas` in Node) | | Crisp vector shapes, UI, kinetic type, logos | SVG / HTML+CSS driven by a paused timeline, or Remotion | | LaTeX-heavy math | Manim (or KaTeX/MathJax rendered to SVG) | | 3D scenes, products, cameras | Three.js (see `crafts/threejs-scenes-and-cameras.md`) | | Shaders, aurora, fields, post effects | WebGL / GLSL fragment shaders | ## 3. Time is the only input - Expose one entry point that draws the exact state at time `t`, for example `window.seek = async (t) => {...}` plus `window.READY = true` once fonts and images are loaded. Frame `i` is rendered at `t = i / fps`. - The same `t` must always give the same image, in any order, in a fresh page. Forbidden: `Date.now()`, `performance.now()`, `requestAnimationFrame` loops that advance state, CSS transitions/animations left running, a playing GSAP timeline, unseeded `Math.random()`, state carried over from the previous frame. - GSAP: build a `paused: true` timeline and call `tl.seek(t)`. CSS animations: set `animation-play-state: paused` and a negative `animation-delay`, or compute styles from `t` yourself. Remotion: derive everything from `useCurrentFrame()`. - Randomness: seed once, then keep values. A tiny seeded generator is enough: ```js function mulberry32(a){return function(){a|=0;a=a+0x6D2B79F5|0;let t=Math.imul(a^a>>>15,1|a);t=t+Math.imul(t^t>>>7,61|t)^t;return((t^t>>>14)>>>0)/4294967296}} ``` - Simulations (physics, flocks, cellular automata, crowds) are computed once with a fixed time step and seed, stored, and sampled by `t` (interpolate between stored steps). Never integrate inside the render call. - Easing: write it as functions of `t` (`smoothstep`, cubic in/out, closed-form damped springs). A value that changes target several times is a sum of eased segments, still a pure function of `t`. ## 4. Capture and encode - **Browser pages:** `scripts/capture.mjs` serves the project folder, opens the page in headless Chromium (Playwright or Puppeteer, resolved from the project's `node_modules` first), calls `seek(t)` for each frame and pipes screenshots into ffmpeg: `node scripts/capture.mjs --page index.html --fps 30 --dur 12 --size 1920x1080 --out out/video.mp4` Test it on 2–3 frames (`--frames 3`) before anything else. If Chromium cannot start (for example missing system libraries) and you cannot fix that quickly, switch early to a path that needs no browser: Canvas 2D in Node with `@napi-rs/canvas`, or Manim. Do not spend your time fighting the browser. - **Node canvas:** draw each frame with `@napi-rs/canvas` and write raw RGBA to ffmpeg's stdin (`-f rawvideo -pix_fmt rgba -s WxH -r FPS -i -`). - **Remotion / Manim:** use their own renderers, then re-encode if the codec or pixel format differs from the spec. - Encode for compatibility: `-c:v libx264 -pix_fmt yuv420p -crf 18 -preset medium -movflags +faststart`. A film without sound has no audio stream (`-an`), not a silent track. Sound is muxed in a separate step (section 10). Width and height must be even. - Headless WebGL may fall back to software rendering, which is slow. Measure the time per frame on a short range first and plan the full render from it. Render previews at half resolution if needed, but the final at full size. - Keep each render in a new versioned path; never overwrite the last good film until the new one passes its checks. - **Frame budget:** before building the whole film, time one full-resolution frame of the busiest moment. Aim for roughly ≤ 300 ms/frame (a 100 s film at 30 fps is then 15 min per full render) so that several review passes fit. If a hand-written NumPy/PIL compositor is far slower, do not optimize it for hours: move the drawing to Canvas 2D or WebGL, which composite layers natively. In NumPy, avoid per-frame work on whole huge textures (boolean-mask assignment into multi-thousand-pixel float arrays, full-frame `np.roll`); crop to the visible window and precompute static layers once. - **Long renders:** a 2-minute film at 30 fps is 3,600 frames. Measure ms/frame early and multiply. Render independent chunks in parallel with `--start` and `--frames` (each chunk to its own file, all from the same `seek(t)` code), then join them losslessly: list the chunks in `chunks.txt` (`file 'part_000.mp4'` …) and run `ffmpeg -f concat -safe 0 -i chunks.txt -c copy video.mp4`. Check the joined frame count. ### Long jobs and background processes - **Smoke-test before a long run.** Run any bake, simulation or render on a tiny case first and assert that it does something: state changes after N steps, the output file grows, an invariant holds. A stepping function whose result is never assigned bakes nothing for half an hour. - Write each run to a new file name and print progress with `flush=True` to a log. Before you inspect a result, confirm it came from the run you just started (timestamp, a config line in the log), not from an older file. - Record the PID of a background job (`echo $! > build/sim.pid`) and stop it with `kill $(cat build/sim.pid)`. `pkill -f ` can match your own shell command and kill it. - The machine may be shared with other jobs. Measure throughput on the real run and re-plan from that; do not assume an earlier measurement still holds. ## 5. Look at the frames Pixels are the only evidence. Repeat this loop at least twice. You know what the code is meant to draw, so you will tend to see it in the frame even when it is not there. Read every still in two steps: - **Cold read first.** Describe the image as a stranger would, without thinking of the code: what is the main subject, could someone name it within a second, how large is it, what is the brightest and most contrasting thing, what can be read. Write that description down. - **Then compare** it with the brief. Every required object must show up in the cold read. If the brief says "koi" and the cold read says "a dark arrow-shaped blob", the koi is not there yet, whatever the code is meant to draw. - **Measure, do not estimate:** the subject's bounding box at key moments (at least about 1/3 of the frame height unless the brief wants it small), the smallest text height in pixels (at least about 28 px at 1080p), and whether the hero effect is brighter or more contrasting than the background details around it. 1. **Stills first.** Render the few moments that carry the most risk (the brief usually names them), view them directly, and fix the problems before rendering the whole film. 2. **Contact sheet.** After a full render: `sh scripts/contact_sheet.sh out/video.mp4 out/sheet.jpg 1` (one frame per second; use 2 s steps and 8 columns for films over a minute) and read it end to end: composition, readability, continuity, empty or broken frames, and for long films whether sections look alike. 3. **Action strips.** For every key action (a collision, a morph, a catch, a reveal), extract frames 0.1–0.2 s apart around it. Stills hide teleports, pops and sliding; strips show them. 4. **Watch at normal speed.** Then name the weakest moment, fix it, re-render, compare both versions and keep the better one. Readability rule of thumb: after a text finishes animating in, hold it about `letters / 15 + 1.5` s (English) or `characters / 4.5 + 1.5` s (Chinese); a title card at least 2 s in a short film. Never shrink text below about 28 px at 1080p to make it fit. ## 6. Verify before delivery Run `sh scripts/check_video.sh out/final.mp4 1920 1080 30 10 15` (add `1` or `0` at the end to require or forbid an audio stream). It reports resolution, fps, duration, frame count, codec, pixel format, audio streams and their length, black segments and frozen segments, and fails on mismatches. Then go through `SPEC.md` line by line against the actual frames and write `VERIFY.md`: each must-have, pass/fail, and the timestamp or still that shows it. Do not write "checked" for anything you did not look at. Evidence in `VERIFY.md` is what the pixels show, not what the code does. "Pavilion mid-slope, drawn by `draw_pavilion()`" is not evidence. "t=38 s still: a small roofed pavilion is visible at about (820, 410), about 60 px tall" is. If the pixels do not show a must-have clearly, mark it FAIL and fix it, or list it as a known gap. ## 7. Failures seen most often | Symptom | Cause | Fix | |---|---|---| | Video plays too fast or jitters | Animation driven by wall clock or rAF during capture | Drive everything from `t` | | First frames blank or fonts wrong | Captured before fonts/images loaded | Set `READY` after `document.fonts.ready` and image decode | | Particles or textures shimmer | Randomness re-drawn every frame | Seed once, store per-object values | | Subject too small or pushed to an edge | Never checked stills at final size | Subject ≥ 1/3 frame height at key moments | | Subject not recognizable (a koi that reads as a leaf, a mountain that is a triangle) | Shape built from primitives and judged against intent | Cold read (section 5); design the silhouette first and check it at thumbnail size | | Hero effect weak (tiny, dim fireworks; reflections brighter than the bursts) | Background details compete with the subject | Make the subject the highest-contrast element; dim the rest | | Hours lost reconstructing a pattern or constant | Recalled from memory and debugged by trial | Timebox (section 1); use reference data in the craft guides | | Text overlaps or leaves the frame | Positions estimated, not measured | Measure text boxes with the real font; keep a 5% safe margin | | Duration or fps off by a few frames | Rounding or a CSS/GSAP duration mismatch | Frame count = round(dur × fps); check with `check_video.sh` | | Invented claims on screen | Copy written from memory | Only the strings in `SPEC.md` | ## 8. Long films (1–2 minutes) - **Plan the structure before coding.** Write `STRUCTURE.md`: the sections or movements in order, what each one shows and changes, roughly how long each takes, and how each hands over to the next. Total the estimates against the target length. - **Every section needs its own visual state** (a new scale, place, palette, camera, density or mechanism) and something meaningful should change at least every few seconds. The typical failure of a long code film is one effect repeated for two minutes. - **Vary the pace:** a hook early, the main content in the middle with at least one acceleration and one breathing pause, a clear ending that echoes the opening. - **Continuity:** keep one world, one palette logic and one type system across sections; transitions grow out of shared shapes or motion rather than generic fades. - **Work in passes:** get a rough full-length cut early (simple shapes, real timing), then refine section by section. Re-render only the sections you changed (chunks) and re-join. - **Review budget:** check stills from every section, not just the opening. Watch the whole film at least twice before delivery. ## 9. Seamless loops When the brief asks for a loop, make every animated quantity periodic with a period that divides the loop length (rotations, oscillations, noise offsets, camera paths, overlay grain). Frame `N` would equal frame `0`; render frames `0 … N-1`. Check by playing the end into the start and by comparing the last and first frames: they must be consecutive states, not duplicates. Simulations that must loop are designed on the loop period from the start (see `crafts/simulations-and-generative.md`). ## 10. Sound - The brief says whether the film has sound and what kind (narration, voiceover, music, effects, ambience, sonification). When it says nothing, a film usually benefits from music and effects; choose a sound plan that fits and state it in `SPEC.md`. When it asks for sound, plan and build it with `crafts/sound-design-and-narration.md`; for music-led films also `crafts/music-and-beat-sync.md`. When it says no audio, deliver no audio stream. - Sound is built from the same timeline as the picture: export the animation's events and state, then synthesize and place sounds from them. Never hand-type a second timing list. - Deliver stereo AAC at 48 kHz, exactly as long as the video, mastered to about −16 to −14 LUFS with true peak ≤ −1 dBTP, and check it with `scripts/audio_check.sh` and `scripts/sync_check.py`. ## 11. Assets, fonts and tools - Supplied assets are used as given: no recolouring, stretching or redrawing of logos; UI rebuilt in code must match the screenshots; photos stay photographic. - Prefer the tools and fonts that are already installed; check fonts with `fc-list`. Use fonts from the environment or from the project's `assets/` (install others only if you may, and respect their licences); check that every glyph renders (no tofu boxes, especially for Chinese). - The picture is made by code. Use image or video generation models, stock footage or downloaded artwork only when the user asks for them or supplies them. ### Video assets inside the film When the brief supplies video clips to show inside the film (in a card, a screen, a wall of tiles): - **Probe first:** `ffprobe` each clip for size, fps, duration and audio. Write in `SPEC.md` which window of which clip appears where (clip in/out time, film start time, speed, size on screen). - **Do not play `