# Control protocol & MCP ## Desktop control channel `filmcraft --control 9876` (or `FILMCRAFT_CONTROL_PORT`) listens on `127.0.0.1:` (loopback only). One JSON request per line → one JSON reply per line. The app opts out of macOS App Nap (`apps/filmcraft/src/app_nap.rs`): a hidden window would otherwise drop the whole process to background priority, and on a busy machine it would stop answering. **Only requests are read.** Every line must be a JSON object with a string `method` (`id` and `params` are optional; blank lines are skipped). Anything else gets one error reply (`{"ok": false, "error": "… closing the connection"}`) and the server **closes the connection**, so nothing sent after it on that connection runs. That covers text that isn't JSON, a JSON array or number, an object without `method`, invalid UTF-8, and a line longer than 4 MiB. An HTTP request (for example a web page's cross-origin `fetch` to `127.0.0.1:`) therefore can't smuggle a command in its body: its request line is rejected first. At most 16 connections are served at once; further ones get an error line and are closed. Clients that get an error reply should reconnect. The port has no authentication, so only enable it while you use it. Transport: `apps/filmcraft/src/control_server.rs`. ```json {"id": 1, "method": "engine.execute", "params": {"command": "sequence.addEdit", "params": {"seconds": 3}}} {"id": 1, "ok": true, "result": {"cuts": 2}} ``` Methods (handlers in `crates/ui-egui/src/control.rs`): | Method | Params | | |---|---|---| | `engine.execute` / `ui.menu.invoke` | `{command, params}` | any engine or UI command id | | `engine.commands` / `ui.menu.list` | – | command registry / menu tree | | `ui.inspect` | – | UI state (tool, workspace, dock, timeline view, playback, fps…) | | `ui.elements` | `{prefix?}` | every on-screen interactive element: id, label, rect | | `ui.set` | `{tool, workspace, mode, theme, focused, playbackRes, timeline:{pps,scroll,fit}, program:{…}, source:{…}, clipDialog:{param: value}, menuDialog:{param: value}, panels:{scopes, timecode, reference, events, progress}, export:{…}}` | `program` / `source` merge fields into the monitor view state (`res`, `paused_res`, `high_quality`, `display`, `zoom`, `pan`, `show_rulers`, `show_guides`, `lock_guides`, `snap`, `guides`, `compare_ref`, …; see [monitors.md](monitors.md)); `clipDialog` sets fields of the open Edit / Clip / File dialog (Paste Attributes, Make Subclip, Frame Hold Options, …); `menuDialog` those of the M3.11 dialogs (Find, Create Search Bin, Project Settings, Scene Edit Detection, Simplify Sequence, Automate to Sequence…; see `panels::menu_dialogs`); `panels` merges into the panel settings (`ui.inspect` → `ui.panels`; see `panels::panel_state`): Lumetri Scopes `{shown: ["vectorscopeYuv"|"vectorscopeHls"|"histogram"|"parade"|"waveform"], waveformType, paradeType, colorSpace, brightness, scale, clamp, targets}`, Timecode `{rows: [{source, mode, display}], showName}`, Reference Monitor `{ganged, time, display: "composite"|"scopes", scopes}`, Events `{level, selected}`, Progress `{showFinished}` `export` deep-merges into Export mode state (`preset` — applied first —, `settings` (ExportSettings JSON), `fileName`, `location`, `range`, `customStart`, `customEnd`, `openSections`, `manager` (Preset Manager: `query`, `favoritesOnly`, `selected`, `saveName`), `quickOpen`, `quickPath`, `quickPreset`); `program` / `source` merge fields into the monitor view state (`res`, `paused_res`, `high_quality`, `display`, `zoom`, `pan`, `show_rulers`, `show_guides`, `lock_guides`, `snap`, `guides`, `compare_ref`, …; see [monitors.md](monitors.md)); `clipDialog` sets fields of the open Edit / Clip / File dialog (Paste Attributes, Make Subclip, Frame Hold Options, …); `menuDialog` those of the M3.11 dialogs (Find, Create Search Bin, Project Settings, Scene Edit Detection, Simplify Sequence, Automate to Sequence…; see `panels::menu_dialogs`); `projectPanel` merges into the Project panel's view state (`bin` shown in place, `tabs` (bins in a tab or window: `{bin, floating, view, iconSize, nav}`), `activeTab`, `selectedBin`, `rename: {item|bin, text}`, `dialog` (`{"metadataDisplay": {columns, filter}}`, `{"savePresetAs": {name}}`, `{"managePresets": {selected, name}}`, `{"freeformOptions": {options}}`, `{"saveArrangement": {name, bin}}`)); `mediaBrowser` into the Media Browser's (`expanded`, `pathEdit`, `editColumns`, `treeWidth`). View settings, columns and presets are engine commands (`project.view.set`, `project.columns.set`, `project.viewPreset.*`, `mediaBrowser.settings`) | | `ui.panel.show` / `ui.panel.close` | `{panel}` | | | `ui.click` / `ui.move` | `{id}` or `{x,y}`, `button`, `count`, `modifiers` | synthetic pointer input | | `ui.drag` | `{from, to, steps, modifiers}` | press–move–release | | `ui.scroll` | `{id|x,y, dx, dy, modifiers}` | wheel / trackpad | | `ui.key` / `ui.type` | `{key}` (`Cmd+K`, `Space`…) / `{text}` | keyboard | | `ui.timeline.hit` / `ui.timeline.locate` | `{x,y}` / `{clip, edge?}` | timeline hit-testing | | `ui.playback` | `{action: play|stop|toggle, speed?}` | | | `ui.screenshot` | `{path?, panel?}` | PNG of the window or one panel; fails after 10 s when no frame is presented (window hidden, display asleep) | | `ui.resize`, `ui.focus`, `app.quit` | | `ui.focus` is the only request that activates the app and takes keyboard focus | | `perf.stats` | – | performance counters (also the command `perf.stats`, so `engine.execute` and MCP `command_run` reach it; headless sessions return the engine part). `decode`: GOP-cache requests / hits / `cacheHitRate`, decoder seeks, samples decoded and skipped while catching up, `draftFrames` (decoded in draft mode), `h264Threads` (frame-threading workers per H.264 decoder), `liveDecoders` (sources holding a decoder now), `cacheMB` / `cacheBudgetMB` (decoded frames all sources hold, and the shared budget), `planePoolMB` / `planesReused` (idle plane buffers of evicted frames, and planes decoded into a recycled one), decoder ms (total and per sample), `framesDecoded`, `hardware` (Settings ▸ Playback ▸ Hardware decoding: `enabled`, pictures from hardware decoders `frames` vs `softwareFrames`, `sessions`, `declined` streams, mid-stream `fallbacks`); `playback`: playing, shown / dropped frames and `dropRate` of the current or last play, resolution, `draftDecode` (Settings ▸ Playback ▸ Draft decoding), `audio` (desktop: sound is mixed 200 ms ahead of the device; device `callbacks`, `underruns` and `missingFrames` played as silence because the mixer fell behind, `minLeadMs` the least sound left buffered); `frames`: frame-worker jobs, cancelled, `requestHitRate`, `decodeMs` / `renderMs` (mean, p50, p95 of the last 256 jobs), queue, render-cost estimate, cache use; `ui`: fps and frame ms; `process`: CPU seconds; `media`, `jobs`. Counters are cumulative: diff two readings to measure an interval ([performance.md](performance.md)) | **Selection or explicit targets.** A command that acts on the selection is "not available right now" when nothing is selected. When `params` name the targets under a key the command documents (`clips` / `clip`: timeline clips of the active sequence, `items` / `item`: project items), enablement is checked against those instead and the selection is left as it is; the command's other conditions still apply. `engine.commands` and the menus always show the selection's enablement. **Focus.** Driving the app never steals the user's keyboard. Started with `--control`, the app opens without activating itself. UI requests and screenshots that need a rendered frame bring the window forward without making it key (macOS `orderFrontRegardless`), so the user's typing keeps going to the app they are in. Only an explicit `ui.focus` activates FilmCraft. Element ids are stable, e.g. `timeline.clip.`, `timeline.track.V1.lock`, `tools.Razor`, `project.item.`, `effects.item.gaussian_blur`, `panel.tab.Timeline`, `program.transport.playback.toggle`. Audio mixing: `mixer.*` commands (strips `"A1"`, `"S1"`, `"Mix"` or ids; lanes `volume`, `pan`, `mute`, `send..level`, `fx..`): `mixer.inspect`, `mixer.setStrip` (volume, pan, mute, solo, record arm, solo safe, mode Off/Read/Latch/Touch/Write, output, input map, channels), `mixer.setValue`, `mixer.touch` / `mixer.release` (a fader gesture; recorded during an automation pass), `mixer.recordStart` / `mixer.recordStop` (playback runs them), `mixer.addSubmix`, `mixer.deleteSubmix`, `mixer.addInsert` / `removeInsert` / `setInsert`, `mixer.addSend` / `setSend` / `removeSend`, `mixer.setKeyframe` / `deleteKeyframe` / `moveKeyframe` / `clearLane`, `mixer.writeAutomation`; `clipMixer.set`; `clip.audioGain {clips?, mode: set|adjust|normalizeMax|normalizeAll, db}`, `clip.audioPeak`; `effects.setDefaultTransition`. UI ids: `mixer..|send.|meter|name>` (popup entries below them, e.g. `mixer.A1.mode.Touch`, `mixer.A1.fx.0.studio_reverb`), `mixer.showEffects`, `mixer.transport.`, `clipMixer.A1.`, `timeline.track.A1.keyframes[.|.clip]`, `timeline.track.A1.lane[.kf.]`, `audioGain.`. Essential Sound: `essentialSound.inspect`, `essentialSound.setType {type: dialogue|music|sfx|ambience}`, `essentialSound.clearType`, `essentialSound.set {key, value}` or `{values: {key: value}}` (keys such as `repair.noise.on`, `repair.noise.amount` (0–10), `repair.humHz`, `clarity.eqPreset`, `clarity.enhanceTone`, `creative.reverbPreset`, `ducking.against`, `ducking.reduceDb`, `ducking.fadeS`, `pan.value`, `volume.levelDb`, `mute`, section switches `
.enabled`), `essentialSound.applyPreset {preset, type?}`, `essentialSound.savePreset {name}`, `essentialSound.deletePreset {name}`, `essentialSound.autoMatch {target?}`, `essentialSound.generateDucking`; all take `clips` (default: the selected audio clips). UI ids: `essentialSound.tab.`, `essentialSound.type.`, `essentialSound.clearType`, `essentialSound.preset[.save|.delete|.name|.ok]`, `essentialSound.section.[.toggle]`, a row per setting key (`essentialSound.repair.noise.on`, `essentialSound.repair.noise.amount`, `essentialSound.repair.humHz.`), `essentialSound.autoMatch`, `essentialSound.ducking.against.`, `essentialSound.generateDucking`, `essentialSound.volume.on`, `essentialSound.volume.levelDb`, `essentialSound.mute`, `essentialSound.browse..`. Project files, auto-save, crash recovery and preferences commands (`file.recover`, `prefs.set`, …) and their automation ids are listed in [project-files.md](project-files.md). That file also lists the media management commands and dialog ids: offline media and relinking (`media.findMissing`, `media.relink`, `media.autoRelink`, `media.search`, `media.makeOffline`, `media.status`, `linkMedia.*`), proxies (`media.createProxies`, `media.attachProxies`, `media.toggleProxies`, `proxies.*`), ingest (`project.ingestSettings`) and the Project Manager (`file.projectManager`, `pm.*`). **Moving clips**: `timeline.move {moves: [{clip, track, time}], insert?, linked?}` → `{moved: [clip ids]}`. With Linked Selection on (the default; `linked` overrides the toggle for one call) the linked partners of a listed clip that are not listed themselves move by the same time offset and stay on their own tracks, so picture and sound stay in sync. Like a listed clip, a partner overwrites what is at its destination (or inserts, with `insert`). A partner on a locked track stays where it is, and so does one whose position would not change (the listed clip only changed track). `moved` lists every clip that moved, partners included. No clip is moved before the sequence start: if one would be, all clips of the call land later by the same amount (their spacing is kept). A `time` beyond the representable range is an "invalid parameters" error. The timeline panel's drag passes `linked: false`, because it already lists exactly the clips it moves. **Trim mode** (`crates/engine/src/trim.rs`): `trim.selectEditPoint` / `trim.selectNearest` enter trim mode (the Program monitor becomes the Trim Monitor, ids `trimMonitor.*`); `trim.monitor` returns what it shows. Dynamic trimming takes an explicit clock (seconds) so it is deterministic: `trim.shuttle {direction, slow?, clock}` (J/L), `trim.tick {clock}`, `trim.shuttleStop {clock?}` (K, one undo step), `trim.cancelDynamic` (Esc), `trim.playAround {clock, loop?}` (Space / Shift+K). **Replace With Clip**: `clip.replaceFromSource`, `clip.replaceFromSourceMatchFrame` and `clip.replaceFromBin {clips?, item?, keepSourceIn?}` swap the media of the clips and return `{clips: [ids], item, short: [{clip, shortByFrames}]}`. `short` lists the replaced clips that now run past the end of their media (the renderer holds the last frame there) with the number of sequence frames that have no media, at most the clip's length; it is empty when the new media covers every clip. The edit is the same either way. From Bin starts each clip at the new item's In point; with `keepSourceIn: true` each clip keeps its own source In instead. A subclip that restricts trims has no media outside its range, so there a kept In is moved to the subclip's first or last frame, and the result's `sourceInClamped: [ids]` (present with `keepSourceIn`) names the clips it was moved for. **Colour** (`crates/engine/src/color.rs`): `sequence.colorSettings {workingSpace: rec709|rec2100-pq| rec2100-hlg, wideGamut, autoToneMap}`, `clip.interpretFootage {items?, colorSpace: auto|}`, `color.spaces`, `media.colorInfo {item}`; LUTs: `lut.import {path, name?}`, `lut.list`, `lut.remove {id}`, `lut.export {lut, path, format?}`, `lumetri.setInputLut` / `lumetri.setLook {clip?, lut: lib:|builtin:|"", path?}`, `lumetri.setSection {clip?, section, on?}`, `lumetri.applyMatch {clip?, referenceTime|referenceFrame|referenceTimecode, faceDetection?}`. Without params the menu entries open dialogs (ids `colorDialog.space.`, `colorDialog.working.`, `colorDialog.wideGamut`, `colorDialog.autoToneMap`, `colorDialog.ok|cancel`). Lumetri panel ids: `lumetri.input_lut`, `lumetri.look_lut`, `lumetri.switch.
`, `lumetri.section.
`, `lumetri.match.*`; HDR scopes: `scopes.hdrWaveform`. `file.exportMedia {sdr: true}` exports an HDR sequence as tone-mapped SDR. **Keyboard shortcuts** (`crates/engine/src/shortcuts.rs`): `shortcuts.list {query?, panel?}`, `shortcuts.get`, `shortcuts.set {command, keys, panel?, add?, keepConflicts?}`, `shortcuts.clear`, `shortcuts.undo` / `shortcuts.redo`, `shortcuts.conflicts {platform?}`, `shortcuts.forKey {key}`, `shortcuts.resolve {keys, panel?}`, `shortcuts.presets`, `shortcuts.loadPreset` / `savePreset` / `deletePreset {name}`, `shortcuts.export` / `import {path}`, `shortcuts.audit`. Keys use `Cmd` (⌘ / Ctrl), `Ctrl` (macOS ⌃), `Alt`, `Shift`. The dialog (Edit ▸ Keyboard Shortcuts…, ⌥⌘K) uses ids `shortcuts.*` (`shortcuts.key.K`, `shortcuts.cell.`, `shortcuts.ok`, …). ## MCP `filmcraft-cli mcp` serves MCP on stdio: headless (in-process session; `--demo` / `--project p.fcproj`) or `--bridge 127.0.0.1:9876` to drive the running app. Tools: `command_list`, `command_run`, `command_batch`, `doc_inspect`, `render_preview`, `project_inspect`, `sequence_inspect`, `media_import`, `render_frame`, `ui_inspect`, `ui_elements`, `ui_click`, `ui_drag`, `ui_key`, `ui_type`, `ui_screenshot`, `ui_control`; resources `filmcraft://document` and `filmcraft://commands`. `.mcp.json` registers both. Annotations, argument checking, errors and export progress / cancellation: [agents.md § Conventions](agents.md#conventions). Time is in ticks (254 016 000 000 per second); commands also accept `seconds`, `frame` or `timecode`.