# Architecture: Clips Kitty Clips Kitty is a local-first AI clipping engine with a desktop front end. It ingests a long video, finds the moments worth posting using an LLM plus multimodal signal analysis, renders speaker-tracked captioned vertical clips, and hands them to a review and editing studio. By default nothing is sent to a cloud AI service and there is no paid inference anywhere in the pipeline; cloud AI is an opt-in on the user's own account ([Cloud AI](#cloud-ai-opt-in)). Opt-in modes change the framing and the scoring for game streams, matches, lives that were already vertical, and 16:9 output. This document describes the system as it is built today. It is the only architecture document: design notes that used to live separately have been folded in here. **Contents** 1. [System overview](#1-system-overview) 2. [Repository structure](#2-repository-structure) 3. [The processing pipeline](#3-the-processing-pipeline) 4. [Clip scoring](#4-clip-scoring) 5. [Face tracking and framing](#5-face-tracking-and-framing) 6. [Creator intelligence](#6-creator-intelligence) 7. [Multilingual publishing](#7-multilingual-publishing) 8. [Long-form output](#8-long-form-output) 9. [The video editor](#9-the-video-editor) 10. [Local API service](#10-local-api-service) 11. [Desktop application](#11-desktop-application) 12. [State and storage](#12-state-and-storage) 13. [Configuration surface](#13-configuration-surface) 14. [Windows packaging](#14-windows-packaging) 15. [Design rules](#15-design-rules) --- ## 1. System overview ``` ┌──────────────────────────┐ ┌──────────────────────────┐ ┌──────────────────────────┐ │ Desktop app │ │ Watched channels, │ │ Assistants │ │ Electron + React + Vite │ │ streamer-tool hand-offs │ │ Ask Clips Kitty · MCP │ └────────────┬─────────────┘ └────────────┬─────────────┘ └────────────┬─────────────┘ └─────────────────────────────┼─────────────────────────────┘ │ HTTP + WebSocket (127.0.0.1:8765) ┌────────────────────────▼────────────────────────┐ │ FastAPI service (server/) │ │ queue · progress events · publisher · webhooks │ └────────────────────────┬────────────────────────┘ │ ┌──────────────────────────────────────────▼───────────────────────────────────────────┐ │ │ │ PIPELINE (core/pipeline.py) │ │ │ │ SOURCE ──► DOWNLOAD ──► TRANSCRIBE ──► ANALYZE ───► RENDER ───────► METADATA │ │ YouTube yt-dlp Whisper, signals + framing by mode titles, │ │ Twitch local, or the LLM, + captions, descriptions, │ │ Kick cloud on the local or here or on a hashtags │ │ local file user's account cloud render PC │ │ │ │ signals: audio · visual · reaction · text · engagement │ │ side channels: creator knowledge · chat replay · heatmaps · game metadata · │ │ score box (OCR) · crowd, whistle and game sounds (PANNs) │ │ framing by mode: speaker tracking (standard) · webcam + game (Gaming / Reaction) │ │ · the whole 9:16 frame (Vertical Live) · the ball (Sports) │ │ · 16:9 (Longform) · per shot (Podcast) │ │ │ └──────────────────────────────────────────┬───────────────────────────────────────────┘ │ ┌─────────────────────┬──────────┴──────────┬─────────────────────┐ ▼ ▼ ▼ ▼ ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ │ SQLite state │ │ Rendered clips │ │ Creator KB │ │ Publishing │ │ core/state.py │ │ data/clips/ │ │ knowledge + │ │ YouTube · │ │ │ │ │ │ events │ │ WoopSocial · │ │ │ │ │ │ │ │ Upload-Post │ └───────────────────┘ └───────────────────┘ └───────────────────┘ └───────────────────┘ ``` By default everything in that diagram runs on the local machine. The only network calls a normal run makes are **yt-dlp fetching the source video** and, for Twitch VODs, an optional chat-replay request. Whisper, the LLM, YOLOv8, and FFmpeg are all local. Three things add network paths, and each is off until the user turns it on: **cloud AI** (the transcript, or the audio for cloud transcription, goes to the provider the user signed in to), **publishing** (finished clips go to the platforms, directly or through a publishing service) and **remote rendering** (one clip's stretch of video goes to the user's own render PC, [10.3](#103-remote-rendering)). Two plain file fetches run on their own, carry nothing about the user, and can each be switched off: the desktop app's update check and Clips Kitty's online list of pipelines (`plugins/registry.py` `ONLINE_URL`, once a day when the Marketplace opens or the app starts with a pipeline installed; `docs/platform/DECISIONS.md` D29). ### End to end The same flow as the README, from a watched channel to a published clip: ```mermaid flowchart TD W["Watched channel
YouTube's RSS feed spots a new video"] --> I I["Video input
YouTube · Twitch VOD · Kick VOD · local file"] --> T T["Transcription
faster-whisper, word-level timestamps
local, or OpenRouter, signed in with OAuth"] --> A A["AI video analysis
audio · visual · reaction signals in 1-second bins"] --> S S["Clip detection and scoring
an LLM scores the transcript with those signals
local (Ollama) by default, or OpenRouter, signed in with OAuth"] --> E E["Video editing
9:16 crop on whoever is speaking (YOLOv8 + TalkNet),
on the ball for a match, webcam + game for Gaming / Reaction"] --> C C["Captions
word-synced, burned in, fully editable"] --> M M["Multilingual
translate, subtitle or dub into 19 languages"] --> X X["Export
organized folders, clean filenames, metadata"] --> P P["Publishing
YouTube, TikTok, Instagram: now, on a schedule,
or hands-off for a watched channel"] ``` ### The AI stack | Job | Tool | Where it runs | |---|---|---| | Download | yt-dlp | local | | Transcription | faster-whisper, word-level timestamps | local, CUDA or CPU; or a cloud Whisper on the user's account | | Clip scoring, titles, translation, edit chat, assistant | any Ollama model (Gemma by default), or a cloud model | local via Ollama; or OpenRouter or a direct provider on the user's account | | Person/pose detection | YOLOv8 (`yolov8n-pose.pt`) + OpenCV | local, CUDA or CPU | | Text on screen (score box, game banners) | RapidOCR | local | | Sound events (crowd, whistle, game sounds) | PANNs (an AudioSet tagger) | local | | Who is talking (the second speaker's caption colour) | pyannote segmentation + WeSpeaker speaker model, on onnxruntime | local, CPU, one core | | Ball detection (Sports) | YOLOv8n, "sports ball" | local | | Rendering | FFmpeg with hardware encoding (NVENC / AMF / QSV) | local | | Dubbing voices | local TTS | local | ### Cloud AI (opt-in) The LLM and transcription can each run in the cloud instead, on the user's own account, chosen in Settings → AI ([docs/AI-BACKENDS.md](docs/AI-BACKENDS.md)). Local stays the default and the first choice. - `llm/providers/catalog.py` lists the providers: OpenRouter as the recommended cloud path, direct APIs (OpenAI, Anthropic, Gemini, xAI, Meta and others) as advanced. An OpenAI-compatible provider is one entry; a new wire format is one adapter in `llm/providers/adapters/`. - Every request goes through `llm/providers/http.py`: the key in a header, plain-English errors, bounded retries. A failing cloud model fails the job; it never falls back to another provider or to local AI behind the user's back. - OpenRouter can be signed in to instead of pasting a key: OAuth with PKCE (`llm/providers/oauth.py`), with the engine on localhost receiving the callback. The key it returns is the user's own and lives in the app's credential store (`llm/providers/keys.py`). - OpenRouter requests carry its app-attribution headers (`llm/providers/openrouter.py`), so the usage counts toward OpenRouter's public app rankings. - Cloud transcription (`transcription/cloud.py`) writes the same transcript cache as local Whisper, so every later stage is unchanged. --- ## 2. Repository structure ``` clips-studio/ ├── main.py # CLI entry: process, serve, models, status, channels ├── config/ │ ├── settings.yaml # quick setup at the top, advanced below │ ├── prompts/ # every LLM prompt as an editable text file │ ├── gaming.yaml # kinds of game and what a highlight is in each │ └── sports.yaml # each sport's moments, windows and commentary words ├── core/ │ ├── pipeline.py # stage orchestration for one video │ ├── state.py # SQLite schema + all queries │ ├── models.py # dataclasses shared between stages │ ├── progress.py # stage events → WebSocket │ ├── cancel.py # cooperative cancellation flags │ ├── modes.py # Vertical Live, Gaming / Reaction and Sports switches │ ├── preflight.py # can this install make a clip? (setup checks) │ ├── binaries.py # bundled FFmpeg, Ollama and model weights │ ├── queue.py # queue manager: order, pause, retry, estimate │ ├── prefetch.py # download-ahead for queued jobs │ ├── housekeeping.py # disk reclamation │ └── scheduler.py # (legacy CLI) `main.py run` poll loop; the app uses server/automation.py ├── sources/ # ── one file per platform ── │ ├── dispatch.py # URL → the right source module │ ├── youtube.py # yt-dlp download, H.264 selection, heatmap │ ├── twitch.py # VOD download + chat replay │ ├── kick.py # VOD download │ ├── channel_feed.py # what a watched channel posted (RSS, yt-dlp, Kick API) │ ├── vod_finder.py # the VOD a finished livestream left behind │ └── ytdlp_common.py # chunked, resumable download shared by all ├── transcription/ │ ├── transcriber.py # faster-whisper, word timestamps, GPU/CPU fallback │ └── cloud.py # transcription on the user's own provider account ├── llm/ # ── swappable model layer ── │ ├── base.py # LLMBackend interface │ ├── ollama_backend.py # anything Ollama serves │ ├── registry.py # config string → backend instance │ ├── manager.py # install/switch/recommend models │ ├── spec.py # which provider and model a backend spec names │ └── providers/ # cloud AI on the user's own account: catalog, adapters, │ # http, keys, OAuth sign-in, OpenRouter attribution ├── analysis/ # ── the clip engine ── │ ├── audio_features.py # loudness, spikes, burst density, laughter proxy │ ├── visual_features.py # scene cuts, motion, flashes, face metrics │ ├── hype.py # Twitch chat replay + YouTube most-replayed │ ├── gaming.py # scores a gaming stream as a gaming stream │ ├── game_audio.py # what the game sounds like │ ├── game_text.py # what the game writes on screen │ ├── game_vision.py # the AI looks at a gaming stream's best candidates │ ├── chat_moments.py # what chat's reactions say happened │ ├── panns.py # PANNs sound tagger: game sounds, crowd, whistle │ ├── voice_turns.py # whose voice it is: the turns of anyone but the main speaker │ ├── intent.py # clip direction: what the clips should be about │ ├── fusion.py # candidate generation, weighted scoring, rerank │ ├── highlights.py # LLM transcript scoring + duplicate prevention │ ├── metadata.py # titles, descriptions, hashtags │ ├── clip_edit.py # AI edit chat → validated edit operations │ └── tighten.py # silence and filler-word removal ├── creator/ # ── creator intelligence ── │ ├── identity.py # channel → creator profile resolution, merging │ ├── extractor.py # transcript → structured facts and events │ ├── retrieval.py # stored knowledge → scoring context │ ├── learning.py # preference learning from user actions │ └── models.py # knowledge types, thresholds ├── video/ # ── rendering ── │ ├── tracker.py # YOLOv8 pose tracking, speaker-aware │ ├── asd.py # TalkNet active-speaker detection │ ├── framing.py # crop-path smoothing and shot logic │ ├── cuts.py # frame-exact cuts for shot-aware framing (§5) │ ├── podcast.py # multi-cam shot detection, per-shot framing │ ├── cropper.py # 9:16 render │ ├── captions.py # word-synced ASS captions │ ├── cutter.py # accurate clip extraction │ ├── filters.py # colour presets and adjustments │ └── encoding.py # encoder detection and selection ├── video_editor/ # ── manual editing applied to a clip ── │ ├── timeline.py # non-destructive edit model │ ├── cuts.py # trims and internal cuts │ ├── audio.py # volume, fades, mutes, music bed │ ├── captions.py # caption edits │ ├── overlay.py # hook text │ ├── watermark.py # branding profiles │ └── export.py # final render ├── gaming/ # ── Gaming / Reaction (opt-in) ── │ ├── detect.py # who the streamer is, where the webcam sits │ ├── panels.py # chat boxes and timers the game crop keeps out │ ├── layout.py # where each part of a layout comes from and goes │ ├── framing.py # the eleven layouts (data in gaming/layouts.json) │ ├── compose.py # the FFmpeg render for a layout, one encode │ └── run.py # the two calls the pipeline makes ├── sports/ # ── Sports (opt-in), one package per sport ── │ ├── __init__.py # the registry: options, profile, prepass, framing │ ├── core/ # evidence, moment windows, selection, match events, story reels │ └── soccer/ # score box (OCR), ball framing, the soccer profile ├── longform/ # ── 16:9 outputs ── │ ├── profiles.py # short_clips / clips_140 / highlights / edited_stream │ ├── highlight_select.py # best-of selection across a whole video │ ├── downtime.py # dead-air detection │ ├── assemble.py # multi-segment assembly │ └── process.py # orchestration ├── multilingual/ # ── translated publishing ── │ ├── languages.py # supported languages and voices │ ├── translate.py # LLM translation with glossary protection │ ├── glossary.py # protected terms per creator │ ├── subtitles.py # .srt writing │ ├── burn.py # burned translated captions │ ├── dub.py # TTS dubbing │ ├── voices.py # voice catalogue and auditioning │ ├── metadata.py # translated titles/descriptions │ └── publish.py # export bundles ├── server/ # ── local API ── │ ├── api.py # FastAPI routes │ ├── jobs.py # SQLite-backed worker │ ├── events.py # WebSocket broadcasting │ ├── feedback.py # in-app bug reports + diagnostics │ ├── ai_api.py # Settings → AI: local or cloud, which model │ ├── automation.py # watched channels: new video → clips → publish │ ├── agent.py # the Ask Clips Kitty assistant │ ├── mcp.py # the same tools over MCP (stdio) │ ├── integrations.py # streamer tools hand over a finished livestream │ ├── webhooks.py # tell another program a job finished │ └── publisher.py # upload thread; *_service.py / *_api.py per provider ├── publish/ # YouTube Data API, WoopSocial, Upload-Post; metadata, scheduling, quota ├── remote_render/ # a second PC renders clips: service, TLS gateway, worker, protocol ├── third_party/talknet/ # vendored TalkNet-ASD — do not edit ├── models/ # TalkNet weights (pretrain_TalkSet.model), PANNs game sounds, the two voice models ├── ui/ # ── desktop app ── │ ├── src/main/ # Electron main process │ └── src/renderer/ # React + TypeScript + Tailwind ├── feedback-relay/ # Cloudflare Worker → GitHub Issues └── data/ # runtime artifacts, gitignored ``` --- ## 3. The processing pipeline One video moves through `core/pipeline.py` as a state machine. Every transition is committed to SQLite **before** the next stage begins, so a crash resumes at the failed stage instead of redoing (or re-uploading) completed work. | Stage | Producer → consumer | Artifact | Status | |---|---|---|---| | 1 | `sources/*` → disk | `data/downloads/{id}.mp4` | `downloaded` | | 2 | `transcription/` → disk | segments + word timestamps | `transcribed` | | 3 | `analysis/` ↔ LLM | `ClipCandidate[]` with subscores | `analyzed` | | 4 | framing by mode: `video/tracker` → `video/cropper` (the speaker), `gaming/` (webcam + game), `sports/soccer/ball.py` (the ball), `video/podcast.py` (per shot); a Vertical Live is one whole-frame encode | crop path or layout per clip |: | | 5 | render + captions → disk | `data/clips/{id}/clip_{n}.mp4` | `rendered` | | 6 | `analysis/metadata` ↔ LLM | title, description, hashtags | `done` | Three things deliberately run **concurrently** with others to keep the GPU busy: - **Signal extraction** (audio + visual features) needs no transcript, so it runs in a background thread *during* transcription. FFmpeg and numpy work while Whisper holds the GPU. If it fails, analysis recomputes it and reports the error properly. - **Creator knowledge extraction** runs during the render stage, when Ollama is otherwise idle. It never affects the current video's clips; it only informs future ones. - **Download prefetch** (`core/prefetch.py`) fetches the *next* queued video while the current one is being processed: pure network against CPU/GPU work, so they overlap perfectly. One slot only, so disk and bandwidth stay bounded, and best-effort: any failure is ignored and the job downloads normally. A job waits for a prefetch of its own video before starting, but **with a timeout**. A bare join meant one stalled download wedged the single worker thread and silently stopped the whole queue. Cancellation is cooperative: `core/cancel.py` sets a flag that every long stage checks at safe points, so a cancelled job stops promptly without corrupting state. FFmpeg renders already in flight are the exception: no process handle is retained, so a cancel lands at the next clip boundary rather than instantly. ### 3.1 Where the time goes Processing is dominated by decoding frames and by moving them between FFmpeg, OpenCV and the models: not by the models themselves. Three decisions follow from that, all measured rather than assumed: - **Frames are read through FFmpeg with `-hwaccel`, not `cv2.VideoCapture`.** A sampling pass that cost ~10 CPU-seconds across 6.2 cores costs ~0.9 across 0.3 this way. The same reader is used for rendering. - **Slow-decoding sources are converted once, up front.** AV1, VP9 and HEVC decode in software and every later stage decodes the file again; one H.264 conversion at the start is cheaper than paying that per clip. YouTube downloads prefer H.264 for the same reason. AV1 roughly doubled processing time. - **CPU is divided deliberately.** `_share_the_cpu()` splits cores across render workers and holds two back, so detection, rendering and the interface don't fight. It latches on the first video of a process run. --- ## 4. Clip scoring The scoring engine treats the transcript as **one signal among several**. A moment that reads flat in text but where the room explodes is exactly the moment a transcript-only scorer misses. ### 4.1 Signals Every signal is computed over the whole video in one-second bins, then normalized to 0–1 **by percentile rank within that video**: "how unusual is this second *for this video*", so a quiet podcast and a screaming stream both produce meaningful peaks. **Audio** (`analysis/audio_features.py`, FFmpeg → numpy): - RMS loudness envelope and spike score against a rolling median (shouts, cheers, hype) - Burst density: rapid onset clusters, which is what laughter and applause look like - High-band energy ratio and zero-crossing rate as a laughter/cheering proxy - Silence→explosion transitions, the classic payoff shape **Visual** (`analysis/visual_features.py`, OpenCV at low sample rates): - Scene cuts via HSV histogram distance - Motion intensity via mean absolute frame difference - Brightness and flash events - Face metrics from YOLOv8: count, max area, and area delta (a sudden lean-in or zoom is an editor or streamer emphasizing something) **Reaction**. A fusion of face presence, face-area delta, motion near the face, and audio excitement at the same instant. This is an honest proxy, not true facial expression recognition; the extractor interface is built so an open-weights expression or laughter classifier can drop in later without touching fusion. **Text and engagement** come from the LLM: what is said, and its judgment of hook strength, payoff, and quotability. **Audience data** (`analysis/hype.py`), where it exists. Twitch chat replay is measured by *unique chatters per window*, so gifted-sub spam and copypasta don't inflate a moment. YouTube's most-replayed heatmap is read when available. Kick keeps no chat after a stream ends, so Kick has no chat signal. This contributes a small capped bonus (`scoring.audience_bonus`), kept below the content signals on purpose. ### 4.2 Candidate generation Two independent pools feed the same scorer: 1. **Transcript candidates**: the LLM's picks from chunked transcript analysis. 2. **Signal-peak candidates**: windows where the combined non-text signal exceeds `scoring.signal_peak_percentile`, snapped to sentence boundaries and duration enforced. This is what catches the laugh or the clutch moment that the transcript describes blandly or not at all. **Selection 2** (`scoring.selection`; on by default, and `scoring.selection: 1` turns it off, which is exactly the behaviour above) changes four things on a standard run, never on a run scored as gaming (Gaming / Reaction, or Vertical Live's Gaming / reaction choice) or a Sports match (`analysis/fusion.py` decides): - Signal-peak and clip-direction windows are scored eight at a time, each answer matched to its window by time, and a window left at the neutral 50 is reported. One request for every window ran past a local model's output budget, so all of them fell back to 50. - Each signal pass's window budget goes to its strongest peaks, not its earliest, so a long video's late peaks can become candidates. - Only a transcript pick at or above `clips.min_score` hides a signal peak it overlaps. - A clip's start never moves past the line it was picked on, or the line its hook quotes. When that line carries on a sentence (the line before it has no sentence end, and the pause is at most 2 s), the start goes back to the sentence's first line, if that is at most 8 s earlier and the clip stays within `clips.max_duration`. In a transcript with no sentence ends at all, this rule moves no start. A signal window gets up to 6 s of complete sentences before it. The end rules are unchanged. Longform keeps selection 1 unless `scoring.longform_selection: 2`, whatever `scoring.selection` says: `longform/process.py` pins it on Longform's own copy of the settings, so that copy never takes the Shorts' default. The constants, and why each change was made, are at the top of `analysis/highlights.py`. ### 4.3 Fusion ``` final = 0.30·text + 0.20·visual + 0.20·reaction + 0.20·audio + 0.10·engagement ``` Weights live in `settings.yaml` under `scoring.weights` and are tunable per content type. On top of the weighted score sit three **additive, capped, deterministic** bonuses. Each can only raise a score, never lower one, so no amount of accumulated data can degrade clip quality: | Bonus | Cap | What earns it | |---|---|---| | `action_bonus` | 10 | A person on screen and moving: workouts, sports, dance. Content that performs socially but that a text-first scorer under-credits. | | `audience_bonus` | 8 | Real audience reaction at that timestamp (chat speed, heatmap). | | `creator_context_max` | 6 | Verifiable callbacks to what the app knows about this creator (§6). | Each is individually disableable by setting it to `0`. ### 4.4 The LLM's role, and working around small models The transcript is chunked (`analysis.chunk_seconds`, sized so a 7B model's context has room to think) with overlap so boundary moments are never lost, and each chunk's prompt carries a compact **events timeline** derived from the signals: ``` TRANSCRIPT: [244.1 - 247.9] dude no way he actually hit that ... EVENTS: [244s] AUDIO spike (99th pct, burst cluster — laughter/cheering likely) [245s] SCENE CUT + high motion [251s] face zoom-in ``` A local model can't watch pixels, but it reasons perfectly well over a fused text description of what the audio and video are doing. Local models are also less reliable at structured output than frontier models, so `analysis/highlights.py` owns the resilience rather than the backend: - Tolerant JSON extraction: strip code fences, find the first balanced `{…}` block. - On parse failure, one retry with a "return only valid JSON" reminder; on a second failure the chunk is skipped and logged, never crashing the run. - Small models score everything in a narrow band, so a **rerank pass** compares the top `scoring.rerank_pool` finalists head-to-head in a single prompt. Relative judgment is much easier for a small model than absolute scoring, and it costs one call per video. ### 4.5 Duplicate prevention Applied highest-score-first, three independent checks, each with a logged reason: 1. **Timestamp overlap**: reject if more than `analysis.max_overlap` of the shorter clip overlaps a kept clip. 2. **Transcript similarity**: reject if spoken text is more than `analysis.max_text_similarity` similar to a kept clip. 3. **Segment reuse**: reject if more than `analysis.max_segment_reuse` of the candidate's transcript segments are already claimed. Then timestamps are validated against the transcript range (dropping hallucinated ranges), snapped to sentence boundaries, and duration-enforced. ### 4.6 The model abstraction ``` LLMBackend (llm/base.py) generate(prompt: str, *, json_mode: bool = False) -> str name -> str ``` `registry.py` maps a config string to a backend. Nothing in `analysis/`, `creator/`, or `multilingual/` imports a concrete backend, so a model swap is one line of YAML and a future cloud backend is one new file. Translation may use a *different* local model than clipping, since multilingual strength and editorial judgment are different strengths. ### 4.7 Game streams and matches Two opt-in profiles change what counts as a moment. Neither touches standard scoring when it is off. - **Gaming** (`analysis/gaming.py`, [docs/GAMING.md](docs/GAMING.md)): the game, from the platform's metadata (about 150 games mapped to a kind of game in `config/gaming.yaml`); chat's reactions (`analysis/chat_moments.py`); sudden jumps in the streamer's voice; game sounds from the PANNs tagger (`analysis/panns.py`); text the game puts on screen. The prompt is told it is a gaming stream and which game. Scoring stays led by what is said: the game adds a capped bonus, like creator context. - **Sports** (`sports/`, [docs/SPORTS.md](docs/SPORTS.md)): the score box read with RapidOCR is ground truth (a goal is a score that changed and stayed changed), the crowd and the whistle come from PANNs, and the commentary from the transcript. Each moment type has its own window (a goal: 14 s before, 10 s after), the Highlights choice selects which types become clips, and match events the user gives are certain moments. --- ## 5. Face tracking and framing `video/tracker.py` takes a source video and a clip window and returns a crop path. It knows nothing about transcripts, scores, or publishing. 1. **Sample** frames at `tracking.sample_fps`: subjects don't teleport between samples. 2. **Detect** with a YOLOv8 **pose** model. Head keypoints (nose, eyes, ears) give head-priority framing even when no face is cleanly visible, which plain person boxes can't do. The Haar cascade is only run when pose found no head: pose succeeds about 78% of the time, and the cascade was 63% of the tracking pass's cost. 3. **Select the subject** per frame by confidence × box area × persistence with the previous choice, so tracking stays locked on the streamer when guests or bystanders appear. 4. **Speaker awareness**: see §5.1. 5. **Smooth** (`video/framing.py`) with an exponential moving average, a dead zone that ignores movements under a few percent of frame width, and a maximum pan speed. This is what removes jitter and the "drunk camera" effect. 6. **Fall back** to a static centre crop when there are no detections. **Shot-aware framing** (`tracking.shot_aware`; on by default, and `tracking.shot_aware: false` turns it off, which is exactly the behaviour above). With it off, cuts are looked for only on the sample grid, at a threshold many real cuts never reach, and are placed on the sample after the cut. A cut also only resets the challenger: the subject on camera stays the target, at their old position, until they have been unseen for a second, so the new shot opens framed on someone who is not in it. And a clip opens on its first sample's centre, the body box, and eases to the head over about a second. With it on: - **Cuts are found to the frame** (`video/cuts.py`). A sample pair whose thumbnails differ by 18 or more (with it off, the bar is 55) is a candidate. Only the frames around it are decoded, with a capture of their own, and a frame is a cut by the hard-cut rules of `sports/basketball/crop/shots.py`: a gray change, a colour change, and standing out from the motion around it, with camera flashes rejected. A handheld pan changes the picture a lot but not its colours, so it is no cut. - **The subject is chosen again at a cut** when the one on camera is not in the new shot (who the shot plan says is talking, else the most prominent). A subject still on screen keeps the camera. The framing is the median of the subject's position over the new shot's first 0.5 s, taken from the new shot only. The old framing holds to the frame before the cut, and the new one starts on the cut's own frame. - **A shot with nobody detected at first** opens on the centre, or keeps the old framing with `tracking.shot_aware_no_subject: hold` (default `center`), and cuts to the first credible person who appears: detector confidence 0.6 or more, with a pose head or a box at least half the frame's height. Anything fainter seen meanwhile (on road-trip footage: a burning car, a cartoon, a dashboard, a rear-view mirror) leaves the shot as it opened. While the shot plan's speaker is on screen below that bar, the shot waits even for someone credible, because the plan would take the camera straight to its speaker on the next sample. Letting the plan's speaker through the bar instead would have opened two road-trip shots on a cartoon and a dashboard, each handed across the cut the id of somebody the plan names (a third opens on a tray of food under either rule: waiting puts it off one sample, and then the tray clears the bar). The cost: a real speaker the detector only finds faintly leaves such a shot on the centre (or the old framing, with `hold`) until they clear the bar or leave. The rule only sees the plan's speaker on the current sample: if the detector misses a faint speaker the plan names on exactly the sample where somebody credible first appears, the shot opens on that person, and the plan's speaker can take the camera on the next sample, a one-sample shot. That never happened on the footage the mode was validated on. A first shot with nobody in it keeps the centre up to the cut: no shot's framing reaches across a cut into another. - A track's first sighting in a shot, from the clip's first sample on and again after each cut, starts its averages (head offset, width, prominence) from that sighting. A pose head is trusted from it. A Haar face counts as one hit, as on a new track, so a single face found on a hand or a bottle cannot move the crop. That is a deliberate trade-off against single false faces: a real face only the cascade finds is trusted just as gradually, so the shot opens between the body and the face, eases back toward the body, and reaches the face about a second after the cut (or the clip's start). - **The clip's opening** is framed like a shot after a cut, without the keys: on the first sample anyone is seen on, the framing is the median of the subject's position over the next 0.5 s, and there is nothing before it to hold or snap from. So a clip opens on the head the pose model finds, where it used to open on the body box and ease to the head. An off first detection without a head no longer sets the opening (one with a pose head is trusted, as after a cut). A first shot with nobody in it still keeps the centre to the cut. - The "subject gone" test reads when each track was last seen as of the sample being framed, not at the end of the clip. The exception is the speaker the shot plan names when their track is seen again within 2 s, before the next cut: a faint speaker the detector loses for a moment on handheld footage keeps the camera, instead of losing it to someone else and taking it back a moment later. A sighting after the next cut does not count, since the id can belong to someone else by then. TalkNet's shot plan is computed as before. The whole-clip decisions (letterbox, facecam layout, `face_y`) use the same rules, but with the setting on they read the subject the shot-aware pass ended on (for the automatic letterbox, the subject on each sample) and the averages started afresh in each shot, so their result can differ. `tests/test_tracker_cuts.py` checks that with the setting off, the output is exactly what it was. ### 5.1 Who the camera follows **TalkNet active-speaker detection** (`video/asd.py`, model vendored in `third_party/talknet/`) decides this, and it is the *only* thing that decides it. It scores a face crop against the audio at 25fps, so it answers "is this person speaking" rather than "is this person moving". Mouth-region pixel movement is still measured, but **only as a prominence tiebreak, and it must never be used as a speaker detector again**: it measures movement, and two people in a lively room move about equally, so it reliably picks whoever is *bigger*. That was the bug it replaced. Cost is in getting frames to the model, not the model. Two things keep it affordable, both measured on an RTX 3060 and both easy to "simplify" into a 452% regression: - Boxes already found by the tracker are reused and interpolated onto TalkNet's 25fps grid. Re-detecting at 25fps instead costs +452%; reusing costs about +15%. - Only **contested spans** are scored: stretches where two or more tracked faces are on screen at once. When one person is alone there is nothing to decide. - Scoring is windowed (1/2/4/6s). Handed a whole clip at once the model reports nobody speaking, for every frame. The ASD crop is **square on purpose**; proportioning it like a face changed the speaker verdict on three of four bench clips. When the speaker changes, the framing **cuts**. It does not pan across. Panning between two people reads as a camera hunting for someone; cutting is what a human editor does. Two people talking are cut between rather than held in one zoomed-out shot. If the weights are absent or scoring fails, the render falls back to motion-based prominence rather than failing. A bad clip must never fail a render. **Podcast mode** (`video/podcast.py`) is a separate, opt-in path for multi-camera footage. It finds the camera's cuts to the frame (`video/cuts.py`, as shot-aware tracking does), then frames each shot on its own: one static crop, punched in tight on **one** person, the one TalkNet hears talking in that shot. In a long shot, a new speaker with a second or more of speech gets the crop from the first frame of their turn. Every change is a one-frame snap: no panning, no split screens, and no short shot slid across. Applying this path to single-camera footage would be strictly worse, hence opt-in. **The output contract is non-negotiable**: crop windows are exactly 9:16, only repositioned and never reshaped, then uniformly scaled to 1080×1920. Distortion is impossible by construction. Captions are generated as ASS subtitles from word-level Whisper timestamps and burned in during the same FFmpeg pass as the crop. One encode, not two. With **Second speaker in another colour** ticked in the caption style, the render first asks `analysis/voice_turns.py` where in the clip someone other than the video's main speaker talks. A segmentation model marks who is talking when, ten seconds at a time; a speaker model says whose voice each of those is. The main speaker is the voice heard throughout the video, found once per video and kept in `data/voice_profiles/`, so the same person keeps the same colour in every clip. Lines end where the speaker changes, the other speaker's take the second colour, and the turns are saved with the clip (`speaker_turns`) for the editor's preview. A clip with one voice, or one where it is not clear there are two, is captioned exactly as without the option. What it hears wrong is put right in the editor (**Fix speakers**, Captions tab). A fix is a statement, saved beside what was heard as `speaker_edits`: `[[start, end, 0 or 1], ...]` in clip seconds, 0 for the main speaker. `video/captions.py` `paint_turns` lays the statements over the heard turns at every render, later ones winning, and the result is captioned like any other turns; `speaker_turns` stays what was heard, so the editor can always go back to it. A clip whose caption text was saved keeps its lines, and `tag_lines` colours each one whole by who says the words in it, so there a fix switches a whole caption. The editor works out the same thing for its instant preview in `ui/src/renderer/src/lib/speakerTurns.ts`, on the same numbers (`clip_words`, which is also what `GET /clips/{id}/words` returns). What a click, a shift-click and Swap add to the list is worked out there too (`fixSpeakers`), and `tests/test_ui_speaker_turns_sync.py` runs that file under Node: against the Python, and clicked through word after word. ### 5.2 Gaming / Reaction layouts With Gaming / Reaction on, a clip is not a face crop: `gaming/` lays out the streamer's webcam and the game (or the video being reacted to) together in one of eleven layouts (data in `gaming/layouts.json`), chosen before processing on cards that show the video's own frames. `gaming/detect.py` finds the streamer's webcam, and people inside the game are never taken for the streamer. `gaming/compose.py` renders a layout in one FFmpeg encode with no frames through Python. If anything fails, that clip is made the standard way. VTubers aren't supported: the detection is for people on camera. ### 5.3 Vertical Live A live already composed as 9:16 (`core/modes.py`, [docs/VERTICAL-LIVE.md](docs/VERTICAL-LIVE.md)) skips everything that decides where to point the frame: no face tracking, no TalkNet, no layout decisions. Each clip is one encode of the whole frame at 1080×1920. Finding the moments is unchanged, and its content setting (Talking / IRL, Gaming / reaction or Soccer) picks the scoring profile. ### 5.4 Sports: following the ball A 16:9 match cropped to 9:16 keeps a third of the picture, so `sports/soccer/ball.py` frames on the ball with the YOLOv8n detector the app already ships, the way automatic match cameras do. A match that is already vertical keeps its frame, as a Vertical Live does. --- ## 6. Creator intelligence Sitting above the video layer: a **creator profile** is a person or group, **platform accounts** are their channels, and a knowledge base holds structured facts extracted from their processed videos. Everything here is optional and failure-safe. A video with no resolved creator processes exactly as before. - **Identity** (`identity.py`) resolves a channel to a profile, creating one on first sight, and suggests merges when the same creator appears on two platforms. - **Extraction** (`extractor.py`) runs after analysis and writes typed facts (topic, game, series, catchphrase, joke, collaborator, format, life detail) and events (announced / in progress / completed). It is deliberately paranoid: anything failing validation is dropped, because a small clean knowledge base beats a large noisy one. Extraction is two-stage (free-form notes first, structuring second) because local models have poor recall on typed extraction from messy stream banter. - **Repetition and dropout.** Claims about repetition are verified, not trusted: a catchphrase is counted in the actual transcript and stays an unscored candidate until it has genuinely repeated. Re-hearing a known fact reinforces it; knowledge that stops being said goes dormant and stops influencing anything, and a one-off that never returns is eventually forgotten. Dormancy requires both elapsed time *and* a number of that creator's videos to have passed, so processing infrequently doesn't decay a knowledge base off the calendar alone. - **Retrieval** (`retrieval.py`) feeds two consumers: a **deterministic, capped, additive-only** score bonus for verifiable callbacks in a clip, and a short context block the metadata LLM may use for accuracy. String matching only: no LLM judgment can move a score here. - **Preference learning** (`learning.py`) derives a bounded bias toward what the user actually keeps, from their own exports and edits. It is inert below a minimum number of signals, each weight may shift at most 20%, and weights are renormalized so the total influence budget never changes. --- ## 7. Multilingual publishing Translation of a finished clip into any of 19 languages. The interface is localized into the same 19, so the app is usable in every language it can publish in. The order of operations matters: captions are **translated, shown for review, and only then** written to `.srt` or burned into video. A creator can read and fix a bad line before it becomes permanent, and human-edited text is marked so a later re-translation never overwrites it. - `translate.py`: LLM translation, chunked, with a glossary that protects terms the creator has ruled on (channel names, sponsors, in-jokes) from being translated. - `subtitles.py` / `burn.py`: sidecar `.srt`, or burned captions using a font that actually has the target script's glyphs. - `dub.py` / `voices.py`: local TTS dubbing with auditionable voices. Each voice sample is spoken *in the language being auditioned*, since hearing English in a Turkish voice tells you nothing about a Turkish dub. - `metadata.py`: translated titles, descriptions, and hashtags. --- ## 8. Long-form output An opt-in 16:9 path (`longform/`) using the same analysis, with four profiles: | Profile | Output | |---|---| | `short_clips` | Horizontal clips up to 60s | | `clips_140` | Up to 140s, sized for X/Twitter | | `highlights` | A best-of assembly, 8–20 minutes depending on how much clears the bar | | `edited_stream` | The full stream with dead air removed | The vertical Shorts workflow is untouched by any of this. **Match story reels.** For a match, `sports/core/reels.py` joins the match's clips into longer videos when asked: a match recap in match order, a reel per team, and a reel per player named in two moments or more. They are joined from the clips already rendered, so they take seconds, with one end card and the moments listed in the description; with Longform they come in 16:9 too. --- ## 9. The video editor `video_editor/` applies **non-destructive** edits to a rendered clip. The edit model is a timeline of operations. Keep ranges, mutes, muted words, volume, fades, speed, hook text, music bed, watermark: stored as data and applied at render time, so any edit can be revised or undone by editing the operation rather than re-cutting a file. `analysis/clip_edit.py` is the AI edit chat: plain-language requests ("make it five seconds longer", "the caption says gost, it should say ghost") are interpreted into *validated* edit operations. The LLM proposes; deterministic code validates and applies. `analysis/tighten.py` removes silences and filler words on request. --- ## 10. Local API service FastAPI + uvicorn on `127.0.0.1:8765`, bound to localhost only. The server wraps the pipeline; it does not fork its logic. The list below is the whole surface, for orientation. **[docs/API.md](docs/API.md)** is the document for building against it: which of these are supported rather than internal, what the requests and responses actually look like, and working examples. ``` POST /jobs queue a video (URL or local file) GET /jobs queue state POST /jobs/batch queue several links at once PATCH /jobs/{id} change one WAITING video's settings DELETE /jobs/{id} drop a waiting job POST /jobs/{id}/move reorder ({delta} or {to: top|bottom}) POST /jobs/{id}/retry re-queue a failed video, same settings GET /jobs/{id}/log that run's log file GET /queue grouped queue + pause state + estimate POST /queue/pause /queue/resume stop/start claiming new work POST /queue/clear clear completed | failed | queued | all POST /cancel cancel the running job WS /ws live stage/progress events GET /videos processed videos GET /videos/{id}/clips clips with metadata and subscores PATCH /clips/{id} edit title/description/hashtags GET /clips/{id}/captions caption lines PUT /clips/{id}/captions save edited captions (re-renders) POST /clips/{id}/render re-render with new range or options POST /clips/{id}/tighten silence/filler removal plan POST /clips/{id}/ai-edit plain-language edit request POST /clips/{id}/export export with the final filename POST /export/batch export many GET /media/{clip_id} serve a clip for preview GET /creators profiles + merge suggestions GET /creators/{id} knowledge, events, preferences DELETE /creators/{id}/knowledge forget unconfirmed and dormant facts GET /languages supported languages and voices POST /translate translate or export a language bundle GET /models installed, active, VRAM estimates POST /models/pull streamed download progress POST /models/activate switch model GET /system/stats CPU, RAM, GPU, VRAM, disk GET /settings PATCH /settings settings.yaml as JSON POST /feedback/submit in-app bug report POST /integrations/streams hand over a finished livestream (OBS plugin) GET /integrations/streams/{id} its state: waiting for VOD, queued, progress, clips GET /integrations/presets named option bundles for integrations GET /automation PATCH ... watched channels: on/off, interval GET /automation/watches channels watched, their settings, last check GET /automation/items what each channel posted, read live from its job POST /automation/items/{id}/publish publish (or retry) with the watch's settings ``` Integration mechanics: - **One worker thread** processes jobs sequentially. GPU and CPU contention make parallel video jobs pointless on consumer hardware. The queue lives in SQLite so it survives restarts. Jobs left `running` by a crash are re-queued and flagged `interrupted`; it is *videos* left mid-pipeline that are marked failed (`recover_stuck_videos`), so nothing appears stuck forever. - Pipeline stages emit progress callbacks that broadcast over the WebSocket, so the UI can show real stage progress and a time estimate. - **Streamer integrations** (`server/integrations.py`) hold the logic an OBS plugin would otherwise have to rebuild: - **One `streams` row per stream.** The integration's session id makes a repeated request a no-op. - **A low-frequency `StreamWatcher` thread.** It finds the VOD with `sources/vod_finder.py` and queues it once, starting the queue only when nothing else is waiting. - **A progress snapshot kept per running job.** A dock that connects mid-run sees the same percentage and time left as the app. - **Watched channels** (`server/automation.py`) are built the same way: - **Detection only in `sources/channel_feed.py`.** It turns every platform's listing into the same `NewSourceVideo`, with the id a pasted link would get. - **A `ChannelWatcher` thread** moves each video along one step per tick: seen, ready (not live, not a Short), queued once through `queue.enqueue_once`, then published, asked about, or left alone. Every step is persisted before the next, so a restart carries on. - **One video, one job.** `watch_items.video_id` is unique across all watches, and the queue and library are checked before anything is added. - **Publishing never repeats.** `publish_clips(once=True)` skips anything sent or on its way, records each send before the request, and `reconcile_sending` settles an interrupted one by finding the post that holds its uploaded media. - Off until switched on, and it only runs while the backend does. With the opt-in "keep watching" setting, closing the window hides it in the tray instead of quitting. - Electron spawns the backend as a child process, health-checks `GET /health`, and kills it on exit. In development they run separately. ### 10.1 The processing queue A video takes about an hour, so the queue exists to let someone start a batch and walk away. It is an orchestration layer around the pipeline, not a second copy of it: ``` Queue Manager (core/queue.py) -> Queue Item a `jobs` row -> Processing Config jobs.payload, a JSON snapshot -> Video Pipeline core.pipeline.process_video -> Result / Error job status + error + data/logs/job_N.log ``` `core/queue.py` knows only about a `StateDB`: no FastAPI, no React. The API translates HTTP to those calls and the worker asks it whether it may claim work, so queue rules never leak into either end. Rules worth knowing before changing this: - **Order is `position`, not `id`.** The user can promote a video past ones queued earlier, so insertion order stopped being run order. Reordering swaps two positions rather than reindexing: two writes, nothing to corrupt if the worker claims a job at the same moment. `id` breaks ties. Do **not** add a `UNIQUE` index on `position`. It would make the swap fail halfway. - **Each job owns its settings.** `jobs.payload` is a snapshot taken at enqueue, so ten queued videos can each have their own captions / 60s+ / longform / podcast / watermark choices. It is editable only while a job is `queued`: a configuration that changed mid-run would render some of a video's clips one way and the rest another. The worker **always deep-copies the config** before applying a payload. It holds one config dict for the process lifetime, so any mutation would otherwise outlive the job that made it and land on a later video. - **Pause is durable and non-destructive.** It lives in `app_state`, so stopping the queue survives a restart, and it only stops *claiming*. The running video finishes rather than throwing away an hour of GPU work. Prefetch is gated on it too, so a paused queue stops using bandwidth and disk as well. - **Retry reuses the row**, sending it to the back of the queue with its settings intact. The queue is a work list, not a ledger: a retried video should appear once, waiting, not as a failure sitting beside its own retry. `attempts` keeps the count, and the log file is keyed by job id so a second attempt appends to the same file. - **A failure is contained.** One bad video is marked failed with its error and the loop moves on; a batch left running overnight must not stop at the first dead link. - **Short jobs jump the queue.** Re-renders and translations are seconds-to-minutes of work the user is sitting and waiting for, so they take the front of the waiting jobs. They cannot starve video processing: each is finite and only exists because someone clicked something. - **Estimates come from measured runs.** `videos.duration` and `videos.process_seconds` give a processing-seconds-per-source-second ratio, so a 6-hour VOD is not predicted to cost the same as a 20-minute upload. Below three samples the API reports `confident: false` and the UI softens its wording rather than inventing a number. - **There is no model manager, deliberately.** YOLO and TalkNet load once per process and are never unloaded (`video/tracker.py`, `video/asd.py`), Whisper is reloaded per video by faster-whisper itself, and Ollama is out of process. Reuse across queue items is already the behaviour; a manager would add a lifecycle nothing asked for. - **Per-run logs.** The in-memory ring holds 400 lines, which is minutes: no use for a batch that failed at 3am. Each job tees output to `data/logs/job_N.log`; the worker keeps the most recent 50 and prunes after every job. ### 10.2 The assistant and MCP The **Ask Clips Kitty** box at the bottom of the app talks to `server/agent.py` (`/agent/chat`): the same LLM as the clips, given the app's tools (queue videos with their settings, a clip direction, sport options and match events, and more). `server/mcp.py` offers the same tools over MCP on stdio, so Claude, Cursor and other assistants can drive Clips Kitty too. ### 10.3 Remote rendering Off by default and hidden under Settings → Advanced settings ([docs/REMOTE-RENDERING.md](docs/REMOTE-RENDERING.md)). Only rendering moves: the main PC still downloads, transcribes, scores, keeps the library and publishes, and sends one clip's stretch of video at a time to a paired render PC (`remote_render/`), which renders it with the same code and settings and sends the finished clip back. The connection is a pinned HTTPS gateway on port 8766. --- ## 11. Desktop application **Electron + Vite + React + TypeScript + Tailwind.** The renderer never touches Python or the filesystem directly: context isolation on, no node integration, everything through the local API. Pages: **Dashboard** (system widgets, processed videos, live log), **Queue** (batch add, per-video settings, reorder/pause/retry, queue-wide time estimate), **Clip Editor** (the core loop: paste a link, watch progress, review the results grid, open the editor), **Creators**, **Models**, and **Settings**. ### Visual theme Dark-first, navy base with a single sky-blue accent, defined as Tailwind design tokens so components inherit it: | Token | Value | Used for | |---|---|---| | `bg-base` | `#0A1628` | app background | | `bg-surface` | `#13243D` | cards, panels, sidebar | | `bg-raised` | `#1C3354` | hover states, inputs, modals | | `accent` | `#38BDF8` | primary buttons, active nav, progress | | `accent-strong` | `#0EA5E9` | hover, selection | | `text-primary` | `#F1F5F9` | headings, body | | `text-muted` | `#94A3B8` | labels, secondary text | | `success` / `warn` / `error` | `#34D399` / `#FBBF24` / `#F87171` | status chips, score badges | Accessibility is treated as a requirement, not a nicety: keyboard focus, reduced-motion support, adjustable font and text size, and colour choices checked for contrast. --- ## 12. State and storage SQLite (`core/state.py`) is the single source of truth. Principal tables: `videos`, `clips`, `rejections` (why a candidate was dropped: auditable), `jobs`, `uploads`, `creators`, `platform_accounts`, `creator_knowledge`, `creator_events`, `clip_feedback`, `branding_profiles`, `creator_terms`, `clip_translations`, and `app_state` (durable app-level flags: currently just whether the queue is paused). `jobs` carries the queue's own columns beyond the payload: `position` (run order), `video_id` / `title` (naming and the duplicate guard), `interrupted`, `attempts`, `started_at` / `finished_at`, and `log_path`. `videos.duration` records source length, which is what the queue's time estimate divides by. Guarantees: a video is never processed twice, a clip is never uploaded twice, and any crash resumes from the last completed stage. Schema changes land as additive migrations in `_migrate()`, so an existing database upgrades in place without losing data. Deleting a creator profile **never** deletes videos or clips. They are only unlinked, so the list can be tidied without losing footage. --- ## 13. Configuration surface `config/settings.yaml` opens with a short quick-setup block and keeps everything else below it. The values that matter most: ```yaml model: gemma:7b # an Ollama tag (local AI), or provider/model for cloud AI # on the user's own account, e.g. openrouter/ content_language: auto # or force es / pt / hi / id / en … clips: min_score: 55 # generous on purpose — creators pick what to post max_clips_per_video: 0 # 0 = keep everything that clears the bar min_duration: 10 max_duration: 60 scoring: weights: {text: 0.30, visual: 0.20, reaction: 0.20, audio: 0.20, engagement: 0.10} signal_peak_percentile: 85 creator_context_max: 6 # 0 disables creator callbacks action_bonus: 10 # 0 disables the movement bonus audience_bonus: 8 # 0 disables chat/heatmap rerank_pool: 8 video: encoder: auto # nvenc / amf / qsv / cpu parallel_renders: 3 tracking: detector: yolov8n-pose.pt sample_fps: 8 transcription: backend: local # or a cloud provider on the user's own account model: "" # that provider's voice model ``` Two choices are not in the file and are on by default: shot-aware framing (§5) and clip selection 2 (§4.2). A line in the file turns either off: `shot_aware: false` under `tracking:`, or `selection: 1` under `scoring:`. `tracking.shot_aware_no_subject` (`center`, the default, or `hold`) is what a shot with nobody in it opens on, and Longform keeps selection 1 unless `scoring.longform_selection: 2`. Prompts are data, not code: every LLM prompt lives in `config/prompts/` as plain text, so the rating system can be tuned without touching Python. `config/gaming.yaml` (kinds of game, and what a highlight is in each) and `config/sports.yaml` (each sport's moments, their windows and the commentary words that mark them) are data in the same way. --- ## 14. Windows packaging The installer has to carry everything, because the audience is creators, not developers. "Install Python, install FFmpeg, add it to your PATH" is where a streamer closes the window and never comes back. **electron-builder (NSIS **web** installer) + a PyInstaller one-dir backend + bundled FFmpeg.** Built by `python scripts/build_installer.py`, which runs the whole chain and stops at the first failure with an explanation. **Why a web installer, not a single .exe:** `makensis.exe` is a 32-bit program and memory-maps the payload in order to embed it, so it fails at roughly 2 GB with `failed creating mmap`. This app is ~5 GB unpacked, nearly all of it CUDA PyTorch, so a self-contained NSIS installer is not possible: not a configuration problem, a hard ceiling. At this size a web installer is also simply better: the setup starts instantly and the large download is **resumable**, which matters over a home connection. A `.zip` ships alongside for offline installs. | Piece | How it ships | Why | |---|---|---| | Front end | electron-builder, NSIS **web** installer + zip | Small setup that downloads a resumable payload; zip covers offline | | Python engine | PyInstaller **one-dir** → `resources/backend/api.exe` | One-file unpacks gigabytes to temp on every launch: slow and fragile with PyTorch in the bundle | | FFmpeg | `scripts/fetch_ffmpeg.py` → `vendor/ffmpeg/` → `resources/backend/ffmpeg/` | Found by `core/binaries.py`; never depends on the user's PATH | | YOLO weights | Bundled as data | Otherwise the first video stalls on a silent download | | TalkNet weights | `models/pretrain_TalkSet.model` bundled as data | ~60 MB, and speaker detection degrades to motion-based framing without it: `asd.available()` gates every call, so a missing file is a quieter clip, not a crash | | Game-sound weights | `scripts/fetch_panns.py` → `models/panns_mobilenetv1.pth` bundled as data | 24 MB (PANNs MobileNetV1, CC BY 4.0); only a gaming stream uses it, and `panns.available()` gates it, so without it a gaming stream is scored on chat and voice alone | | Voice models | `scripts/fetch_voice_model.py` → `models/pyannote_segmentation_3.onnx` and `models/wespeaker_resnet34_lm.onnx` bundled as data | 33 MB (pyannote segmentation-3.0, MIT; WeSpeaker ResNet34-LM, CC BY 4.0); only a clip with the second speaker's colour ticked uses them, and `voice_turns.available()` gates it, so without them captions stay one colour | | PyTorch | CUDA build, bundled | Not just for tracking. The CUDA wheels carry the cuDNN DLLs that CTranslate2 needs for GPU transcription. A CPU build makes *both* Whisper and tracking fall back to CPU | | cuBLAS 12 | `nvidia-cublas-cu12`, bundled | CTranslate2 (Whisper) loads cuBLAS 12 by name, and current PyTorch CUDA builds carry only cuBLAS 13. If it can't be loaded, transcription carries on on the CPU instead of failing | | Ollama + LLM | **Not bundled**. The setup wizard detects and installs | Separate product with its own installer, GPU handling and update cycle; models are gigabytes and the right one depends on the user's VRAM | Two details that are easy to get wrong and expensive to discover late: - **The backend is a console app on purpose.** A windowed PyInstaller build gives the process no stdout, and every `print()` in the pipeline then raises. Electron passes `windowsHide` so no console is ever shown. - **Text encoding is forced to UTF-8 at the entry point.** Windows gives a spawned process the system locale's encoding: cp1252 on most Western installs, which cannot encode an emoji. Stream titles are full of them. This is invisible in development because a developer's terminal usually has UTF-8 configured, and Electron does not inherit that. `core/preflight.py` checks what an install actually has. FFmpeg, Ollama, the model, GPU, disk, and reports each in words a creator can act on, so a missing piece surfaces before processing rather than as a stack trace twenty minutes into a video. It is served at `GET /health/preflight`. --- ## 15. Design rules These are the constraints that keep the system modular and safe to change: - **Stages communicate only through dataclasses and files.** The analyzer never touches video; the tracker never reads transcripts. - **`analysis/` depends on `llm/base.py` only**: never on a concrete backend. - **`sources/` and `publish/` are plugin folders.** Adding a platform means adding one file; nothing downstream changes. - **Learned data can never degrade output.** Every score contribution derived from accumulated knowledge is additive, capped, deterministic, and individually disableable. - **The LLM proposes, deterministic code disposes.** Parsing, validation, duration enforcement, deduplication, and edit application are all plain Python, so a model swap changes quality but never correctness. - **Failure is contained.** Optional subsystems: creator learning, chat replay, heatmaps, signal prefetch: are wrapped so that a failure degrades a feature instead of breaking a run. - **Prompts are editable text.** Tuning behaviour should not require writing code. ### Scaling path The state DB and file-artifact handoff between stages mean stages can later become queue workers (transcription on the GPU box, rendering elsewhere) without redesign. The contracts between stages don't change.