**English** · [简体中文](./README.md) # aivideo-shotkit > **Automatically split one AI-generated video (or screen recording) into clips that can be fed directly to character-replacement tools.** > Automatically detect and crop the player UI overlay → split into shots by scene → measure motion per shot → split over-long shots into sub-segments → export a workflow bundle with prompts and a stitching plan. [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) [![CI](https://github.com/loubaji083-rgb/dsh-plugin-aivideo-shotkit/actions/workflows/ci.yml/badge.svg)](https://github.com/loubaji083-rgb/dsh-plugin-aivideo-shotkit/actions/workflows/ci.yml) [![Node](https://img.shields.io/badge/node-%5E22.19.0%20%7C%7C%20%3E%3D24.0.0-brightgreen.svg)](package.json) [![Dependencies](https://img.shields.io/badge/runtime%20deps-0-success.svg)](#依赖) [![DSH Plugin](https://img.shields.io/badge/DSH-plugin-4f46e5.svg)](https://github.com/deepseek-ai) --- ## Table of Contents - [What problem it solves](#what-problem-it-solves) - [What it does not do](#what-it-does-not-do) - [Features](#features) - [Installation](#installation) - [Quick start](#quick-start) - [Output](#output) - [Command and tool reference](#command-and-tool-reference) - [How it works](#how-it-works) - [The numbers behind the parameters (all sourced)](#the-numbers-behind-the-parameters-all-sourced) - [Capability limits](#capability-limits) - [License red lines](#license-red-lines) - [FAQ](#faq) - [Development and testing](#development-and-testing) - [Contributing](#contributing) - [License](#license) --- ## What problem it solves Nearly every character-replacement / face-swap / motion-transfer model is a **single-call, single-shot** tool, and its window is narrow: - ComfyUI's `WanAnimate2ToVideo` takes only **81 frames @16fps ≈ 5.06 seconds** per call by default; - Kling 3.0 series: **3–15 seconds** per call, Runway Act-Two: **3–30 seconds**, Viggle: **5–15 seconds**; - Runway's own docs also explicitly require **"No cuts that interrupt the shot"** (no cut inside a single clip). So a finished video tens of seconds long **cannot be fed in directly** — you first have to split it into shots, then split the over-long shots into sub-segments, and the adjacent sub-segments need to overlap so they can be stitched back together. This tool is that preprocessing step. It turns "eyeball it and cut with manual ffmpeg" into one repeatable command, and writes the **recommended tool, parameters, prompts, and stitch points for every clip** into a single workflow bundle. It was originally written to handle **screen-recording footage**: many AI videos are recorded off a player, with a progress bar and danmaku box pressed onto the bottom of the frame, and if you feed that straight into face swap the model models the controls as picture content. So the first step is to automatically detect and crop that overlay. ## What it does not do Let's get the ugly part out of the way first, so you don't install it and only then find it doesn't match your expectations: - **It has no face detector.** It does not decide whether there is a person in frame, whether a face is visible, or whether it is a close-up. The runtime needs only Node + ffmpeg, and downloads no models. - **It does not do character replacement for you.** It only produces "clips you can take to a replacement tool" plus the accompanying notes; the replacement itself is done by Wan Animate / VACE / Kling / LivePortrait and the like. - **It does not recognize people.** No face recognition, no distinguishing between characters. - **Its "warm/skin-tone pixel share" is only a picture-statistics heuristic**, and runs high in warm-lit interiors; **it cannot be used as evidence of "is there a person"**, only for relative ordering. ## Features | Capability | Description | | --- | --- | | **Automatic player-UI cropping** | Per-row temporal statistics find strips that are "dark + static frame to frame", recognize OBS/screen-recording sources, and give a crop suggestion. Measured on a 2560×1440 bilibili screen recording, it correctly detected the 80px control strip at the bottom. | | **Flicker-resistant shot detection** | It does not use `select='gt(scene,TH)'` directly (on high-motion footage that splits flicker into a pile of fake shots); instead it takes the whole-video scene-score series and applies an "isolated spike" criterion plus minimum-shot-duration clustering. | | **Per-shot motion magnitude** | Normalized to **the average luma difference per model frame (1/16 s)**, so footage with different source frame rates can be compared directly. Comes with motion coverage, flicker, sharpness, brightness and other metrics. | | **Grouping by motion magnitude** | Four buckets — still / subtle / moderate / intense — each with its own workflow strategy and recommended tools. | | **Sub-segmentation of over-long shots** | Split evenly by the tool's per-call window preset (15s / 5s / 2s) with an overlap kept, emitting `shot_005_seg02_….mp4`. | | **Workflow bundle** | One run produces `workflow.json` / `prompts.csv` / `README.md` / `analysis.json` / `clips.json`, with per-clip prompts and the command to stitch back into the full video. | | **Three ways to use it** | DSH plugin (4 native tools + a skill + a status panel), standalone CLI, or `import` the core module directly. | | **Zero runtime dependencies** | The core uses only Node built-ins (`spawn` + typed arrays); no numpy / opencv / onnxruntime. | | **No ffprobe needed** | Metadata is parsed from the `ffmpeg -i` banner, so a single ffmpeg is enough to run. | ## Installation ### Prerequisite: ffmpeg You need a **callable ffmpeg** (ffprobe is not required). Detection order: 1. `ffmpegPath` in the plugin config / parameters 2. The environment variables `AIVIDEO_FFMPEG`, `FFMPEG_PATH` 3. The system `PATH` 4. Common install locations (Windows: `%LOCALAPPDATA%\oopz\ffmpeg.exe`, `%APPDATA%\bilibili\ffmpeg\ffmpeg.exe`, etc.) ```bash ffmpeg -version # just confirm it works ``` > A build **4.x or newer** is recommended. During development the core paths were also verified on ffmpeg 3.0.1 (a 2016 build), but old builds lack filters such as `scdet`, and have no NVENC. ### Option 1: Install into DSH (recommended — gives you tools + skill + panel) ```bash # install from GitHub (the package name is the directory name) dsh plugin --profile add github:loubaji083-rgb/dsh-plugin-aivideo-shotkit # or install from a local clone git clone https://github.com/loubaji083-rgb/dsh-plugin-aivideo-shotkit.git dsh plugin --profile add file:/absolute/path/dsh-plugin-aivideo-shotkit ``` You can also install it directly in DSH's graphical plugin manager. **After installing, the profile must be reloaded (restart DSH)**, because a profile's bundle list is assembled at startup. Once installed you get: - 4 native tools: `aivideo_probe` / `aivideo_detect_shots` / `aivideo_analyze` / `aivideo_split` - 1 skill, `aivideo-shotkit` (the model picks it up automatically when you mention video splitting / character replacement) - 1 read-only status panel: "Settings → AI Video Shotkit", showing a summary of the most recent analysis/cut - 1 read-only route, `GET /aivideo-shotkit/state` ### Option 2: CLI only (no DSH) ```bash git clone https://github.com/loubaji083-rgb/dsh-plugin-aivideo-shotkit.git cd dsh-plugin-aivideo-shotkit node bin/shotkit.mjs run "" --out ./out --filmstrip ``` The CLI needs no dependencies installed and no DSH. ### Option 3: Use it as a library ```js import { analyzeVideo, exportShots, resolveToolchain } from './lib/core/pipeline.mjs'; const tc = resolveToolchain(); const analysis = await analyzeVideo(tc.ffmpeg, 'input.mp4', { outputDir: './out' }); const result = await exportShots(tc.ffmpeg, 'input.mp4', analysis, { outputDir: './out' }); ``` ## Quick start ### 0. Run the synthetic demo first (30 seconds, does not touch your footage) The repo ships a **fully synthetic** demo-footage generator — it builds a video with a "fake player control strip" on the spot, which is exactly what you want to verify the whole pipeline: ```bash node scripts/make-demo-video.mjs --out demo/demo.mp4 node bin/shotkit.mjs run demo/demo.mp4 --out demo/out --filmstrip ``` Expected result (measured): ``` video 640x388 -> crop {"x":0,"y":0,"width":640,"height":358} playerUi true chromeBottomPx 30 scene.max 1 shots 6 #1 0->3 motion 0.00 static local cat still score 0.61 #2 3->6 motion 3.91 moderate mixed cat subtle score 0.86 #3 6->9 motion 0.00 static local cat still score 0.52 #4 9->12 motion 2.36 low mixed cat subtle score 0.81 #5 12->15 motion 0.00 static local cat still score 0.61 #6 15->18 motion 3.23 moderate mixed cat subtle score 0.85 ``` All 6 scenes land on the exact 3.000 / 6.000 / 9.000 / 12.000 / 15.000 second cut points, and the bottom control strip is cropped automatically. ![Filmstrip of the synthetic demo footage](assets/demo-filmstrip.jpg) ### 1. Process your own footage **Inside DSH** (the model calls them in order automatically): ``` first, aivideo_probe to check whether the footage has a player UI overlay then aivideo_analyze to detect shots and evaluate each one once you are happy, aivideo_split to cut the clips and write the workflow bundle ``` **On the command line** (one command runs the whole flow): ```bash node bin/shotkit.mjs run "" --out ./out --filmstrip ``` `run` = `analyze` + `cut`. See the next section for the artifacts. ### 2. Adjust as needed ```bash # local open-source pipeline (ComfyUI / VACE): tighten the per-call window to 5s node bin/shotkit.mjs run "