# yueying — let AI watch videos **Point Claude, Cursor or any MCP client at a video and get back a timestamped transcript plus keyframe contact sheets — offline, no API key.** Local files first; URLs (YouTube, Bilibili, Douyin, Xiaohongshu, TikTok, Vimeo, …) are videos you are entitled to process, fetched via yt-dlp at ≤720p and deleted after processing by default. [](https://pypi.org/project/yueying/) [](https://pypi.org/project/yueying/) [](LICENSE) [](pyproject.toml) [](https://cursor.com/en/install-mcp?name=yueying&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJ5dWV5aW5nIiwibWNwIl0sImVudiI6eyJQWVRIT05VVEY4IjoiMSJ9fQ==) [](https://insiders.vscode.dev/redirect?url=vscode%3Amcp%2Finstall%3F%257B%2522name%2522%253A%2522yueying%2522%252C%2522command%2522%253A%2522uvx%2522%252C%2522args%2522%253A%255B%2522yueying%2522%252C%2522mcp%2522%255D%252C%2522env%2522%253A%257B%2522PYTHONUTF8%2522%253A%25221%2522%257D%257D) [中文说明 ↓](#中文说明) Yueying (阅影) means "read video". One package gives you an **MCP server**, a **CLI** and an **agent skill**. ## What you get  *Claude Desktop with yueying connected: paste a link, wait about forty seconds, get the video back as timestamped notes. This video ships captions, so speech recognition never ran, and the model asked for the transcript only. Keyframes and contact sheets come back through `get_frames` when it needs to see the screen. Demo video: [GitInGifs: Git Branches](https://www.youtube.com/watch?v=Q5OaMTd7PwM) by GitLab, CC BY.*  *Contact sheet from a 24-second demo clip (four app screenshots with Chinese narration). The yellow label on every tile is the keyframe number and timestamp; the model cites them back to you.* The transcript of the same clip — local speech recognition, language auto-detected as Chinese: ``` [00:00] 这是阅读,一个安静的桌面小说阅读器。整本书连续滚动,按段落记住进度。 第二个画面是桌面模式,窗口变透明,只留文字浮在桌面上。 第三个画面是伪装皮肤,一键变成代码编辑器。 最后是伪装成表格的样子。 ``` (The app is called 月读; ASR heard the homophone 阅读. Speech recognition does that to names — the model corrects it from the on-screen text in the frames.) Every video becomes one folder: ``` report.md index for the model: metadata, chapters, contact sheets, keyframes, transcript transcript.txt paragraphs with [mm:ss] timestamps transcript.srt subtitles for any player grid_01.jpg … 3x3 contact sheets, 9 keyframes each, in time order frames/ full-size keyframes, e.g. f003_00m15s.jpg manifest.json machine-readable result (paths, segments, chapters, options) ``` ## Why yueying - **Captions first, Whisper only when needed.** Platform subtitles are used when they exist. Otherwise local [faster-whisper](https://github.com/SYSTRAN/faster-whisper): `large-v3-turbo` on an NVIDIA GPU, `small` on CPU, automatic CPU fallback — nothing is uploaded, no key. - **ffmpeg bundled.** Works on Windows 11 out of the box (imageio-ffmpeg); no PATH fiddling. - **Token-efficient.** Keyframes are taken at scene changes, near-duplicates dropped, then packed into 3x3 contact sheets with burned-in timestamps. One sheet ≈ 1–2K tokens for nine moments; one transcript with `[mm:ss]` paragraphs. - **Chinese platforms and the rest.** Bilibili (multi-part, collections, member videos with your browser login), Douyin, Xiaohongshu — and YouTube, TikTok, Vimeo, X and every other yt-dlp site. - **Zero API keys, zero telemetry.** The only network traffic is the video site you name and one Whisper model download. See the [privacy policy](#privacy-policy). Benchmark: a 6-minute Bilibili video → report in ~90 s on an RTX 5060 laptop; on CPU with `model=small` expect ~1–2 min per 10 min of speech. ## Quick start 1. Install [uv](https://docs.astral.sh/uv/) (Python is not required): ```bash winget install astral-sh.uv # Windows brew install uv # macOS curl -LsSf https://astral.sh/uv/install.sh | sh # Linux / macOS ``` 2. Warm up and check everything once (installs the package, probes the GPU, downloads the speech model, runs a 2-second smoke test, prints config to paste): ```bash uvx yueying mcp --setup ``` 3. Add the server to your client (below), then ask: *"Watch C:\videos\lecture3.mp4 and turn the steps into notes"* or *"What does this video say about docker compose: https://www.bilibili.com/video/BV…"*. ### Claude Desktop `%APPDATA%\Claude\claude_desktop_config.json` (Windows) · `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS). Fully quit and reopen Claude afterwards. ```json { "mcpServers": { "yueying": { "command": "uvx", "args": ["yueying", "mcp"], "env": { "PYTHONUTF8": "1" } } } } ``` Windows note: Claude Desktop does not always see your PATH — if the server fails to start ("spawn uvx ENOENT"), use the absolute path, e.g. `"command": "C:\\Users\\\\.local\\bin\\uvx.exe"` (`where uvx` prints it). Logs: `%APPDATA%\Claude\logs\mcp-server-yueying.log` (`~/Library/Logs/Claude/` on macOS). Keep `wait_seconds` at its default there; see [the RUNNING rule](#the-running-rule). ### Claude Code ```bash claude mcp add --transport stdio --scope user yueying --env PYTHONUTF8=1 -- uvx yueying mcp ``` Or drop this repo's [`.mcp.json`](.mcp.json) into a project (it ships with `"timeout": 1800000` so one `watch_video` call can wait for a long video). To raise Claude Code's tool timeout globally, set `MCP_TOOL_TIMEOUT=1800000` (ms) in your environment. The repo is also a Claude Code plugin (`.claude-plugin/plugin.json`: server + skill). ### Cursor Click the **Add to Cursor** badge above, or put the same JSON in `~/.cursor/mcp.json` (global) or `.cursor/mcp.json` (project): ```json { "mcpServers": { "yueying": { "command": "uvx", "args": ["yueying", "mcp"], "env": { "PYTHONUTF8": "1" } } } } ``` ### Cline MCP Servers → Configure (`cline_mcp_settings.json`). `timeout` is in seconds; the five read-only tools are safe to auto-approve. Step-by-step agent instructions: [llms-install.md](llms-install.md). ```json { "mcpServers": { "yueying": { "type": "stdio", "command": "uvx", "args": ["yueying", "mcp"], "env": { "PYTHONUTF8": "1" }, "timeout": 1800, "autoApprove": ["get_transcript", "search_transcript", "get_frames", "get_frame_at", "list_videos"] } } } ``` ### Windsurf `~/.codeium/windsurf/mcp_config.json`: ```json { "mcpServers": { "yueying": { "command": "uvx", "args": ["yueying", "mcp"], "env": { "PYTHONUTF8": "1" } } } } ``` ### VS Code (Copilot agent mode) Click the **Install in VS Code** badge above, or create `.vscode/mcp.json` (note the root key `servers`): ```json { "servers": { "yueying": { "type": "stdio", "command": "uvx", "args": ["yueying", "mcp"], "env": { "PYTHONUTF8": "1" } } } } ``` Direct links for hosts that accept custom URL schemes: `cursor://anysphere.cursor-deeplink/mcp/install?name=yueying&config=eyJjb21tYW5kIjoidXZ4IiwiYXJncyI6WyJ5dWV5aW5nIiwibWNwIl0sImVudiI6eyJQWVRIT05VVEY4IjoiMSJ9fQ==` and `vscode:mcp/install?%7B%22name%22%3A%22yueying%22%2C%22command%22%3A%22uvx%22%2C%22args%22%3A%5B%22yueying%22%2C%22mcp%22%5D%2C%22env%22%3A%7B%22PYTHONUTF8%22%3A%221%22%7D%7D`. ### Without uv (pip / pipx) and Windows one-click ```bash pip install yueying # or: pipx install yueying yueying mcp --setup # prints a config with the absolute path of the yueying-mcp executable ``` Use that absolute path as `"command"` with no `args` (Windows: `...\Scripts\yueying-mcp.exe`; also works as `python -m yueying mcp`). Windows users without Python tooling can double-click [`install.cmd`](install.cmd) from a checkout: it creates `%LOCALAPPDATA%\yueying\venv`, installs the Claude Code skill, runs `yueying mcp --setup` and prints the JSON block with the right path. ### GPU ```bash uvx --from "yueying[cuda]" yueying mcp # NVIDIA: adds the CUDA runtime wheels (cuBLAS, cuDNN) pip install "yueying[cuda]" ``` Device and model are chosen automatically (`model=auto`: large-v3-turbo on CUDA, small on CPU); if the GPU trial fails, recognition falls back to CPU by itself. ### Docker ```bash docker build -t yueying . docker run --rm -i -v yueying-data:/data -v "$PWD/videos:/videos:ro" yueying ``` The image is CPU-only (containers get no GPU by default), so it defaults to the `small` model. Mount your videos read-only and give the tools container paths (`/videos/lesson.mp4`); results and the downloaded Whisper weights live in the `/data` volume. In a client config the `command` is `docker` and `args` are `["run", "--rm", "-i", "-v", "yueying-data:/data", "-v", "/your/videos:/videos:ro", "yueying"]`. ## Tools | Tool | When the agent uses it | What it returns | Limits | |---|---|---|---| | `watch_video(video, mode="full", language="auto", model="auto", frame_interval_seconds=None, cookies_from_browser=None, output_dir=None, refresh=False, wait_seconds=45, max_chars=12000)` | First call for any video: an absolute local path or a URL. `mode`: `full` (transcript + keyframes), `transcript`, `frames`. | `DONE` overview: title, source, duration, text source, folder, files, chapters, contact-sheet ranges, transcript in `[mm:ss]` paragraphs — or `RUNNING` with stage/percent/ETA, or `ERROR` with a plain-English hint. | Blocks up to `wait_seconds` (0–1500). Transcript truncated at `max_chars` with a `get_transcript` start time. Cached per video; `refresh=true` reprocesses. | | `get_transcript(video, start="0", end=None, format="paragraphs", max_chars=8000)` | The overview was truncated, a specific time range, or exporting subtitles (`format="srt"`). | Header + `[mm:ss]` paragraphs / `[mm:ss-mm:ss]` segments / SRT blocks; `TRUNCATED — next_start="…"` when cut. | `max_chars` 1000–100000. Times: seconds, `mm:ss`, `h:mm:ss`. | | `search_transcript(video, query, context_seconds=15, limit=10)` | "When does he mention X?" | Hits with time, surrounding sentences, nearest keyframe number and contact-sheet number. | Space-separated terms, any matches, more terms rank higher; `limit` ≤ 50. | | `get_frames(video, kind="grids", start=1, count=2, max_width=1280)` | See what is on screen: contact sheets first (`grids`), single keyframes (`frames`) only for detail. | Absolute path + JPEG per image, in time order; `Next: get_frames(start=…)` when more remain. | ≤ 3 images per call (default 2, keep ≤ 2 in Claude Desktop); ~150 KB per sheet at 1280 px. | | `get_frame_at(video, time, max_width=960)` | Read code, a slide, a chart or UI at one moment. | The exact frame (extracted from the source when the local file still exists) or the nearest cached keyframe, plus the paragraphs spoken around then. | One image, ~100 KB at 960 px. | | `list_videos(limit=20)` | The user refers to an earlier video, or to check disk use. | Table: video_id, title, duration, text source, date, size, folder; running jobs. | Instant, read-only. | `video` for the read tools accepts the `video_id` from `watch_video`/`list_videos`, the results folder, or the same path/URL you gave `watch_video`. ### The RUNNING rule Processing can take minutes, and most hosts cap a tool call at about a minute. So `watch_video` waits at most `wait_seconds`, then answers `RUNNING video_id=… · stage 2/4 speech recognition 40% · elapsed 46 s · est. ~1 min remaining`. The agent simply **calls `watch_video` again with the same `video`** — it re-attaches to the same job (options are ignored while it runs; `refresh=true` restarts). Recommended `wait_seconds`: | Host | `wait_seconds` | Why | |---|---|---| | Claude Desktop | 45 (default) | hard ~60 s client timeout | | Cursor | 45 (default) | 60–120 s | | Claude Code | up to 1500 | with `.mcp.json` `timeout` / `MCP_TOOL_TIMEOUT` = 1800000 ms | | Cline | up to 1500 | with `"timeout": 1800` (s) | Progress notifications are sent every 1.5 s for hosts that display them. One video is processed at a time per server; extra requests queue. **First run:** the first speech recognition downloads a Whisper model once (~480 MB `small` on CPU, ~1.6 GB `large-v3-turbo` on GPU). `uvx yueying mcp --setup` does this ahead of time; otherwise the `RUNNING` line says "first run downloads ~… this can take several minutes". ## Where files go Root: `$YUEYING_OUT_DIR` if set, else `~/yueying_out`. One entry per video, named `-` (`yt-`, `bili-`, or the file name) — never renamed; the title lives in `manifest.json`. ``` ~/yueying_out/ └── bili-BV1xx-3f9a2c1e/ ├── report.md transcript.txt transcript.srt manifest.json ├── grid_01.jpg … grid_07.jpg ├── frames/ f001_00m02s.jpg … (+ frames/extra/ for get_frame_at) ├── .job only while a job runs └── _download/ only with YUEYING_KEEP_SOURCE=1 ``` | Environment variable | Meaning | Default | |---|---|---| | `YUEYING_OUT_DIR` | root folder for results (absolute, `~` ok) | `~/yueying_out` | | `YUEYING_MODEL` | default for the `model` parameter | `auto` | | `YUEYING_DEVICE` | `auto` / `cuda` / `cpu` | `auto` | | `YUEYING_LANG` | language of report.md written by the server (`en`/`zh`) | `en` | | `YUEYING_COOKIES_FROM_BROWSER` | default browser for cookies (`chrome`, `edge`, `firefox`, …) | unset | | `YUEYING_KEEP_SOURCE` | `1` keeps the downloaded ≤720p source in `_download/` (enables exact-moment frames for URLs) | unset | | `YUEYING_MAX_JOBS` | pipelines running at once per server | `1` | | `YUEYING_JOB_TIMEOUT` | hard limit per video, seconds | `7200` | | `HF_HOME` | Hugging Face cache (Whisper weights live here) | HF default | | `HF_ENDPOINT` | mirror, e.g. `https://hf-mirror.com` | huggingface.co | | `PYTHONUTF8` | set to `1` on Windows to avoid mojibake | — | Disk budget: ≈ 25 MB per hour of video; 300–600 MB/h more with `YUEYING_KEEP_SOURCE=1`. Nothing is deleted automatically — `list_videos` shows sizes; delete a folder to free space; `watch_video(refresh=true)` reprocesses one video. Editing a local file changes its size/mtime and therefore gets a new entry. ## Supported sources - **Local files:** anything ffmpeg reads — mp4, mkv, mov, webm, avi, flv, ts, and audio (mp3, m4a, wav, …). Audio-only input gives a transcript without frames. - **URLs:** every site yt-dlp supports. Fetched at ≤720p and deleted after processing unless `YUEYING_KEEP_SOURCE=1`. - **Bilibili:** without login Bilibili serves 480p — enough for slides and code. For HD or member-only videos pass `cookies_from_browser="chrome"` (or `edge`, `firefox`, `brave`, `chromium`, `safari`); on Windows close Chrome first, it locks its cookie database. Multi-part videos and collections: `p=` links are separate entries; the CLI's `--all` processes them all. - **Not for** live streams or images. Short links (`b23.tv`, `v.douyin.com`) are processed but not de-duplicated against their long form (the server never resolves URLs itself). ## Also a CLI and an agent skill ```bash yueying video.mp4 yueying "https://www.bilibili.com/video/BVxxxx" --ui-lang en yueying "https://www.youtube.com/watch?v=xxxx" --out ./notes/xxx yueying lesson1.mp4 lesson2.mp4 "https://www.bilibili.com/video/BVyyyy" # several at once, one folder each + index.md yueying "https://www.bilibili.com/video/BVxxxx" --all # every part of a multi-part video / collection yueying --install-skill # Claude Code skill -> ~/.claude/skills/yueying ``` Default output: `./yueying_out//` (parent folder when several inputs). CLI log lines and report.md are Chinese by default (`--ui-lang en` for English); 0.3 will flip the default to English. | Flag | Meaning | |---|---| | `--out DIR` | output folder (default `./yueying_out/`; the parent folder when several inputs) | | `--all` | when the URL is a Bilibili multi-part video / collection / playlist, process every entry (default: only the first) | | `--lang zh` | spoken language code (`zh`, `en`, `ja`, …); default auto-detect | | `--model auto` | Whisper model: `auto` / `tiny` / `base` / `small` / `medium` / `large-v3` / `large-v3-turbo` (CLI default). `auto` = large-v3-turbo on an NVIDIA GPU, small on CPU | | `--device cpu` | force CPU (`auto` / `cuda` / `cpu`) | | `--interval 3` | roughly one keyframe every N seconds. Default by duration: 2 s under 1 min, 3 s under 3 min, 6 s under 10 min, 12 s under 30 min, 20 s beyond | | `--frames 30` | maximum number of keyframes (default by duration, cap 150; 300 with `--interval`) | | `--scene 0.2` | scene-change sensitivity 0–1, lower = more sensitive (default 0.3) | | `--no-dedupe` | keep frames that are almost identical to the previous one (default drops them: < 2 % of thumbnail pixels changed) | | `--no-asr` | no speech recognition even without subtitles (pictures only) | | `--no-frames` | no keyframes (text only) | | `--force-asr` | run speech recognition even when subtitles exist | | `--cookies-from-browser chrome` | download with your browser login (Bilibili HD / member videos, sign-in-gated YouTube) | | `--keep` | keep the downloaded source video | | `--ui-lang en` | language of report.md headings and labels: `zh` (default) or `en` | | `--json` | print one line of manifest JSON at the end (for scripts) | | `--install-skill` | install the agent skill into `~/.claude/skills/yueying` | | `mcp` | run the MCP server (`mcp --setup`, `mcp --check`, `mcp --version`) | The skill (`src/yueying/skill/SKILL.md`) tells a coding agent to prefer the MCP tools when present and otherwise run the CLI and read `report.md` plus the contact sheets. Tools that support the Agent Skills standard can copy `~/.claude/skills/yueying/SKILL.md` into their own skills folder. ## Compared with similar projects (September 2026) | | yueying | claude-video | claude-real-video | mcp-video-analyzer | |---|---|---|---|---| | MCP server | yes | no (skill only) | no (skill) | yes (Node) | | Offline speech recognition | yes — subtitles first, local faster-whisper otherwise | cloud Whisper fallback | ASR-first | whisper installed separately | | GPU auto-detect + CPU fallback | yes | – | – | – | | ffmpeg bundled | yes | – | manual ffmpeg | – | | Windows tested | yes (Windows 11) | – | – | – | | Contact sheets (3x3) | yes | – | – | – | | Burned-in timestamps on frames | yes | – | – | – | | Bilibili / Douyin / Xiaohongshu | yes | – | – | – | "–" means the project did not advertise the feature when we looked; check their READMEs, they may have moved on. ## Privacy policy yueying collects nothing and has no telemetry, analytics, crash reporting or update checks. All processing is local. The only network connections are (1) to the video site of the URL you pass, through yt-dlp, and (2) to Hugging Face (or `HF_ENDPOINT`) to download a Whisper model once. Outputs are stored in your folder until you delete them. The transcript and any frames you request are sent only to the model your MCP client is configured to use — that transfer is governed by your client's and provider's terms, not by yueying. Questions: [GitHub issues](https://github.com/vsh5dvsch7-png/yueying/issues). Full text: [docs/privacy.md](docs/privacy.md). ## Troubleshooting / FAQ - **"No result received" in Claude Desktop.** Keep `wait_seconds` at 45 (the agent then re-calls `watch_video`), and run `uvx yueying mcp --setup` once so the first call is not also the model download. - **`spawn uvx ENOENT` / server fails to start.** The host cannot see your PATH: use the absolute path to `uvx` (`where uvx` / `which uvx`) or to `yueying-mcp` as `"command"`. - **Console windows flash on Windows.** Update to 0.2.0+: child processes are started without a window. If you still see them, you are running an old install (`uv cache clean yueying`). - **Mojibake / `?????` in titles.** Add `"env": { "PYTHONUTF8": "1" }` to the server config (all snippets above include it). - **Bilibili error 412 / "-352".** The site wants a login: `cookies_from_browser="edge"` or `"chrome"` (close Chrome first on Windows). - **YouTube "Sign in to confirm you're not a bot".** Same fix: `cookies_from_browser`. Also try updating yt-dlp: `uv cache clean yueying` or `pip install -U yt-dlp`. - **Slow on CPU.** `model="small"` is already the automatic choice without an NVIDIA GPU; use `mode="frames"` when only the pictures matter, or `mode="transcript"` to skip keyframes. - **Names, numbers and code are wrong in the transcript.** Expected with any ASR — the agent is told to trust on-screen text; ask it to `get_frame_at` the moment. - **Model download is slow or blocked (mainland China).** Set `HF_ENDPOINT=https://hf-mirror.com` in the server `env` (the CLI switches to the mirror automatically when huggingface.co is unreachable). - **GPU error (CUDA / cuDNN / out of memory).** `model="small"` or `YUEYING_DEVICE=cpu`; install the CUDA wheels with `yueying[cuda]`. - **Want to reprocess with different settings.** `watch_video(video=…, refresh=true, …)` — it kills a running job for that video, deletes the entry and starts again. ## How it works ``` video / URL ──► yt-dlp (≤720p) + platform subtitles ──► ffmpeg 16 kHz audio ──► faster-whisper (skipped when subtitles exist) ──► ffmpeg scene detection ──► keyframes (near-duplicates dropped) ──► Pillow: burn "#n mm:ss", pack 3x3 contact sheets ──► report.md · transcript.txt · transcript.srt · manifest.json MCP client ──stdio──► mcp_server.py ──spawns──► python -m yueying.cli