# Command Reference [← Back to the Soup README](../README.md) > The full `soup` command list. ## All Commands ``` soup init [--template chat|code|...|audio] Create config soup init --template hipaa|soc2|eu-ai-act|sr-11-7 Compliance-shaped starting config + the commands for that regime (v0.71.35) soup autopilot [--model ] [--data d.jsonl] [--goal ] Zero-config: pick task/quant/LR/epochs (prompts when omitted) soup advise --goal "..." Pre-flight decision: PROMPT_ENG / RAG / SFT / DPO / GRPO β€” run BEFORE spending GPU hours soup advise compare Show prior verdicts from advise history soup advise explain Rubric + evidence trail of the last verdict soup fetch Fetch a ready-to-edit example config from the bundled catalog soup train --config soup.yaml Start training soup train --config soup.yaml --resume ./output/checkpoint-100 Resume from local checkpoint (not with training.relora_steps) soup train --config soup.yaml --tensorboard Train with TensorBoard logging soup train --config soup.yaml --replay old.jsonl --replay-ratio 0.1 Continual-learning rehearsal: interleave old data so the new task doesn't erase it (sft/pretrain) soup train --config soup.yaml --fsdp full_shard Train with FSDP2 soup train --config soup.yaml --deepspeed zero++ DeepSpeed ZeRO++ (quantized comms) soup train --config soup.yaml --gpus auto|N Launch on local GPUs through Accelerate soup train --config soup.yaml --gpus 8 --nodes 2 --node-rank 0 --master-addr 10.0.0.10 --master-port 29500 Multi-node launch; run on each node with its own rank soup train --config soup.yaml --gpus 4 --no-reexec Print the launch command without running it soup train --config soup.yaml --gate evals/gate.yaml Eval-gated training soup train --config soup.yaml --push-as user/repo Auto-push each checkpoint to HF as branch (a repository it creates is private) soup train --config soup.yaml --push-as user/repo --push-public Create that repository public (an existing one keeps its visibility) soup train --config soup.yaml --push-as user/repo --hf-resume Resume from latest HF checkpoint branch (not with training.relora_steps) soup train --config soup.yaml --find-lr LR range finder: write recommended LR JSON soup train --config soup.yaml --cloud modal|lambda|runpod --gpu a100 Render a cloud GPU controller (plan-only; --cloud-submit submits live) soup infer --model ./output --input p.jsonl --output out.jsonl [--batch-size N] Batch inference; default batch size is 1; fp16/bf16 batching may not be bit-exact with batch size 1 soup infer --model ./output --input p.jsonl --output out.jsonl --cuda-graphs Experimental CUDA graph decode (Qwen2/Llama, one GPU, PyTorch >= 2.14; batch size 1 only) soup infer --task asr --model --input a.jsonl --output o.jsonl [--audio-dir d --asr-language en --asr-task transcribe|translate] Whisper transcription + WER/CER soup chat --model ./output Interactive chat soup infer|chat|diff ... --device cpu|cuda|cuda:N|mps Pick where the model loads (see below) soup push --model ./output --repo user/name Upload to HuggingFace soup push --model ./output --repo user/name --collection user/coll-abc123 Add to HF Collection soup merge --adapter ./output Merge LoRA with base model soup merge --adapter ./output --low-memory Merge tensor by tensor without loading the whole model into RAM soup merge --adapter ./output --save-format 4bit --no-double-quant Save a BNB-4bit merge with double-quantization disabled (default on; #321) soup merge-sharded-fsdp-weights ./shards -o merged.safetensors Consolidate FSDP shards into one safetensors (v0.71.14; --plan-only previews) soup delinearize-llama4 ./src --target ./out [--num-experts N] [--plan-only] Live Llama-4 fused-expert reshape [E*din,dout] -> [E,din,dout] + sidecar copy (v0.71.21) soup spectrum scan --model --top-percent 50 [--modules mlp,attn] [-o patch.yaml] Spectrum SNR scan (no model load) -> training.unfrozen_parameters YAML patch (v0.71.23) soup train --config sft.yaml # training.lisa_enabled: true [lisa_num_layers lisa_interval_steps lisa_train_embeddings] LISA layerwise importance sampling β€” full-FT quality at LoRA-like memory; lisa_train_embeddings: false freezes embeddings+head+norm for the memory saving (sft or pretrain/transformers/text/quantization=none) (v0.71.34, pretrain #307, #377) soup train --config sft.yaml # training.stream_layers: true [stream_source stream_ngram_source stream_buffers stream_read_ahead] BETA layer streaming β€” the frozen base streams from CPU RAM/NVMe one decoder layer at a time; Qwen4-Exp PLE rows can stream read-only from original safetensors; embed_tokens + untied lm_head reuse one large-layer device slot; quantization: 4bit streams validated decoder families as NF4, ~4x smaller (sft/dpo/orpo/simpo/kto on transformers+text, 10 archs; grpo/ppo permanently excluded) (v0.72.0; NF4 v0.72.2; disk+batch+accum v0.72.3; preference losses v0.72.4; Qwen4 PLE #602) soup export --model ./output --format gguf Export to GGUF (Ollama) soup export --model ./output --deploy ollama Export GGUF + auto-deploy to Ollama soup export --model ./output --format onnx Export to ONNX soup export --model ./output --format tensorrt Export to TensorRT-LLM soup export --model ./output --format gguf --verbose Show export phases and stream converter/quantizer logs while they run (#1760) soup export --model ./output --format gguf --no-install-deps Never invoke pip; refuse missing conversion packages before merge/toolchain work (#1760) soup deploy ollama --model m.gguf --name x Deploy GGUF to Ollama soup deploy ollama --list List Soup-deployed models soup deploy ollama --remove Remove model from Ollama soup deploy hf-space --model user/m --space user/s --template gradio-chat|streamlit-chat [--hardware slug] Create HF Space (Streamlit deprecation warning in v0.78.0; use gradio-chat for new Spaces) soup deploy autopilot --target mac-m3|rtx-4090-24gb|... Pick PEFT+quant+spec-decoding for a hardware target soup deploy autopilot --list List all 10 deploy profiles soup agent synth --spec api.yaml -o ds.jsonl Parse OpenAPI/MCP/GraphQL spec into a tool-calling SFT dataset soup agent train --spec api.yaml --base model One-shot synth + planned soup train invocation soup agent eval --spec api.yaml --predictions p.jsonl Score predicted tool-calls vs spec catalog soup agent eval --spec api.yaml --predictions p.jsonl --sandbox Execute each tool-call in the RLVR sandbox: ok/tool_error/timeout/arg_error soup eval benchmark --model ./output Evaluate on standard benchmarks soup eval aider --model openai/gpt-4.1 --output ./aider-results --exercises-dir ./polyglot-benchmark Run Aider Polyglot in Docker soup eval custom --model ./output --tasks eval.jsonl Custom eval tasks from JSONL soup eval judge --target resp.jsonl LLM-as-a-judge evaluation soup eval auto --config soup.yaml Auto-eval from config soup eval compare Compare eval results soup eval leaderboard Local model leaderboard soup eval human --model-a ./a --model-b ./b --input p.jsonl Human A/B evaluation soup eval gate --suite gate.yaml Run eval-gate suite standalone soup eval quant-check --before X --after Y --tasks t.jsonl Before/after quantization eval (OK/MINOR/MAJOR verdict) soup eval retention --model --label v3 [--adapter ] [--add-task NAME=PATH ...] [--record] [--threshold 0.05] [--ledger retention.json] [--output r.json] [--config soup.yaml] [--chat-template-kwargs JSON] [--require-history] Accumulated forgetting across a series of versions: every control task vs its best-ever score (exit 0=no regression / 2=regression / 3=usage / 1=runtime) soup eval design --goal "..." Draft an eval suite from training data + goal soup eval discover Discover a held-out canary set soup eval lock Freeze a design as a checksummed suite soup eval coverage --task Coverage / gap analysis for an eval suite soup eval against --candidate Run-vs-run regression check (paired bootstrap) soup eval gate-install --baseline Install a pre-push regression gate soup eval behavior Behaviour battery pre/post diff soup eval capability Capability profile (MMLU-Pro / GPQA / AIME ...) soup eval checklist CheckList MFT / INV / DIR tests soup eval irt-subset Minimum-cost eval subset via IRT soup diagnose Post-training report card: forgetting / refusal / format / mode collapse / memorization / contamination soup serve --model ./output --port 8000 OpenAI-compatible API server soup serve --model ./output --backend vllm vLLM backend (2-4x throughput) soup serve --model ./output --backend sglang SGLang backend soup serve --model ./output --backend mii DeepSpeed-MII backend (live) soup serve --model ./output --speculative-decoding draft-model Speculative decoding soup serve --model --auto-spec Auto-pair draft model for speculative decoding soup serve --model --auto-quant Refuses until candidates can be loaded and measured; quantize explicitly instead soup serve --model --backend vllm --prefix-cache vLLM prefix caching (RAG/agent) soup serve --model --structured-output json --json-schema s.json Constrained output soup serve --model --structured-output regex --regex-pattern '...' Regex-constrained output soup serve --model --dashboard Live dashboard + /metrics endpoint (transformers + vllm only; warns on sglang/mii) soup serve --model --backend vllm --max-model-len 8192 Cap the vLLM sequence length (lower it when the KV cache does not fit) soup serve --model --trace --trace-endpoint http://localhost:4317 OpenTelemetry tracing soup serve --model --trace-log ./serve.jsonl Per-request JSONL log + rotation + secret redaction soup serve --model --record-thumbs ./rl.db Capture πŸ‘/πŸ‘Ž feedback into local-RL SQLite + POST /v1/thumbs (transformers) soup serve --model --kv-cache-type bf16|f16|q8_0|fp8 KV-cache type (transformers; q8_0 needs hqq; fp8 = vLLM+Hopper only) (v0.71.14) POST /v1/adapters/activate/ Hot-swap active LoRA adapter soup sweep --config soup.yaml --param lr=... Hyperparameter search soup sweep --config soup.yaml --sweep-config sweep.yaml Sweep from a standalone strategy/n_runs/seed/params file soup diff --model-a ./a --model-b ./b Compare two models soup data inspect [--image-dir ] View dataset stats; optionally override vision image directory soup data validate Check format (auto-detect) soup data doctor --model Chat-template compat report: 8 checks, OK/MINOR/MAJOR soup data doctor --model --show-mask N Per-token trained/masked colouring via the real collator soup data lint Preference-data linter: length bias, near-dups, chosen==rejected soup data clean [-o out] Sanitize dataset: control chars, zero-width spaces, empty turns soup data convert --to chatml Convert between formats soup data merge data1.jsonl data2.jsonl Combine datasets soup data dedup --threshold 0.8 Remove duplicates (MinHash) soup data dedup --semantic Dedup by embedding cosine β€” catches rewordings MinHash misses ([train]) soup data topics [--clusters N|auto] Cluster + c-TF-IDF labels + coverage table + thin-topic warnings ([train]) soup data canary insert -o --manifest Insert K secrets to later prove memorization (manifest = SECRET) soup data canary check --manifest --base Rank each secret's loss vs never-inserted controls; exit 2 = leak soup data stats Extended statistics soup data generate --prompt "..." --count 100 Generate synthetic data soup data generate ... --provider ollama Use local Ollama instance soup data generate ... --provider anthropic Use Claude API soup data generate ... --provider vllm Use local vLLM server soup data generate ... --template code Domain templates (code/conversation/qa/preference/reasoning) soup data generate ... --quality-pipeline Auto validate + filter + dedup soup data augment --strategy rephrase|translate|style [--provider ollama|vllm --model --base-url ] LLM-driven augmentation soup data from-traces --logs l.jsonl --format langchain --signal thumbs_up --output p.jsonl Preference pairs from traces soup data from-traces ... --judge --min-confidence 0.7 LLM-judge confidence filter soup data review prefs.jsonl --sample 10 Preview preference pairs soup data filter --coherence 0.3 Quality filter (perplexity/coherence) soup data sample --n 1000 Random sample subset soup data sample --n 1000 --strategy diverse Cluster-based diverse sampling soup data sample --n 1000 --strategy hard Sample hardest examples soup data sample --pct 10 Sample by percentage soup data split --val 10 --test 10 Split into train/val/test soup data split --val 500 --absolute Split with absolute counts soup data split --val 10 --stratify category Stratified by field soup data split --val 10 --stratify-semantic --num-clusters 5 Semantic stratified split soup data search "code instructions" Search HuggingFace Hub for datasets soup data search --sort likes --limit 10 Sort and paginate search results soup data preview teknium/OpenHermes-2.5 Preview remote dataset metadata soup data download user/dataset -o data.jsonl Download HF dataset as JSONL soup data download user/ds --samples 1000 Stream first 1000 samples soup data register --name my-ds --path d.jsonl --format alpaca Register dataset soup data unregister --name my-ds Remove from registry soup data push --input d.jsonl --hf-dataset user/name Upload local JSONL as HF dataset soup data push --input d.jsonl --hf-dataset u/n --hub modelscope|modelers Upload to an alternative hub soup data registry List all registered datasets soup data demo List bundled demo JSONL fixtures soup data demo alpaca_demo --output ./d.jsonl Copy a bundled demo JSONL fixture soup data ingest PDF/DOCX/MD/TXT -> JSONL (PDF/DOCX: pip install "soup-cli[documents]") soup data preprocess AOT-tokenize and cache for reuse across runs soup data recipe Validate / execute a Data Recipe DAG soup data mix BETA mixture-weight optimiser (proxy runs) soup data forge --docs ./docs --task sft --target-rows 1000 Synthetic data pipeline + provenance soup data forge --docs ./docs --hub modelscope --teacher owner/name Pre-fetch the teacher from an alternative hub soup data score --input rows.jsonl Composite quality scorecard (PII + keyword triage + lang + edu) soup data score --input rows.jsonl --benchmark-file benchmark.jsonl --threshold 0.8 Score with real n-gram comparison texts soup data score ... -b gsm8k --benchmark-file benchmark.jsonl Labels are validated only, not corpus selectors; all file texts are compared soup data decontaminate --input rows.jsonl --benchmark-file benchmarks.jsonl Drop rows overlapping the benchmark texts in that file soup data leakage --train train.jsonl --eval eval.jsonl [--json] [--samples 5] Read-only exact input-text overlap (0 clean, 2 overlap, 3 input) soup data toxicity --input rows.jsonl -o tox.jsonl Flag abuse-keyword matches (heuristic) soup data langdetect --input rows.jsonl -o tagged.jsonl Tag each row with language code soup data pii --input rows.jsonl -o pii.jsonl Flag rows containing email/phone/SSN/credit-card soup data educational --input rows.jsonl -o scored.jsonl Score educational value per row soup train --config soup.yaml --tracker mlflow MLflow / SwanLab / Trackio integration soup profile --config soup.yaml Estimate memory/speed before training soup profile --config soup.yaml --gpu a100 Estimate for specific GPU soup profile --config soup.yaml --json Profile JSON on stdout; diagnostics on stderr soup profile --model Qwen/Qwen2.5-7B-Instruct --gpu rtx5070 --method qlora Will a model fit, without a soup.yaml soup cost --config soup.yaml Estimate training cost in USD across providers soup cost --config soup.yaml --gpu H100 Estimate training cost for specific GPU soup cost --config soup.yaml --json Cost JSON on stdout; diagnostics on stderr soup prune-layer-cache --model org/base [--yes] List stale layer-stream cache slots; --yes removes them soup adapters list ./output/ Scan for LoRA adapters soup adapters info ./output/checkpoint-500/ Show adapter metadata soup adapters compare adapter1/ adapter2/ Compare two adapters soup adapters audit --config soup.yaml [--json] Compare a finished run's adapter_config.json against the config that started it (#762); a setting the record cannot speak to is 'unknown', never 'ok'. Exit codes 0=agreement, 2=DIVERGED, 1=usage error, so CI can tell a verdict from a mistyped path. soup adapters branches List snapshotted branches soup adapters checkout Restore a snapshotted branch's config soup adapters diff Per-layer Ξ”W Frobenius diff + effective-rank drift soup loop init --eval --baseline [--pre-wired] [--thumbs-db ] Create .soup/loop.yaml (data flywheel; --pre-wired = real stages; --thumbs-db = harvest `serve --record-thumbs` thumbs) soup loop status Counters + status + pre_wired flag soup loop watch [--detach] [--max-iterations N] [--pre-wired] [--pack-cans] [--tool-auth-token T] Harvest β†’ train β†’ gate β†’ deploy daemon (pre-wired stages + Soup Can packing) soup loop pause Atomic status flip: pause watch daemon at next iteration boundary soup loop resume Atomic status flip: resume a paused loop soup loop canary --traffic 5% Route keyed serve traffic to a loaded adapter; watch auto-rolls back on MAJOR soup loop replay [] [--extract ] Replay / unpack a recorded iteration manifest soup serve --model m --adapters chat=./c --adapters code=./d Multi-adapter serving soup migrate --from llamafactory config.yaml Import config from LLaMA-Factory soup migrate --from axolotl config.yml Import config from Axolotl soup migrate --from unsloth notebook.ipynb Import config from Unsloth notebook soup migrate --from llamafactory c.yaml --dry-run Preview without writing soup recipes list List all 180 ready-made recipes soup recipes show llama3.1-8b-sft Print recipe YAML soup recipes show qwen3.5-0.8b-dpo Print the tiny Qwen3.5 text-preference recipe soup recipes use llama3.1-8b-sft Copy recipe to soup.yaml soup recipes search "reasoning" Search by keyword/task/size soup recipes verify Attach every shipped config on the meta device; exit 2 if one cannot (#1116) soup recipes verify --config soup.yaml Same check for one config, before downloading weights soup recipes verify --no-templates Recipes only; skip src/soup_cli/templates and examples/ soup recipes verify --json One JSON row per config on stdout; advisories go to stderr soup registry push --run-id --name n --tag v1 Register run soup registry push ... --data train.jsonl Also record that file's SHA-256 as data_hash soup registry push ... --idempotency-key k Retry-safe: same key + same registration -> existing entry soup registry list [--name n] [--tag v1] List registry entries soup registry show Entry details + artifacts + ancestors soup registry diff Side-by-side config + eval delta soup registry search "medical" Search name/base/task/notes soup registry promote --tag prod Tag an entry (e.g. promote to prod) soup registry delete --yes Remove entry (cascades) soup history Lineage DAG tree for a name soup can pack --entry-id --out r.can Pack registry entry as .can soup can inspect r.can Preview manifest without extracting soup can verify r.can Verify schema + config parseability soup can fork r.can --out fork.can --modify training.lr=5e-5 Fork + re-pack soup can run r.can --yes [--deploy] [--env-capture env.txt] Run a .can end-to-end soup can publish r.can --hf-hub user/name Publish .can to HF Hub as dataset soup runs List training runs soup runs show Run details + loss graph + cost (shows an Error: line for failed runs, and distinguishes terminated/launching from running; #767) soup runs compare Compare two runs soup runs replay Replay summary + loss curve from history (also plots a benchmark-score curve when the metric lives in eval_results) soup runs clean [] [--all] Clean redundant checkpoint files soup runs curriculum-curve BETA curriculum bucket-weight plot soup runs delete Delete a run and its metrics soup why [run_id] Explain training anomalies (heuristic) soup rewind [run_id] [--step N] Name the dataset rows behind a loss spike soup ship --base --adapter --task-eval t.jsonl SHIP / DON'T-SHIP verdict: task win AND no regression on the bundled suite (exit 0=SHIP / 2=DON'T / 3=usage / 1=runtime) (v0.71.25; leg-2 real + usage-off-2 v0.71.38) soup ship --evidence ev.json [--output v.json] Decide offline from pre-computed scores (no model load) soup ship ... --progress jsonl Emit versioned completion events on stderr; keep the verdict on stdout (#1759) soup ship ... --task-mode judge_score --judge-model ollama://llama3.1 Leg-1 via LLM-as-a-judge soup ship ... --task-mode pairwise --judge-model ollama://llama3.1 Leg-1 via swap-debiased judge win-rate (base=0.5) (v0.71.31) soup ship ... # leg-2 default = 8 bundled offline suites (MCQ/arithmetic/over_refusal + tool_call/format_json/safety, extraction scorer, ~40 items each) (v0.71.38; +mini_over_refusal v0.73.2) soup ship ... --general-suite mmlu,gsm8k --baseline base.json lm-eval leg-2 override + recorded base scores soup ship ... --emit-evidence ev.json Re-serialise the scores as replayable --evidence input (output-is-input, #312) (v0.71.39) soup ship ... --output v.json `schema_version: 2` verdict JSON: `task_win.rows` (better / worse / same), `task_win.truncated` (answers cut off at the token cap, plus the `truncation_growth` warning), `task_win.generation_failures`, `judged` (adapter, checkpoint step); a warning never changes the verdict or exit code, a side whose every generation failed is refused (exit 1); `--emit-evidence` carries the same blocks, which soup-cli 0.77.x refuses as `unsupported field(s)` soup ship ... --config soup.yaml Read eval.ship gate defaults; --evidence GATES on provenance.config_sha and numerics family, --emit-evidence STAMPS both (v0.71.39 / #367); live-loads base/tuned at training.quantization when it is 4bit/8bit, else full precision (#367) soup ship ... --push owner/repo#N Post the verdict as a GitHub PR comment (best-effort; never flips the exit code) (v0.71.39) soup ship ... --noise-floor N Re-run base model N times; per-axis floor = max-min spread; gate at max(threshold, floor); leg-1 measured in every --task-mode (judge modes cost N judge passes, floor labelled decode+judge) (v0.73.2) soup ship ... --chat-template-kwargs '{"enable_thinking":false}' Pass template variables equally to base/tuned live generation; JSON object, live-only (#1758) soup card -o MODELCARD.md HF model card from a registry entry: training config, evals, hashes, lineage, artifacts (v0.71.35) soup push --model ./out --repo you/m --card Upload that registry-driven card as the README (HF only) (v0.71.35) soup ci init [--data d.jsonl --suite s.yaml --evidence ev.json] [--config soup.yaml] [--branch main --python 3.11] [--force] Write .github/workflows/soup-gate.yml: data validate -> expect -> ship gate on every PR (v0.71.35; --python range is 3.11 to 3.13); --config binds the gate to a committed config so it refuses stale evidence (v0.71.39) soup mcp serve MCP server over stdio (drive Soup from Claude Code / Cursor / Cline; requires [mcp] extra) (v0.71.28) soup mcp serve --allow-mutating Also expose plan-only train_start / export tools (never execute) (v0.71.28) soup mcp serve --allow-execute Implies --allow-mutating; enables train_execute / export_execute via server confirmation tokens soup mcp serve --transport sse [--host H --port N] Serve the same registry over HTTP+SSE instead of stdio; binds 127.0.0.1 and requires a Bearer token (#296) soup mcp serve --transport http [--auth-token T] Same over the streamable-HTTP transport (/mcp); --auth-token pins the token instead of generating one (#296) soup mcp serve --transport sse|http --allow-execute REFUSED - gated execution spawns real processes and is stdio-only (#296) soup mcp runs reconcile [--expunge-launching] [--older-than-seconds N] Clear stale launching rows left by a crashed server; refuses if any candidate PID is still alive (#402) soup shrink --model --drop-ratio 0.25 --calib c.jsonl -o shrunk Depth-prune least-important layer block + SHIP/DON'T-SHIP ppl verdict (exit 0/2/1) (v0.71.29) soup shrink ... --drop-layers N --heal h.jsonl --heal-steps 200 --device cpu Drop N layers + distill-heal (fuse LoRA back to one dense model) soup shrink ... --tolerance 0.10 --plan-only [--attach-to-registry ] Ppl-regression tolerance / print importance table only / registry attach soup draft measure --target --draft --prompts p.jsonl Draft acceptance rate + real plain-vs-assisted tok/s (exit 0 measured β€” a best-effort assisted-arm failure stays 0 and is recorded as `assisted_status`; 2 below `--min-acceptance`; 1 error before results) (v0.71.33) soup draft measure ... --min-acceptance 0.6 -o report.json Exit 2 below the floor (CI gate) / write the JSON report (fields incl. `assisted_status`: pending/complete/untimed/crash/interrupted) soup draft measure ... --sweep-k 1,2,3,5,8 Time assisted generation at each draft length (at most 8, each 1..64). The JSON adds `tok_s_draft`, `draft_status`, `latency_ratio`, `breakeven_acceptance`, `modelled_best_k`, `modelled_speedup_best_k`, `k_sweep` and `measured_best_k` (#843) soup draft distill --target --draft-base --data d.jsonl -o draft/ Distil a DENSE speculative-decoding draft + register it (v0.71.33) soup draft distill ... --steps N --device cpu --force --plan-only Training budget / device / overwrite -o / render the config only soup draft list List local drafts that `soup serve --auto-spec` will pick up (v0.71.33) soup reward synth refs.jsonl -o reward.py Synthesize a deterministic reward verifier from gold outputs (v0.71.40) soup reward synth ... --kind numeric|json_schema|regex|tool_call Force a verifier family (default: auto-detect) soup reward synth ... --plan-only Report the induced spec + calibration plan; write nothing soup reward synth ... --output-report r.json --min-discrimination 0.5 Save the calibration JSON / set the refusal threshold (exit 0 emit / 2 refuse / 1 error) soup reward stress reward.py --references golds.jsonl Adversarially probe a verifier for gameability β€” classic and structure-preserving attacks (v0.71.41) soup reward stress verifiable --verifiable-domain math --references golds.jsonl Probe a builtin verifier instead of a .py file soup reward stress ... --attacks empty,length,repetition,sentinel,wrapped_junk,answer_spray --sentinel GOLD --threshold 0.5 --max-gameable 0.0 Tune the attack set / accept threshold / tolerance soup reward stress ... --output-report r.json Save the per-attack report JSON (exit 0 robust / 2 gameable / 1 error) soup tui Full-screen Textual dashboard (requires [tui] extra) soup train --config soup.yaml --profile Record torch.profiler trace to /profiles/ soup --log-level quiet|normal|verbose|debug Global logging tier (Rich-formatted) soup ui [--port 7860] Web UI (experiments, training, data); loopback Host must name the bound host or localhost / 127.0.0.1 / [::1] with the bound port (omitted = HTTP 80), otherwise 403 soup ui --public [--auth-token T] Phone-scannable Web UI (v0.53.9); /docs + /openapi.json are loopback-only soup tokenizer train --input c.jsonl --vocab-size N Train BPE tokenizer (v0.53.9) soup bench Inference speed + memory (same as `soup bench infer `) soup bench infer --p50 --p95 Bench with tail-latency percentiles (v0.53.9) soup bench infer --cuda-graphs Bench with experimental CUDA graph decode soup bench train --config soup.yaml --steps 20 --warmup 3 -o bench-train.json Timed SFT steps; exits 1 when the model was not training (#836) soup bench --backend auto Auto-detect transformers/mlx backend (v0.53.9) soup serve --model ./output --reasoning-parser deepseek-r1 Strip blocks from responses (v0.53.9) soup docs search [-k N] [-f json] Offline BM25 search over the guides; -f json prints the hits as a JSON array soup doctor [--nccl] [--disk] [--config F] Check environment (optionally check NCCL bandwidth, media type; --disk ~9s cold / ~2.4s warm). --config also reports which settings that config switches on (a `false` or unset value is not reported; `seed: 0` is) that its task/backend does not read (#755, #1330); exits 2 if it cannot be read, and exits 1 when a required core dependency is missing, or when any installed package is beyond its declared ceiling β€” core or [train] (#828, #874). soup monitor NVIDIA / Apple Silicon GPU monitor: util / temp / VRAM / power soup quickstart [--dry-run] Full demo soup plugins list|install|enable|disable Manage Soup plugins soup llama cli|mtmd-cli|gguf-split|server|quantize ... Proxy to the llama.cpp binaries soup quantize --to Quantize a model β€” ergonomic alias for `soup export --format ` soup bom emit --name --base-sha --config-sha --format cyclonedx|spdx|both CycloneDX ML-BOM / SPDX AI bill of materials soup adapters scan Spectral backdoor scan (rank-1 dominance + outlier detection) soup adapters sign [--backend unsigned|ed25519|sigstore] [--key |--generate-key ] [--interactive-oidc] Merkle manifest + ed25519/Sigstore sign soup adapters verify [--strict] [--public-key ] [--cert-identity --cert-oidc-issuer ] Verify manifest + ed25519/Sigstore signature soup adapters check [--json] Is the adapter trained? Total/per-layer ||dW||_F, live fraction, all-zero lora_B, leaked .inner. keys; ends in alive or inactive: (exit 0 / 2; 1 = usage error) soup adapters check-safetensors [--strict] Refuse pickle / PyTorch-classic weights soup adapters merge ... [--license ] [--license-override ] [--allow-unscanned] License + backdoor-scan gates (auto-detect license; scan FAIL refused) soup adapters arithmetic "coder + 0.5*math - toxic" --adapter coder=

--adapter math=

--adapter toxic=

-o [--allow-unscanned --allow-cross-base] Task-vector algebra over LoRA adapters (add/scale/negate; same-rank; scan + same-base gated) (v0.71.34) soup attest emit ... [--sign unsigned|ed25519|sigstore] [--key ] [--interactive-oidc] [-o att.json] in-toto/SLSA-3 attestation; Sigstore requires -o soup attest verify --signature [--public-key ] [--cert-identity --cert-oidc-issuer ] Verify ed25519/Sigstore attestation signature soup airgap-bundle --model --output [--repro-receipt ] Signed tarball for data-diode transfer (embeds repro-receipt) soup train --config soup.yaml --annex-xi EU AI Act Annex XI/XII doc (markdown or PDF; top_domains auto-filled) soup train --config soup.yaml --track-energy [--energy-country USA] codecarbon offline kWh/CO2 β†’ annex-xi (pip install soup-cli[carbon]) soup train --config soup.yaml --track-energy --energy-out persist measurement for `soup bom emit --energy ` soup train --config soup.yaml --repro-receipt SR 11-7 reproducibility receipt soup can pack --entry-id --out r.can --attest Embed in-toto Statements into a v3 can manifest soup audit-log tail / rotate Tail / rotate the per-command HIPAA/SOC2 audit log (~/.soup/audit.jsonl) soup --no-audit-log / SOUP_NO_AUDIT_LOG=1 Opt out of the per-command audit line soup eval unlearning --benchmark tofu|muse|wmdp Forget Quality + Model Utility + PrivLeak verdict soup edit set --base --method rome|memit|alphaedit|grace --subject "..." --target "..." [--output

] [--device cpu] [--governor/--no-governor] [--registry-id ] [--cov-corpus ] Live surgical knowledge edit (GPT-2 Conv1D + Llama; --cov-corpus = covariance-preconditioned ROME, rome-only; --plan-only available) soup edit diff --probes p.jsonl --before-model --after-model Knowledge-injection diff (live before/after generation across probe prompts) soup train # task: unlearn NPO/SimNPO/RMU unlearning from data.forget_set (+ optional data.retain_set) soup train # data.format='raft' Answer-only span-mask RAFT training (golden+distractor docs, [doc-N] citations); generator-stage configs auto-link the latest RA-DIT retriever soup ra-dit --retriever-config --generator-config [--retriever-model ] [--plan-only] One-shot two-stage RA-DIT: train retriever β†’ record pairing β†’ train generator soup eval citation [--style bracket|inline|footnote] [--shuffle-seed N] [--output o.json] Citation precision/recall/F1 over predictions or RAFT rows soup steer train --base --method caa|iti|repe --name --pairs Fit a CAA/ITI/RepE activation-steering vector from {positive, negative} pairs soup steer apply --name --strength Preview a stored steering vector soup steer list List locally-stored steering vectors soup serve --steer [--steer-strength ] Apply a steering vector at decode time via a forward hook (transformers backend) soup serve --bank [--bank-strength ] Multi-tenant VeRA/VB-LoRA serving; active user per request via X-User-Id header, ContextVar-isolated (v0.71.12 / v0.71.17) soup serve --mole Serve a trained MoLE: base + N frozen task LoRAs + mole_gate.pt, blended per-token at decode (transformers-only) (v0.71.17) soup ingest --source langfuse|langsmith|helicone|openpipe|otel|openai-stored --logs Universal trace importer (6 SaaS adapters β†’ normalised JSONL) soup ingest --source langfuse --pull [--since 7d --max-pages 100 --allow-private-host] Live pull of Langfuse generations (Observations API v2; LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY / LANGFUSE_HOST) (#204) soup prune-prompt --input --output --min-frequency 0.95 Detect + strip shared system-prompt prefix soup prune-prompt ... --tokenizer Tokenizer-aware prefix detection (decodes remaining ids, boundary-safe) soup data active-sample --input --output --budget N Top-N uncertain prod traces for human review soup ab --input --metric latency|judge_score|retry_rate Two-sided sequential A/B harness on latency / judge_score / retry_rate (Bayes factor with a confidence-sequence accept rule; decision: continue / reject_h0 / accept_h0; a reject_h0 reports direction better / worse; valid when re-run after every new row; the first 5 rows per arm set the --effect-size scale) (v0.63.0) soup ingest|prune-prompt|ab|data active-sample ... --slack-url | --discord-url Shared SSRF-validated webhook on completion soup drift-alarm --reference --live --threshold 0.2 Rolling-KL drift alarm (exit 3 on drift) soup drift-alarm ... --slack-url | --discord-url Optional SSRF-validated webhook on drift detected soup tunability --list List built-in candidate-base catalogue soup tunability --dataset [--candidates a,b,c] Probe candidate bases + Pareto frontier report soup tunability --dataset --live [--device cpu] LIVE per-candidate LoRA probe (loads each repo) soup plan --config soup.yaml [--minillm-on-policy] Pre-flight summary + write soup.tfstate soup apply --config soup.yaml [--dry-run] [--minillm-on-policy] Record reviewed plan; refuses on drift (exit 3) soup env lock | status | check Hermetic env lockfile + ABI drift + declared-bound violation detection (exit 3) soup env fix [--format uv-pip|requirements] [--output req.txt] Render a reproducible install plan from soup-env.lock (print-only) soup completions bash | zsh | fish Shell completion script (sourceable via eval) soup license-advisor --target b2c|defense|embedded Recommend license-clean base for deploy target soup license-advisor ... --license --mau N Per-license downstream-risk check (exit 3 on block) soup probe sae-diff [--top-k N] SAE feature diff between pre/post-FT activations (v0.66.0) soup probe sae-diff --auto-download Fetch an allowlisted SAE into ~/.soup/sae-cache (v0.71.8) soup probe sleeper [--evidence ev.json] [--weights w.npz] [--output o.json] Sleeper-agent defection probe; --weights = real calibrated probe (v0.66.0; v0.71.8) soup probe truth [--evidence ev.json] [--weights w.npz] [--output o.json] TruthfulQA-style honesty probe (v0.71.8) soup probe harm [--evidence ev.json] [--weights w.npz] [--output o.json] HarmBench-style misuse probe (v0.71.8) soup probe interference [--output o.json] Pairwise NΓ—N adapter interference matrix (exit 2 on MAJOR; v0.66.0) soup probe interference --measure --base-model --adapter name=path ... [--device cpu] Auto-measure live interference (v0.71.8) soup probe pack [--output o.json] Per-base calibrated probe pack manifest (v0.66.0; +truth/harm v0.71.8) soup probe pack --list List bundled probe-pack bases (v0.66.0) soup train --capture-activations --capture-prompts Post-train SAE-diff-ready per-token activation snapshot (v0.71.8) soup adapters blame ... --top-k 50 Live DataInf-style influence runner (v0.66.0, closes #171) soup adapters merge ... --strategy cmaes --eval --budget 1h CMA-ES evolutionary merge β€” live loop (v0.67.0 schema / v0.71.4 live) soup adapters merge ... --canary [--strict-verdict] Live OK/MINOR/MAJOR canary verdict, exit 2 on MAJOR (v0.71.4) soup adapters pr --base-sha <hex> --adapter <path> GitHub-shaped adapter PR Markdown / JSON (v0.67.0) soup adapters pr <title> ... --push owner/repo#N Post the PR as a GitHub comment via gh api (v0.71.4) soup adapters branch <name> --from-registry <id> | --attach-to-registry <id> Branch ↔ Registry lineage (v0.71.4) soup adapters bisect <ckpt>... --eval-command "..." Binary search over training history (v0.67.0) soup lock write --base-sha <h> --dataset-sha <h> --env-hash <h> Write soup.lock (v0.67.0) soup lock write --base-sha <h> --dataset-sha <h> --env-lock soup-env.lock Auto-derive --env-hash from soup-env.lock (v0.71.1) soup lock show [PATH] Print tracked lock file soup lock check [PATH] --base-sha <h> --dataset-sha <h> --env-hash <h> --base-model <m> Refuse with exit 2 on drift, 3 on usage/missing lock soup compile <program.py> --eval <suite> [--optimizer mipro|gepa|textgrad|copro|bootstrap_fewshot] [--plan-only] DSPy / GEPA / TextGrad prompt-program compiler β€” live (v0.71.13; pip install "soup-cli[compile]") soup distill-prompt --traces <jsonl> --teacher <m> --student <m> --strategy sft|preference|kl [--provider ollama|anthropic|vllm] [--base-url <url>] [--temperature F] [--max-rows N] Distill prompt-heavy traces via a live teacher (v0.71.13); a teacher/student call that fails or returns nothing drops that trace, and the command exits 1 without writing the file when nothing is left; when no trace row has a prompt it exits 3 before any provider call and leaves an existing output untouched, and the done panel counts rows skipped for want of a prompt soup compile-tools <spec.json|yaml> --eval <jsonl> [--optimizer textgrad|gepa] [--task-lm <m>] [--reflection-lm <m>] [--engine <name>] [--plan-only] TextGrad / GEPA tool-schema optimiser β€” live (v0.71.13; pip install "soup-cli[compile]") soup apple-adapter <source-dir> --direction hf-to-mlx|mlx-to-hf|hf-to-apple|mlx-to-apple --output <dir> [--sign] [--plan-only] PEFT LoRA <-> mlx-lm adapter conversion β€” live (v0.71.21; *-to-apple upstream-gated exit 3) soup local-rl init --db <path> Create personal-LLM flywheel SQLite schema (v0.68.0) soup local-rl status --db <path> Print interactions / thumbs-up / thumbs-down counters soup local-rl record --db <path> --prompt <q> --response <r> --thumb up|down Append thumbs record soup local-rl harvest --db <path> -o <pairs.jsonl> Harvest DPO pairs from thumbs into JSONL soup local-rl train --db <path> --model <id> --once [--train-method dpo|kto|orpo] [--min-pairs N] [-o <dir>] Ad-hoc DPO/KTO/ORPO train from harvested thumbs β€” live (v0.71.13) soup local-rl train --db <path> --model <id> [--scheduler-dir <dir>] [--hour H] [--minute M] [--train-method dpo|kto|orpo] [--min-pairs N] [-o <dir>] Render a systemd/launchd nightly-train scaffold (no --once) that runs `--once` with the same flags (v0.71.13) soup build <manifest.yaml> [--dry-run] [--output-dir <dir>] dbt-for-SFT DAG: validate + plan + live materialise (v0.69.0; live v0.71.6) soup expect <data.jsonl> <suite.yaml> [--judge URL] Expectations suite: PII / token-length / refusal / judge (v0.69.0) soup data gen-magpie --base <m> --provider ollama|vllm --target N --output <jsonl> [--base-url <url>] [--quality-filter] Magpie synthetic generator β€” live (v0.69.0; live v0.71.6) soup data best-of-n (--base <m> | --provider ollama|vllm --model <m> [--base-url <url>]) --prompts <jsonl> --n 8 --judge <url> -o <sft.jsonl> [--emit-pairs <dpo.jsonl>] [--resume] [--checkpoint <journal.jsonl>] [--manifest <manifest.json>] Best-of-N rejection sampling with durable per-prompt recovery and manifest-last publication soup data best-of-n (--base <m> [--revision <rev>] | --provider ollama|vllm --model <m>) --prompts <jsonl> --n 8 --export-candidates <jsonl> [--checkpoint <jsonl>] [--resume] Resumable sampling-only phase; no judge is constructed soup data best-of-n --candidate-artifact <jsonl> --judgments <jsonl> -o <sft.jsonl> [--emit-pairs <dpo.jsonl>] [--manifest <json>] Validate offline judgments and materialize byte-stable training rows; a final manifest binds the requested output set soup data evolve --input <seeds.jsonl> --provider ollama|vllm --model <m> --strategy depth|breadth --rounds N -o <jsonl> Evol-Instruct (WizardLM) instruction evolution (v0.71.31) soup data persona-mix --prompts <jsonl> --n N --output <jsonl> Persona-Hub diversity sampler (v0.69.0) soup data brain-rot <data.jsonl> [--strict] Brain-rot detector β€” arXiv 2510.13928 (v0.69.0) soup iterative-dpo --base-model <m> --reward-model <rm> --prompts <p.jsonl> --output-dir <out> --rounds N --pairs-per-round N [--plan-only] Iterative DPO loop driver β€” LIVE sampleβ†’scoreβ†’pairβ†’train (v0.70.0; live v0.71.11) soup train --reward-hack-detector info_rm|rm_ensemble [--reward-hack-halt] Reward-hacking detector for GRPO β€” LIVE callback (v0.70.0; live v0.71.11) soup train --reward-hack-mitigation off|log_only|kl_control|pid_lagrangian Closed-loop reward-hacking auto-mitigation (detect β†’ raise KL/Ξ² β†’ rollback β†’ early-stop); GRPO only, requires --reward-hack-detector; refused on task: ppo (v0.71.26) soup train --config soup.yaml # training.uld_strategy: wasserstein_aligned Cross-tokenizer ULD on task='distill' (different tokenizers) β€” LIVE (v0.71.18) soup train --config soup.yaml # training.minillm_enabled: true [minillm_teacher_mix_ratio 0.3] MiniLLM reverse-KL distillation, config-only β€” LIVE; offline mix 0 rejected (v0.70.0; live v0.71.11; #692, #979) soup train --config grpo.yaml # training.rl_checkpoint_save_every_steps: N [rl_checkpoint_keep_last: N; rl_checkpoint_include_optimizer: true] Mid-epoch checkpoint for GRPO/PPO β€” LIVE (v0.70.0; live v0.71.11) soup train --config grpo.yaml # training.echo_trap_enabled: true [echo_trap_threshold: 0.6; echo_trap_halt: true] RAGEN echo-trap detector for GRPO β€” LIVE callback (v0.70.0; live v0.71.11) soup train # task='moe_lora_routing' + mole_task_adapters MoLE per-token gate over N frozen task LoRAs (gate-only train) β€” LIVE (v0.71.12) soup train # task='distill' + distill_mode=token|sequence Token logit-KL or sequence-level teacher-continuation KD β€” LIVE (v0.71.12) soup train # task=classifier|reranker|cross_encoder + lora LoRA-adapter classifier (frozen encoder) β€” LIVE (v0.71.12) soup train # use_mod | expand_layers Mixture-of-Depths / LLaMA Pro (Llama/Qwen/Mistral) β€” LIVE (v0.71.12). use_mod is applied only on the resident transformers LoRA route of task sft or tts (text, no layer streaming, lora.r >= 1, no LISA or Spectrum) and on transformers pretrain (LoRA or LISA); every other route is refused at config load (#1752) soup train # use_longlora LongLoRA SΒ² is refused at config load: the override leaked future tokens and applied no SΒ² grouping; use rope_scaling_type with plain LoRA (#1240) soup train # task='tts' + tts_family + modality='audio_out' Codec-string SFT + emotion templating β€” LIVE (v0.71.20); native Sesame CSM with raw audio + 32 Mimi codebooks (#265); Spark/Oute raw-audio refused soup train # task in {sft,tts} + moe_expert_quant=nf4|int8_rowwise [+moe_lora] bnb per-expert quant of fused-MoE experts (CUDA); refused on other tasks, which never applied it (#798) β€” LIVE (v0.71.20) soup train # task in {sft,tts} + train_router_only=true [+moe_lora] Freeze MoE experts, train only the gating router; refused on other tasks (#798). moe_lora needs lora.dropout: 0.0 on fused-expert MoEs β€” LIVE (v0.71.20) soup train # quantization='bitnet_1.58' BitNet 1.58 training is not implemented; config load refuses it soup export --model ./output --format bitnet|tq1_0 BitNet 1.58 TQ1_0 ternary GGUF via llama.cpp β€” LIVE (v0.71.20) soup version [--full] [--json] Show version (--full: system info, --json: JSON output) soup --version Show version (prints the same text as soup version) soup --verbose <command> Full traceback on errors ``` ### Choosing the device for `soup infer`, `soup chat` and `soup diff` `--device` decides where these commands load the model. Without it, the device is auto-detected and the same rules apply to what was detected (this also covers `soup bench`, which loads through the `soup infer` path). | `--device` | Placement | Dtype | |---|---|---| | `cpu` | pinned to the CPU | `float32` (twice the RAM of `float16`: a 7B model needs about 28 GB) | | `cuda` | `device_map="auto"`: accelerate may use every visible GPU and spill to the CPU if the model does not fit | `float16` | | `cuda:N` | pinned to GPU `N` | `float16` | | `mps` | pinned to the Apple GPU | `float16` | | `mlx` | runs on the CPU on this path | `float32` | Values are case-insensitive. Any other value (`gpu`, `auto`, `cuda:0,1`, ...) is refused with an error instead of being ignored. ### Interactive Autopilot (`soup autopilot`) When standard input is an interactive terminal and any of `--model`, `--data`, or `--goal` are omitted, `soup autopilot` prompts for the missing values: - **Model**: Base model Hugging Face repository ID (or local path). Blank entries are refused. - **Data**: Local training dataset path (JSONL). Must exist under current working directory. - **Goal**: One of `chat`, `reasoning`, `code`, `classification`, `tool-calling`, `alignment`, or `domain-adapt`. When run with all three options supplied, or when standard input is not a terminal (e.g. CI, scripts, or pipelines), `soup autopilot` never prompts: complete flags execute directly, and omitted required flags exit with code 2 and the standard `Missing option` error. ### Best-of-N recovery and publication `soup data best-of-n` appends and synchronizes one checkpoint record after each fully sampled and judged prompt. The default journal is `<output>.checkpoint.jsonl`; override it with `--checkpoint`. If a sampler or judge backend fails, Soup reports the completed prompt count and journal path. Rerun the same command with `--resume` to reuse that exact prefix without sampling it again. The prompt sequence and generation options are bound into the journal, so changing them makes resume fail closed. Final SFT and optional DPO JSONL files are published only after every prompt is complete. The manifest (default `<output>.manifest.json`, or `--manifest`) is written last and records their row counts and SHA-256 digests. Consumers should treat only files listed by that final manifest as committed output. Invalid local data such as a non-finite judge score remains a validation failure rather than being presented as a recoverable backend outage. For offline Best-of-N, the candidate and judgment artifacts are the recovery checkpoint. Offline materialization rejects `--resume` and `--checkpoint`; rerun the exact materialization command after a late failure. The final manifest is written last and is the only commit marker; consumers must verify it and use only the SFT/DPO files it lists. ## Fine-tune from your coding agent (MCP) `soup mcp serve` runs a [Model Context Protocol](https://modelcontextprotocol.io) server over **stdio** (default) or a local **SSE / HTTP** listener, so any MCP client β€” Claude Code, Cursor, Cline, Continue β€” can drive Soup conversationally. Install the extra first: ```bash pip install "soup-cli[mcp]" ``` Register it with your client. For **Claude Code** (`.mcp.json` in the repo) or the **Claude Desktop** config (`claude_desktop_config.json`): ```json { "mcpServers": { "soup": { "command": "soup", "args": ["mcp", "serve"] } } } ``` ### Remote or multi-client: the SSE and HTTP transports stdio suits a client that spawns Soup as a subprocess. For a client on another machine, or several clients sharing one server, run a listener instead: ```bash soup mcp serve --transport sse # GET /sse + POST /messages/ soup mcp serve --transport http # streamable HTTP on /mcp soup mcp serve --transport sse --host 127.0.0.1 --port 8765 --auth-token "$TOKEN" ``` Both bind `127.0.0.1:8765` by default and **require a Bearer token on every request**. With `--auth-token` omitted, a fresh one is generated at startup and printed to stderr; it is 16-128 urlsafe-base64 characters, the same shape `soup ui` uses. Point a client at it with a header: ```json { "mcpServers": { "soup": { "url": "http://127.0.0.1:8765/sse", "headers": { "Authorization": "Bearer <token>" } } } } ``` The registry is identical across transports β€” same tools, same schemas, same `--allow-mutating` / `--allow-execute` gating. Only the wire changes. **Security:** - **The token is required and there is no opt-out.** A loopback listener is reachable by every process on the machine, not just by you. - **The token travels in the header, never the URL.** There is deliberately no query-string fallback, so it cannot be captured in proxy or access logs. - **DNS-rebinding protection is on.** A request whose `Host` is not the address the server bound to is refused with `421`, a foreign `Origin` with `403`. This is the gate the Bearer token cannot be: a web page the operator merely visits attaches no `Authorization` header, but its request still arrives at the port. - **A non-loopback `--host` prints a warning**, and a wildcard bind (`0.0.0.0`) prints a second one β€” with no single advertised name there is nothing to pin, so the Host check degrades to accepting any `Host`. - **`--host` / `--port` / `--auth-token` are refused under `--transport stdio`** rather than silently ignored: stdio has no listener and nothing to authorize. The network transports need `mcp >= 1.10.0` β€” streamable HTTP landed in 1.8.0 and rebinding protection in 1.10.0 β€” which is the floor `soup-cli[mcp]` pins. The server exposes 14 read-only tools β€” `advise`, `data_inspect`, `data_validate`, `data_score`, `data_doctor`, `recipes_search`, `recipes_show`, `runs_list`, `runs_show`, `registry_list`, `registry_show`, `profile`, `diagnose_evidence`, `ship_evidence` β€” each returning JSON. Two **plan-only** mutating tools (`train_start`, `export`) are gated behind `--allow-mutating` (`"args": ["mcp", "serve", "--allow-mutating"]`); when `--allow-mutating` alone is active, they only render the exact command that would run β€” they never execute training or export. `--allow-execute` implies `--allow-mutating` and enables full background subprocess execution via two execution tools (`train_execute` and `export_execute`). Execution requires a server-issued one-time confirmation token returned during the planning phase (`train_start` or `export`). The token state is kept in-memory with a 5-minute TTL and is consumed before subprocess invocation. **Execution Security & Boundaries:** - **Flag Safety:** `--allow-execute` is default-off and dangerous. `--allow-mutating` alone can NEVER trigger subprocess execution. - **One-Time Confirmation Tokens:** Authorization requires a server-issued random token. Client confirmation is UX-only; security relies entirely on the server-side token state. - **Subprocess Isolation:** Execution runs the Soup CLI as an isolated subprocess (`shell=False`, `stdin=DEVNULL`, `cwd` pinned to server startup directory). Child stdout/stderr is redirected to `.soup/mcp-runs/<run_id>.log` to avoid corrupting the MCP JSON-RPC stdio stream. Planning (`train_start` config snapshot) and execution are refused if `.soup` or `.soup/mcp-runs` is a symbolic link or junction, or if a hard link exists at the run log path. Planning also refuses a link or anything already present at the run directory `.soup/mcp-runs/<run_id>`: the snapshot is written only into a run directory the plan has just created, with `config.yaml` created exclusively and without following a link. - **Concurrency & Disconnects:** Enforces 1 active execution at a time, gated on a persisted run whose process is still alive β€” so a **restarted** server does not launch a second training while a child from a previous server is still running, and a stale record whose process is gone never blocks execution. A `launching` record (committed before the child process exists, pid not yet written) blocks restart capacity too: it is indistinguishable from "about to spawn", so treating it as live is the safe direction after a crash in that window. If a server crash leaves that row behind, `soup mcp runs reconcile --expunge-launching` removes rows older than five minutes and prints every removed `run_id`; use `--older-than-seconds N` to choose another positive threshold. The command refuses the whole operation if any candidate records a PID that is still alive. Launches run in background (fire-and-forget). Disconnecting the MCP client does not terminate an already-running subprocess. - **Config Snapshotting & Input Revalidation:** At plan time (`train_start`), the validated config is snapshotted to `.soup/mcp-runs/<run_id>/config.yaml`, and execution uses this snapshot rather than the original mutable config path. External protected inputs (such as datasets and model/checkpoint directories or files) are not frozen in the snapshot; instead, their content digests (computed via SHA-256 for regular files, or deterministic recursive content hashing over sorted relative file paths for directory trees, bounded by file-count and total-byte safety limits) are recorded at plan time and revalidated immediately before spawn. Modifying an external protected input between plan and execute invalidates the token. Modifying the original config path after planning has no effect because execution strictly uses the snapshotted config. Snapshotting freezes only the configuration itself, not external filesystem assets. - **Every Local Input Is Pinned:** A train plan pins every local file or directory the run reads (model, datasets, reward/teacher/PRM models, reward `.py` files, unlearning sets); an input outside the working directory is refused at plan time, and an input that did not exist at plan time must still not exist at execution. ## Gate and Verdict Exit Codes Soup gate and verdict commands follow a unified, CI-friendly exit-code contract: | Exit Code | Constant | Meaning | Examples | |---|---|---|---| | `0` | `EXIT_OK` | PASS / OK / SHIP | Model passes gate; no regression; lock matches closure; valid dataset | | `2` | `EXIT_GATE_FAILED` | GATE FAILED / REGRESSION / DRIFT / DON'T SHIP | Metric regressed; behavior/checklist MAJOR; lock closure drifted; dataset unusable | | `3` | `EXIT_USAGE_ERROR` | USAGE / INPUT / CONFIG ERROR | Bad or invalid CLI flag; missing or unparseable input file; empty series | | `1` | `EXIT_RUNTIME_ERROR` | RUNTIME ERROR | Unexpected crash, environment incompatibility, or live inference error | The taxonomy applies consistently across `soup ship`, `soup eval gate`, `soup eval against`, `soup eval checklist`, `soup eval behavior`, `soup eval quant-check`, `soup lock check`, `soup expect`, `soup data validate`, `soup data lint`, and `soup recipes verify`. `soup eval checklist` requires `--evidence`. `soup eval behavior` also requires `--evidence` unless `--base-model` selects the live path; the `elephant` and `syceval` batteries cannot be scored live and exit `3` with `--base-model`. Omitting the required evidence exits `3` and names the JSON input to provide; it never reports a neutral pass for a gate that measured nothing. ### Gate Verdict Drift vs. Operational Drift `soup lock check` operates as a CI gate command where exit `2` (`EXIT_GATE_FAILED`) indicates that the lock closure has drifted from the configuration (requiring regeneration via `soup lock write`), and exit `3` (`EXIT_USAGE_ERROR`) indicates an unparseable or missing lock file. In contrast, operational and environment commands (`soup env check`, `soup apply`, and `soup drift-alarm`) belong to the operational drift family where runtime divergence or ABI mismatch signals exit `3` (with exit `2` reserved for invalid arguments or path errors in those commands). Operational inspection tools such as `soup adapters audit` deliberately reside outside the gate taxonomy, exiting `1` on usage errors so CI can distinguish an agreement (`0`) or diverged (`2`) verdict from invocation errors.