# Supported Models & Optional Extras [← Back to the Soup README](../README.md) > Recommended model families, the VRAM size guide, and the pip extras matrix. > Model sizes across recipes represent the total parameter count (all resident parameters for MoE models, answering card VRAM requirements). ## Supported Models Soup works with **any** of the **340,000+** text-generation models on [HuggingFace Hub](https://huggingface.co/models?pipeline_tag=text-generation). If a model supports `AutoModelForCausalLM`, it works with Soup — zero config changes needed. ### Recommended Models | Model Family | Models | Sizes | Best For | |---|---|---|---| | **Llama 4** | Llama-4-Scout-17B, Llama-4-Maverick-17B | 109B-400B | General, multilingual | | **Llama 3.x** | Llama-3.1-8B-Instruct, Llama-3.3-70B-Instruct | 1B–70B | Chat, instruction following | | **Llama 3.2 Vision** | Llama-3.2-11B-Vision-Instruct, Llama-3.2-90B-Vision | 11B–90B | Image understanding | | **Gemma 3** | Gemma-3-4B-IT, Gemma-3-9B-IT, Gemma-3-27B-IT | 4B–27B | Efficient, multilingual | | **Qwen 3.5 / 3.6 / 3.8** | Qwen3.5-0.8B…397B-A17B, Qwen3.6-27B/35B-A3B, Qwen3.8-27B | 0.8B–397B | 262K context, native vision, MoE | | **Qwen 3** | Qwen3-8B, Qwen3-14B, Qwen3-32B, Qwen3-235B-A22B | 0.6B–235B | Reasoning, code, MoE | | **Qwen 2.5** | Qwen2.5-7B-Instruct, Qwen2.5-Coder-32B-Instruct, Qwen2.5-Math-7B-Instruct | 0.5B–72B | Code, math | | **DeepSeek** | DeepSeek-R1-Distill-Llama-8B, DeepSeek-V3-0324, DeepSeek-V4-Flash/Pro | 1.5B–1.6T | Reasoning (GRPO), code, MoE | | **GLM** | GLM-4.6, GLM-5, GLM-5.1 | 357B-754B | Chinese + English, MoE | | **Kimi** | Kimi-K2.5, Kimi-K2.6 | ~1T (MoE) | Long-context agentic, MoE | | **MiniMax** | MiniMax-M2, MiniMax-M3 | 230B–428B | Agentic, MoE (community license) | | **Phi-4** | Phi-4-14B, Phi-4-mini-reasoning | 3.8B–14B | Compact reasoning | | **Mistral** | Mistral-7B-Instruct-v0.3, Mistral-Small-24B | 7B–24B | Fast, efficient | | **Mixtral** | Mixtral-8x7B-Instruct-v0.1, Mixtral-8x22B | 47B–141B | MoE architecture | | **CodeLlama** | CodeLlama-7b-Instruct-hf, CodeLlama-34b-Instruct | 7B–34B | Code generation | | **StarCoder 2** | StarCoder2-15B, StarCoder2-7B | 3B–15B | Code completion | | **Yi** | Yi-1.5-34B-Chat, Yi-1.5-9B-Chat | 6B–34B | Multilingual chat | | **InternLM 3** | InternLM3-8B-Instruct | 8B | Chinese + English | | **Falcon** | Falcon-11B, Falcon-40B-Instruct | 7B–180B | Open-weight | **Mistral-Large-3 is not shipped as a recipe.** `mistralai` publishes it in Mistral's native layout (`params.json` + `consolidated-*.safetensors`), with no `config.json`, which `AutoConfig` cannot read, and no flag changes that. A Hugging Face-format conversion exists on the Hub as `FriendliAI/Mistral-Large-3-675B-Instruct-2512-BF16`; it is a third party's re-upload, so Soup does not point a recipe at it (#1145). **Resident Kimi K2 trains natively for `sft` and `grpo` on Kimi-K2-Base and Kimi-K2-Instruct.** The known Hugging Face Base and Instruct IDs declare `model_type: kimi_k2` and an `auto_map` onto repository code that uses removed transformers APIs, including `is_torch_fx_available`, and an inference-only MoE training path, so a plain resident load still fails and `trust_remote_code` does not repair that. For `task: sft` and `task: grpo` only, Soup instead passes the checkpoint's `config.json` through the field-by-field guard of [#2107](https://github.com/MakazhanAlpamys/Soup/pull/2107) into transformers' own `DeepseekV3Config` (rewritten in memory; the config read passes `trust_remote_code=False`, so no repository code is executed) and loads the safetensors natively, dequantising each FP8 block weight to bf16 per tensor at load. The route requires `training.quantization: none` — any other value is refused at config validation — and the tokenizer is still repository code on this route: it needs `--trust-remote-code` and `pip install tiktoken` to load. Migration: for `task: sft` and `task: grpo` with `modality: text` on the transformers backend, a local directory whose `config.json` declares `model_type: kimi_k2` no longer loads through `from_pretrained`; it loads through the guarded native route or is refused naming every failing field, and `training.rope_scaling_type` on it is refused; other tasks, other model types and `training.stream_layers: true` are unchanged. The load never holds a second whole model; its largest transient is one layer's fused routed experts at the fuse step (about 33.8 GB per layer at published K2 scale), not a single tensor's working set. A `quantization_config` menu entry (the 4-bit path) is refused on this route: 4-bit K2 stays streamed-only. The other trainers still refuse K2 bases on the resident path at config validation, as do the Instruct-0905 and Thinking IDs for every task: they have no supported route yet. This refusal does not qualify local copies, other hubs or Unsloth as working routes. K2.5/K2.6 use native `kimi_k25` registration and are outside this boundary. For a Hub checkpoint, the resident route checks available cache space against the files not already cached at the resolved revision before downloading. With `HF_HUB_OFFLINE=1`, it uses the local snapshot without requesting Hub metadata. For a 4-bit K2 base, use the guarded `training.stream_layers: true` route added by [#2107](https://github.com/MakazhanAlpamys/Soup/pull/2107). It requires `--trust-remote-code` for the tokenizer and `pip install tiktoken`; see [streaming guide, Kimi K2 paragraph](performance-and-quantization.md#layer-streaming-beta-v0720-nf4-v0722-disk--wider-archs-v0723-preference-losses-v0724) for the guard and prerequisites. Its evidence is CPU-only parity on a tiny K2-like checkpoint; full K2 weights, training and GPU execution remain unverified. The resident route's own evidence is CPU-only parity against K2's own pinned modelling code on a tiny K2-shaped checkpoint ([gate](../benchmarks/gate-kimi-k2-resident.md)); routing at real scale is unverified. `kimi-k2-sft` stays unavailable: its YAML carries `quantization: 4bit`, which this route refuses at config validation (the refusal names the remedy — `training.quantization: none` for the resident route, or 4-bit through `training.stream_layers: true`, where the tokenizer still needs `--trust-remote-code` and `pip install tiktoken`). `kimi-k2-thinking-grpo` remains discoverable with an unsupported resident-route description: `soup recipes show` preserves its YAML for inspection and `soup recipes use` exits 3 (`EXIT_USAGE_ERROR`) with the per-recipe unavailability reason, writing no config. Catalog snapshots record both unavailability reasons. Full K2-weight validation remains separate work. Qwen3.5, Qwen3.6, and Qwen3.8 checkpoints advertise a multimodal conditional-generation architecture on the Hub, but Soup's catalog recipes are deliberately text-only. Their explicit `modality: text` selects `AutoModelForCausalLM`, which instantiates the language decoder without the visual tower. This is appropriate for text-only SFT, pre-training, DPO and GRPO data; it is not a full multimodal fine-tune. The Transformers backend requires Transformers 5.16.1 or newer, which also supplies the `qwen4_exp` text decoder used by Qwen3.8-Flash-Next (#571). On Torch runtimes whose `scatter` operator still requires `int64` indices, Soup widens only the QSA indexer's local index tensors before LoRA injection; Torch and the upstream Transformers class remain unchanged. `soup recipes use qwen3.5-0.8b-dpo`, `qwen3.5-2b-dpo` and `qwen3.5-4b-dpo` select the 0.8B, 2B and 4B language decoders for text preference pairs (`prompt`, `chosen`, `rejected`). All three inherit the shipped Qwen3.5-9B DPO settings: `lr: 5e-6`, three epochs, LoRA rank 16 / alpha 32, 4-bit quantization and a 4096-token limit. These are inherited starting values, not measured 0.8B, 2B or 4B tuning recommendations. The model's visual tower is excluded; image preference data is not supported by these recipes. The 2B config's native text architecture and LoRA attachment have been checked on the meta device, with a tiny random decoder's forward/backward pass checked on CPU. This does not establish full-checkpoint 2B training quality, NF4 execution, or GPU memory use. `soup recipes use qwen3.5-4b-grpo` selects the 4B language decoder for text reasoning data in `./data/reasoning_train.jsonl`. It inherits the 0.8B, 2B and 9B GRPO starting settings, including four generations, the `accuracy` reward, eight accumulation steps and gradient checkpointing. Supply a reference answer for each prompt, either as an `answer` column or as the final assistant message. The native decoder and LoRA path are checked with the public configuration on meta and a tiny random model on CPU. Full 4B GRPO training, NF4 execution, GPU memory use and reward quality need separate validation. `soup recipes use qwen3.5-9b-pretrain` selects `Qwen/Qwen3.5-9B-Base` for continued pre-training on plaintext rows in `./data/corpus.jsonl`. Its values are the shipped `qwen3.5-4b-pretrain` row's, inherited and not tuned for 9B: one epoch, `lr: 1e-4`, `batch_size: auto`, eight accumulation steps, LoRA rank 32 / alpha 64 on `target_modules: auto`, 4-bit quantization and a 4096-token limit. The config's native text architecture and LoRA attachment are checked on the meta device, with a tiny random decoder's forward/backward pass checked on CPU. Full-weight training, NF4 execution, a real pre-training run and memory fit are not checked. `soup recipes use qwen3.5-27b-dpo` selects `Qwen/Qwen3.5-27B`'s language decoder for the same text preference layout. It inherits the 9B DPO recipe's 4096-token limit, learning rate, epochs, LoRA and quantization settings. It uses eight accumulation steps, matching the Qwen3.5 27B SFT, Qwen3.6 27B DPO and Gemma 3 27B DPO recipes. These settings are starting values. Full-size tuning and memory fit have not been measured. The pinned public configuration builds on the meta device, and a tiny random decoder with both attention block kinds passes a CPU LoRA forward/backward check. Full 27B weights, NF4 execution, GPU memory use and training quality are unverified. `soup recipes use qwen3.5-27b-grpo` selects the same 27B language decoder for `./data/reasoning_train.jsonl`, with a reference answer in an `answer` column or the final assistant message. It inherits the 4B/9B GRPO starting values: `lr: 1e-5`, three epochs, four generations, `grpo_beta: 0.1` and the `accuracy` reward. The 4096-token limit, auto batch size, eight accumulation steps, LoRA rank 16 / alpha 32 and 4-bit quantization also match the shipped 27B SFT/DPO recipes; gradient checkpointing is explicit. These are inherited starting values. The pinned public configuration's full text architecture and LoRA attachment have been checked on meta, and a tiny random hybrid decoder passes a CPU LoRA forward/backward check. This does not validate full 27B weights, GRPO rollouts, NF4 execution, GPU memory fit or training quality. `soup recipes use qwen3.5-122b-a10b-dpo` selects the 122B-A10B language decoder for `prompt`, `chosen` and `rejected` text preferences in `./data/preference_train.jsonl`. It inherits the 122B SFT recipe's one epoch, batch size 1, 16 accumulation steps, LoRA rank 32 / alpha 64, gradient checkpointing, 4-bit quantization and 4096-token limit, with the 35B MoE DPO recipe's `lr: 5e-6` and `dpo_beta: 0.1`. These are starting settings, not measured 122B tuning recommendations. `moe_lora: true` includes the routed experts and `lora.dropout: 0.0` permits PEFT's fused-expert adapters. Model quantization is distinct from the SFT-only `moe_expert_quant` and `train_router_only` options; DPO does not apply Soup's `moe_aux_loss_coeff` setting. The pinned public configuration builds the complete native text MoE on the meta device with the recipe's LoRA attached, and a tiny random decoder retaining linear attention, full attention and expert routing passes a CPU forward/backward check with attention and expert adapter gradients. The visual tower is excluded. Full 122B weights, DPO training, NF4 execution, GPU memory fit and tuning quality remain unverified; meta tensors do not allocate the public checkpoint's weights. `soup recipes use qwen3.5-122b-a10b-grpo` selects the 122B-A10B language MoE for text reasoning rows whose final assistant message holds the reference answer. The shipped `format: auto` path is tested with that ChatML layout. It inherits one epoch, batch size 1, sixteen accumulation steps, LoRA rank 32 / alpha 64, zero dropout, a 4096-token limit, 4-bit quantization and checkpointing from the 122B SFT recipe. The 35B MoE GRPO sibling supplies learning rate 1e-5, four generations, beta 0.1 and the `accuracy` reward. `moe_lora: true` includes fused experts; the SFT-only auxiliary-loss option is omitted. These are untuned starting values. GRPO promotes the configured batch size of 1 to at least 4 for four generations, so it is not a one-completion workload. Ten billion active parameters do not mean ten billion resident parameters: the whole 122B base, rollout state and reference-model requirements matter. No GPU count or memory-fit recommendation has been measured. The public config was verified at `dc4d348443bc740c68e2d77492492c11606384d5`; offline tests use a tiny random native hybrid text MoE with expert LoRA. They do not qualify full 122B weights, tokenizer/rollout behavior, full GRPO training, NF4 execution, distributed training, GPU memory fit or reward quality. `soup recipes use qwen3.5-397b-a17b-grpo` selects the 397B-A17B language MoE for the same text reasoning rows: the reference answer is the final assistant message or an `answer` column. It inherits one epoch, batch size 1, 32 accumulation steps, LoRA rank 32 / alpha 64, zero dropout, a 4096-token limit, 4-bit quantization and checkpointing from the 397B SFT recipe. The Qwen3.5 MoE GRPO rows (35B-A3B, 122B-A10B) supply learning rate 1e-5, four generations, beta 0.1 and the `accuracy` reward. `moe_lora: true` includes fused experts; the SFT-only auxiliary-loss option is omitted. These are untuned starting values, and GRPO promotes the configured batch size of 1 to at least 4 for four generations. Seventeen billion active parameters do not mean seventeen billion resident parameters: the whole 397B base, rollout state and reference-model requirements matter, and the SFT sibling already needs multi-node DeepSpeed or FSDP. No GPU count or memory-fit recommendation has been measured. The public config was verified at `8472618112abcbd45acbcdc58436aff4233c23f7` (60 text layers, 512 experts, 10 active); offline tests use a tiny random native hybrid text MoE with the same three-linear-to-one-full attention pattern and expert LoRA. They do not qualify full 397B weights, tokenizer/rollout behavior, full GRPO training, NF4 execution, distributed training, GPU memory fit or reward quality. With `training.stream_layers: true`, the Qwen4-Exp text decoder is admitted by its own parity-tested path. Its very large frozen PLE N-gram table is excluded from the decoder-layer shard and can be served from the original safetensors with `training.stream_ngram_source: disk`. Reads are sparse and read-only; Soup does not create a second dense PLE copy in its shard cache. Dense Transformers checkpoints and oMLX/oQ affine Qwen4 bundles are accepted by this layer-streamed SFT path. For oQ, Soup dequantizes each frozen decoder layer into the reusable stream cache and maps the fused Switch-MLP expert tensors to the Transformers text decoder. The packed PLE table remains in the original checkpoint and only selected rows are dequantized, so oQ requires `stream_ngram_source: disk` (or `auto`). Vision-tower and MTP weights are not part of the text-only CausalLM and are ignored. `training.quantization` must remain `none`; streamed NF4 parity is separate and is not validated for Qwen4. The Qwen4 resident-versus-streamed parity oracle currently covers float32 CPU. BF16 parity on CUDA has not been measured, so CUDA BF16 production readiness is still pending even though the runtime selects BF16 there. The 176.9B-parameter production oQ checkpoint completed cache construction and training setup on an M4 Max but not a full optimizer step; it is not validated as trainable on a 128 GiB Mac. See the [M4 Max gate record](../benchmarks/gate-qwen4-ple-m4-max.md). ### Vision Models (with `modality: vision`) | Model | Size | Supported Formats | |---|---|---| | LLaMA-3.2-11B-Vision-Instruct | 11B | LLaVA, ShareGPT4V | | Qwen2-VL-7B-Instruct | 7B | LLaVA, ShareGPT4V | | Pixtral-12B-2409 | 12B | LLaVA, ShareGPT4V | ### ASR Models (`task: asr`, Whisper — v0.71.32) | Recipe | Base | Size | Status | |---|---|---|---| | `whisper-tiny-asr` | openai/whisper-tiny | 39M | Live on 4 GB | | `whisper-base-asr` | openai/whisper-base | 74M | Live on 4 GB | | `whisper-large-v3-asr` | openai/whisper-large-v3 | 1.5B | Parse-only (larger GPU) | Rows are `{"audio": , "text": }` with `data.format: asr`. See [Training → ASR fine-tuning](training.md). ### Quick Size Guide | VRAM | Max Model (QLoRA 4-bit) | Example | |---|---|---| | 8 GB | ~7B | Llama-3.1-8B, Mistral-7B | | 16 GB | ~14B | Phi-4-14B, Qwen2.5-14B | | 24 GB | ~34B | CodeLlama-34B, Yi-1.5-34B | | 48 GB | ~70B | Llama-3.3-70B | | 80 GB+ | 70B+ (full) or MoE | Mixtral-8x22B, DeepSeek-V3 | > **Note:** Soup auto-detects your GPU and estimates the optimal batch size. Use `soup doctor` to check your setup. ## Optional Extras > **Python 3.11, 3.12 or 3.13.** Since v0.77 the package declares > `requires-python = ">=3.11,<3.14"` (v0.73.0 to v0.76: `">=3.10,<3.13"`). Those are exactly > the versions CI tests. Python 3.10 reached end of life on 2026-10-01; on it, pip keeps > installing v0.76. Without the upper bound, pip on an untested Python resolved PyTorch wheels > nobody had validated, and the failure was not a Soup error message — it was a loader crash > inside `c10.dll` / `libc10.so` before any Soup code ran. If you are on 3.14+, create a 3.13 > environment; support widens when CI does. ### Quoting the extra **Use double quotes.** `pip install "soup-cli[train]"` is the only spelling that works in every shell — `cmd.exe`, PowerShell, bash, and zsh. Every command in the table below uses it. Older tutorials and videos (including some of ours) show the single-quoted `pip install 'soup-cli[train]'`. That is bash / zsh / PowerShell syntax, and it fails on Windows `cmd.exe`, which has no single-quote quoting and hands the quotes straight to pip: ``` ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at the start of dependency specifier ``` If you hit that, swap the `'` for `"` — pip is rejecting a literal quote character, nothing is wrong with the package. (Dropping the quotes entirely works on Windows too, but zsh then reads `[train]` as a glob and fails.) ### The extras table The core `pip install soup-cli` is a light install — the CLI, config system, and data tools, with no PyTorch. Add `[train]` to fine-tune, or install other extras only when you need them: Every row below is written with `pip`, and works verbatim inside a virtualenv, a Colab notebook or a Docker image. Outside one, on Debian 12 / Ubuntu 23.04 or later, `pip` refuses with `error: externally-managed-environment`; substitute `pipx` or `uv tool` for `pip` and the extras spelling is unchanged. See [the README's install section](../README.md#1-install). | Extra | Install | What it adds | |---|---|---| | `train` | `pip install "soup-cli[train]"` | Training stack: torch, transformers, peft, trl, datasets, bitsandbytes, accelerate | | `all` | `pip install "soup-cli[all]"` | `train` + `serve` + `ui` + `data` in one shot | | `fast` | `pip install "soup-cli[fast]"` | Unsloth backend (2-5x faster, lower VRAM) | | `vision` | `pip install "soup-cli[vision]"` | Vision / multimodal fine-tuning (Pillow) | | `audio` | `pip install "soup-cli[audio]"` | Audio / speech fine-tuning (librosa, soundfile) | | `mlx` | `pip install "soup-cli[mlx]"` | Standalone Apple Silicon backend for SFT, DPO, ORPO, SimPO, KTO and GRPO with local data; `[train]` is not required | | `qat` | `pip install "soup-cli[qat]"` | torchao: FP8 training (`quantization_aware: fp8`), FP8 attention, NVFP4, the `ao_adamw_*` optimizers and `soup export --format torchao` | | `serve` | `pip install "soup-cli[serve]"` | Inference server (FastAPI + uvicorn) | | `serve-fast` | `pip install "soup-cli[serve-fast]"` | vLLM inference backend (2-4x throughput) | | `sglang` | `pip install "soup-cli[sglang]"` | SGLang inference backend | | `ui` | `pip install "soup-cli[ui]"` | Web UI + inference server | | `tui` | `pip install "soup-cli[tui]"` | Full-screen Textual dashboard (`soup tui`) | | `eval` | `pip install "soup-cli[eval]"` | Benchmark evaluation (lm-evaluation-harness) | | `aider` | `pip install "soup-cli[aider]"` | Aider CLI; Polyglot evaluation also needs Aider's source-built Docker image | | `data` | `pip install "soup-cli[data]"` | Deduplication (MinHash via datasketch) | | `documents` | `pip install "soup-cli[documents]"` | PDF / DOCX ingestion (pypdf, python-docx); Markdown and TXT need no extra | | `data-pro` | `pip install "soup-cli[data-pro]"` | Language detection + PII (langdetect, presidio) | | `deepspeed` | `pip install "soup-cli[deepspeed]"` | Multi-GPU training (DeepSpeed ZeRO) | | `liger` | `pip install "soup-cli[liger]"` | Liger Kernel fused ops | | `ring-attn` | `pip install "soup-cli[ring-attn]"` | Ring FlashAttention (sequence parallelism) | | `onnx` / `tensorrt` | `pip install "soup-cli[onnx]"` | ONNX / TensorRT-LLM export | | `trackers` | `pip install "soup-cli[trackers]"` | MLflow / SwanLab / Trackio logging | | `remote` | `pip install "soup-cli[remote]"` | Remote datasets (s3 / gs / az / oci) | | `dev` | `pip install "soup-cli[dev]"` | Tests + lint + types (pytest, ruff, mypy, pre-commit) | The complete, authoritative extras list is in [`pyproject.toml`](../pyproject.toml).