# DeepSeek-V4-Flash-DSpark on 4× RTX PRO 6000 — stock vLLM 0.25.1, no fork, no patch

**`deepseek-ai/DeepSeek-V4-Flash-DSpark` (FP8 linears + MXFP4 experts, ~157 GB) serving on 4× RTX PRO 6000 Blackwell
(sm_120), on stock vLLM 0.25.1. No fork. No patch. No Docker.** Two `pip install`s and one `vllm serve`. It is
coherent, does full OpenAI + Anthropic tool calling, and decodes at **~133 tok/s**.

This repo is the honest **stock-path** recipe. It also tells you exactly where the stock ceiling is — **256K
single-request context, no speculative decoding** — *why*, what does **not** lift it, and which forks do. Everything
below is measured on real hardware on 2026-07-18.

> Base sm_120 support for this model is upstream work (**vLLM #43477**, merged) — this repo is not that, and claims no
> kernel work. It is the released-wheels recipe plus the two limits the stock path still has, each with its root cause,
> so you don't lose an afternoon rediscovering them.

---

## Results

Model: [`deepseek-ai/DeepSeek-V4-Flash-DSpark`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark) —
`DeepseekV4ForCausalLM`, FP8 block-scaled (ue8m0) linears + MXFP4 experts (256 routed / 6 active), ~157 GB / 48 shards,
DeepSeek Sparse Attention, `max_position_embeddings` 1,048,576.

Hardware: 4× NVIDIA RTX PRO 6000 Blackwell, 96 GB each, sm_120, PCIe, **no NVLink**.

| Metric | Measured |
|---|---|
| Context served (single request) | **256,000 tokens** (`--max-model-len 262144`) |
| KV cache pool @ 256K | **2,923,171 tokens** (11.15× concurrency) |
| KV cache pool @ 1M `--max-model-len` | 6,525,363 tokens (6.22×) — pool holds 1M, but a single prefill >256K crashes ([BUGS](docs/BUGS.md)) |
| Decode throughput | **~133 tok/s** single-stream (via `usage.completion_tokens`) |
| Prefill | ~12,500 tok/s @ 256K |
| Needle-in-haystack | **PASS** @ ~119K and ~238K input tokens |
| Tool calling | **OpenAI single / parallel / chained + Anthropic `tool_use` — all pass** |
| Weights / resident | 38 GB / GPU · ~90 GB / GPU @ 256K |
| Warm start | ~2–4 min (sm_120 autotune cache persists) |

Full numbers and method: **[docs/BENCHMARKS.md](docs/BENCHMARKS.md)**.

---

## Is this you?

Pasted one of these into a search engine? See **[docs/BUGS.md](docs/BUGS.md)** for the full trace, cause, and the
thing that actually fixes (or doesn't fix) it.

- **`719 (CUDA_ERROR_LAUNCH_FAILED)` from `deepgemm-src/.../runtime_utils.hpp:144`**, and a GPU dropping off the bus
  (`Unable to determine the device handle for GPU N: Unknown Error`) when you push context past ~256K → the stock
  single-request ceiling (Limit 1). It is **not** a chunk-size problem, and `VLLM_USE_DEEP_GEMM=0` doesn't fix it.
- **`Check failed: num_tokens > 64 ... must go through sparse_mla_sm120_decode_dsv4; got num_tokens=1`** with
  `--speculative-config` → spec-decode isn't on the stock sm_120 path (Limit 2).
- **`RuntimeError: 'dispatch_scaled_mm ... scaled_mm_helper.hpp:17'`** after you set `VLLM_USE_DEEP_GEMM=0` → that flag
  is a dead end here; the E8M0 FP8 linears have no non-DeepGEMM sm_120 kernel.
- **`NotImplementedError: DeepGEMM MegaMoE requires SM100 GPUs`** → use `--kernel-config '{"moe_backend":"marlin"}'`.

---

## Quickstart

Assumes a working NVIDIA driver + CUDA toolkit (`nvcc` on `PATH`) and the checkpoint downloaded.

```bash
export MODEL=/path/to/DeepSeek-V4-Flash-DSpark   # the checkpoint directory
export VENV=$HOME/venvs/ds4                       # a dedicated venv
export PORT=8000

bash scripts/setup-venv.sh "$VENV"                # stock vllm 0.25.1 + flashinfer 0.6.14
MODEL="$MODEL" VENV="$VENV" PORT="$PORT" bash scripts/serve.sh
```

`scripts/serve.sh` is, in full:

```bash
export FLASHINFER_DISABLE_VERSION_CHECK=1     # flashinfer-python 0.6.14 vs cubin 0.6.13 (no 0.6.14 cubin on PyPI)
export NCCL_P2P_DISABLE=1                     # Blackwell PCIe P2P allreduce deadlocks without this
export CUDA_HOME=/usr/local/cuda; export PATH="$CUDA_HOME/bin:$PATH"   # nvcc for the MHC-prenorm TileLang JIT

"$VENV/bin/vllm" serve "$MODEL" \
  --served-model-name deepseek-v4-flash \
  --host 127.0.0.1 --port 8000 \
  --tensor-parallel-size 4 --enable-expert-parallel \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.90 \
  --kernel-config '{"moe_backend":"marlin"}' \
  --enable-auto-tool-choice --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 \
  --trust-remote-code
```

A healthy boot auto-selects the sm_120 backends with no extra flags: `Detected ... scale_fmt=ue8m0; enabling UE8M0 for
DeepGEMM`, the MHC-prenorm TileLang kernel JIT-compiles, `Using 'MARLIN' Mxfp4 MoE backend`, and
`Autotuning FlashInfer SM120 sparse MLA DSv4 decode`. Verify:

```bash
BASE=http://127.0.0.1:8000 python3 bench/toolcalls.py     # tool calling: OpenAI + Anthropic
BASE=http://127.0.0.1:8000 python3 bench/needle.py 128000 0.5   # long-context retrieval
```

---

## The ceiling — read before you raise `--max-model-len`

The KV pool easily holds a million tokens, so it is tempting to set `--max-model-len 1048576`. **Don't serve single
requests past 256K on stock.** A single prefill in that range crashes with a DeepGEMM launch failure (719) that takes
a GPU off the PCIe bus. Full evidence in **[docs/BUGS.md](docs/BUGS.md)**; the short version:

- It is **not** prefill chunk size — it crashes identically at `max_num_batched_tokens` 8192 **and** 512. The fault
  scales with *total* sequence length.
- **`VLLM_USE_DEEP_GEMM=0` won't even boot** — the E8M0 FP8 linears fall to CUTLASS, which asserts on sm_120.
- DeepGEMM upstream has no sm_120 plans, so there is no stock fix. **1M single-request context needs a fork** — see
  [docs/PRIOR-ART.md](docs/PRIOR-ART.md).

**Speculative decode** (MTP or DSpark) also isn't on the stock sm_120 path — it dies at init because the sm_120 sparse
decode kernel wants ≥64-token decode batches and spec verification submits 1–7. Serve base. vLLM **PR #41834** (open)
adds it.

---

## Tool calling & Claude Code

Both API surfaces work with `--tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4`:

- **OpenAI** `tool_calls`: single, parallel, and chained (tool result → final answer) all pass.
- **Anthropic** `/v1/messages`: `stop_reason=tool_use` with correct `tool_use` blocks.
- DeepSeek-V4 **emits a reasoning block first**, so clients must send a generous `max_tokens` (≥1024) or a small budget
  can return a silently empty answer.

If you point **Claude Code** at this server, note that recent Claude Code sends a `role:"system"` message *inside*
`messages[]` and an `output_config.effort` value vLLM rejects — both make the session hang rather than error. The fix
is the usual one: a small request-normalizing reverse proxy in front of vLLM that moves non-`user`/`assistant` messages
into the top-level `system` field and clamps the effort value.

---

## What this is NOT

- **Not a fork.** Stock `pip install vllm==0.25.1` + a flashinfer version bump. Nothing patched.
- **Not the base enablement.** sm_120 DeepSeek-V4 support is upstream (vLLM #43477, merged; PR #41834, open). This repo
  is the *released-wheels recipe + honest limits*, not the kernels.
- **Not a new quantization, and not a model release.** The checkpoint is DeepSeek's; no weights are hosted here.
- **Not the highest-context or fastest path.** It stops at 256K with no spec-decode by design — see below.

## Going further than stock

| | This repo | fork paths |
|---|---|---|
| Install | `pip install vllm==0.25.1` + flashinfer bump | source build / Docker |
| Single-request context | 256K | up to 1M |
| Speculative decode | none (stock kernel gap) | MTP / DSpark / EAGLE |
| Framework | stock vLLM | vLLM PR #41834 branch · B12X fork · SGLang |

See **[docs/PRIOR-ART.md](docs/PRIOR-ART.md)** for the fork map (jasl's vLLM PR #41834, `0xSero/deepseek-v4-flash-sm120`,
`hikarioyama/dsv4-flash-nvfp4-sm120`, `sakamakismile/DSv4-Flash-FP8-SM120-Configs`).

---

## Credits

- **vLLM #43477** (merged) — the base sm_120 DeepSeek-V4 support that makes stock 0.25.1 run this model.
- **vLLM PR #41834** (`jasl`, open) — the sm12x DeepSeek-V4-Flash "stock-deps" path that adds MTP2 + DSpark
  speculative decode; the fix-in-flight for the two limits documented here.
- **vLLM #40802** — the SM120 DeepSeek-V4 tracking issue.
- **[0xSero](https://github.com/0xSero/deepseek-v4-flash-sm120)**, **[hikarioyama](https://github.com/hikarioyama/dsv4-flash-nvfp4-sm120)**,
  **[sakamakismile](https://huggingface.co/sakamakismile/DSv4-Flash-FP8-SM120-Configs)** — community fork/framework recipes that go further.

See [NOTICE](NOTICE). Licensed Apache-2.0 ([LICENSE](LICENSE)).

## Docs

| | |
|---|---|
| **[docs/BUGS.md](docs/BUGS.md)** | The two stock limits in full: traces, root causes, and what does *not* fix them. |
| **[docs/BENCHMARKS.md](docs/BENCHMARKS.md)** | KV/concurrency/decode numbers, needle, tool-calling, method. |
| **[docs/PRIOR-ART.md](docs/PRIOR-ART.md)** | Why stock works, where it stops, and the forks that go further. |