# DGX Spark

[README](../README.md) | [Getting started](../README.md#start-here)

Use the NVIDIA driver and CUDA development
toolkit for the machine; the build needs `nvcc` and cuBLAS.
Check that `nvidia-smi` sees the GPU before building.

## Build and run

```sh
make cuda-spark
./download_model.sh ds4f-q2
./ds4 --cuda
```

The build selects `sm_121` and enables the Blackwell-specific kernels.
Do not use `--cuda-tensor-parallel` on a single Spark.
Stop other inference services before loading a model so they do not compete
for memory. Restore any services you stopped when finished.

## DeepSeek V4.1 Flash

Q2 needs SSD streaming on one 128 GB Spark:

```sh
./download_model.sh ds41f-q2
./ds4 --cuda -m gguf/DeepSeek-V4.1-Flash-Q2.gguf \
  --ssd-streaming --ctx 32768
```

The file is 341 GiB, including 189 GiB of disk-only Engram tables. Keep it on
a fast local SSD. The expert cache is sized automatically; leave memory for
other programs and the context. Use the same options with `ds4-agent` or
`ds4-server`.

Two Sparks can instead keep half the experts each, using
[network tensor parallelism](DISTRIBUTED.md#tensor-parallelism-between-two-sparks).
Both need the complete GGUF on their local SSD. V4.1 CUDA vision and DSpark
are not supported. The [model guide](MODELS.md#deepseek-v41-flash) covers
thinking levels and the separate Metal configurations.

September 13-14, 2026, Q2 SSD with automatic cache sizing: a 3,241-token
`/read README.md` prefill reached 93-96 t/s, up from 49-54 t/s, with 32K
allocated context. No extra flag is needed. A separate 64 GiB cache-budget
run with 64K allocated context reached 384 t/s for a 32K initial prefill and
88 t/s for a 3.2K append. These are SSD-streamed V4.1 results, not resident
V4 Flash numbers; speed depends on prompt length and the expert cache.
The automatic-cache 256-token reply test decoded about 5% slower, but
prefill plus the reply fell from about 87 to 59 seconds.
See the [QA record](../QA_BEFORE_RELEASES.md#cuda-ssd-streaming) for conditions
and longer-context measurements.

On two Sparks over RoCE, September 13 measurements averaged **21.9 t/s** for
one session: Q2, a 1K prompt, 64K allocated context and 2,048 teacher-forced
decode tokens. This does not use speculative decoding. Separately, eight
ready sessions reached about **28 aggregate t/s**, not 28 t/s per client. Decode batching
starts at five ready sessions; smaller groups run in order. Separate long
runs measured about 400 t/s for a 32K prefill and 290 t/s for an 8K append. The
[network TP QA record](../QA_BEFORE_RELEASES.md#cuda-network-tensor-parallelism)
has the timing controls and memory limits.

## Qwen3.8 Flash Next

Both Q2 and Q4 fit on one Spark. After building:

```sh
./download_model.sh qwen38-q2
./ds4 --mtp --ctx 32768
```

Q2 keeps 41.73 GiB of weights resident; Q4 keeps 69.74 GiB. Both GGUFs also
contain 95.37 GiB of BF16 n-grams, read directly from the SSD. Leave room
for context and runtime buffers. Vision, the native agent, server APIs and
text-session checkpoints work on CUDA; see [Qwen setup](QWEN38_FLASH_NEXT.md).

September 15, 2026, Q2 on one Spark: a 16K prefill reached 250 t/s, extending
it to 30K reached 243 t/s, and ordinary decode stayed around 17.3 t/s.
These are single runs with 8,192-token prefill chunks and 32 decode tokens.
A separate 256-token prose run measured 17.1 t/s ordinarily and 19.9 t/s with
MTP, at temperature zero and 8K allocated context. Speculation is opt-in;
its benefit depends on the prompt.

## GLM 5.3 Flash

Q2 is the resident target for one Spark:

```sh
./download_model.sh glm53-q2
./ds4 --cuda -m gguf/GLM-5.3-Flash-Q2.gguf --ctx 16384
```

Q4 does not fit resident. GLM Spark-to-Spark tensor parallelism is not
implemented; the two-Mac RDMA instructions do not apply to GLM on CUDA.

## Vision and speculative decoding

DeepSeek Vision Experimental needs its matching text model and encoder:

```sh
./download_model.sh ds4f-vision-q2
./ds4 --cuda --vision gguf/DeepSeek-V4-Flash-Vision-Encoder.gguf
```

Use `/read image.png` in the CLI. GLM vision also works on this backend; see
[models and vision](MODELS.md#vision).

For Flash 0731, [DSpark](SPECULATIVE_DECODING.md) uses the separate 0731 support
file. Vision Experimental has a different drafter. GLM uses its built-in MTP
block with `--mtp`. None is enabled by default.

## Flash Q2 with DSpark

September 6, 2026, fully resident Flash 0731: median generation speed from
three runs of 256 tokens, 4K allocated context and a 512-token prefill chunk.

| Prompt | Temperature | Ordinary decode | DSpark |
| --- | ---: | ---: | ---: |
| C hash table | 0 | 19.72 t/s | 31.41 t/s |
| C hash table | 1 | 19.53 t/s | 29.98 t/s |
| Unpredictable prose | 1 | 19.53 t/s | 18.81 t/s |

Longer coding runs reached about 33 t/s. Poor draft acceptance can still make
DSpark slower. Temperature 1 uses the default opportunistic policy; these are
not exact-sampling results. DSpark does not accelerate prefill.

To reproduce the greedy coding case after downloading `ds4f-q2` and
`ds4f-dspark`:

```sh
./ds4 --cuda -m ds4flash.gguf \
  --dspark --mtp-model gguf/DeepSeek-V4-Flash-DSpark-support-0731.gguf \
  --ctx 4096 --prefill-chunk 512 --nothink --temp 0 --seed 12345 -n 256 \
  -p 'Write a complete C hash table implementation with string keys, insert, find, delete, and a test main. Output only C code.'
```

For the temperature-1 rows, use `--temp 1 --top-p 0.95 --min-p 0.05`.
The [QA record](../QA_BEFORE_RELEASES.md#16-speed-regression) includes the previous
implementation, continued-context checks and quality comparisons.

## Larger models and serving

CUDA also has [SSD streaming](SSD_STREAMING.md) paths for larger weights.
Memory fit and speed depend on the model layout; start with Q2 for normal use.

V4.1 Q2 SSD serving batches up to eight decode rows together. An eight-session
test reached 11.0 aggregate t/s versus 8.4 with ordered execution; this is
total throughput, not speed per client. Other single-Spark model paths retain
their existing scheduling.
See [serving](SERVER.md) and the recorded [benchmarks](PERFORMANCE.md).