# qwen38-4090

Run **Qwen3.8-27B** on a single **RTX 4090** — 128K context, vision, and
speculative decoding, all at once, in 22.6 GB of the card's 24 GB.

Measured on this setup:

| | |
|---|---|
| Decode | **~104 tok/s** (44.8 without speculative decoding) |
| Context | **128K** (160K ceiling; 256K without speculative decoding) |
| Prompt processing | ~1800 tok/s at 120K tokens |
| Vision | on, and speculative decoding still applies to image requests |
| VRAM | 22648 MiB of 24564 |

Everything here is a wrapper around upstream [llama.cpp][llama] — there is no
custom inference code. The value is the configuration, which is not obvious, and
the measurements behind it.

[llama]: https://github.com/ggml-org/llama.cpp

## Requirements

- **RTX 4090 / 24 GB Ada (sm_89)**, headless. Every default here was measured on
  that card and tuned to fill it. A card driving a desktop has less free VRAM;
  drop `CTX` if you hit OOM.
- NVIDIA driver new enough for CUDA 13.x.
- Linux, x86_64. `setup/00-cuda.sh` assumes Ubuntu 24.04 or a derivative; on
  anything else, install a CUDA toolkit matching your driver by hand.
- ~40 GB free disk for weights.

Other 24 GB cards (3090, A5000) will likely work but are untested — they are a
different architecture, and `setup/01-llama-cpp.sh` compiles for sm_89 only.

## Setup

```sh
./setup/00-cuda.sh        # CUDA 13.2 toolkit (Ubuntu/Debian)
./setup/01-llama-cpp.sh   # build llama.cpp for sm_89
./setup/02-models.sh      # fetch weights (~20 GB, or ~21.5 GB with the drafter)
./bin/serve-qwen38.sh
```

Server listens on `0.0.0.0:8080`, OpenAI-compatible at `/v1`.

Paths are overridable; defaults are `$HOME/code/llama.cpp` and
`$HOME/models/qwen3.8-27b`:

```sh
LLAMA_DIR=/opt/llama.cpp MODEL_DIR=/data/qwen ./bin/serve-qwen38.sh
```

## What it runs

- **Weights:** `unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL` (17.9 GB)
- **Speculative decoding:** the model's **built-in MTP head**, embedded in the
  same GGUF as `blk.64.nextn.*` — no sidecar drafter file, no duplicated weights
- **Vision:** `mmproj-F16.gguf` (~1.1 GB in practice)
- **KV cache:** q4_0, which is what buys 128K
- **Context:** 128K

```sh
./bin/serve-qwen38.sh          # MTP + vision, 128K — the daily driver
./bin/serve-qwen38.sh --plain  # no speculative decoding, for A/B
./bin/serve-qwen38.sh --dspark # DSpark sidecar drafter, for comparison

CTX=163840 ./bin/serve-qwen38.sh              # ceiling with MTP, ~1 GB headroom
CTX=262144 ./bin/serve-qwen38.sh --plain      # full 256K, but gives up MTP
KVTYPE=q8_0 CTX=65536 ./bin/serve-qwen38.sh   # highest-fidelity KV cache
```

## The three settings that matter

Most of the win comes from three non-obvious choices. Full reasoning and the
numbers behind each are in [notes/decisions.md](notes/decisions.md).

### 1. `--parallel 1`

llama-server defaults to **4 slots**. They share one unified KV cache, so this is
not 4x the cache — but each slot still allocates compute buffers that scale with
context, which is pure waste for a single user.

This one flag took q8_0 KV at 64K from **OOM to 21880 MiB** — less than 32K used
before it. Free context, no quality cost.

### 2. q4_0 KV cache

Not lossless, but not degraded, and it is what makes 128K fit:

- **3/3 needle recall at 120K tokens** (depths 10/50/90%)
- **Full GPU speed** — 1782 tok/s prompt at 120K, same as q8_0
- **Faster decode than q8_0** at equal context (104.0 vs 97.6 tok/s, acceptance
  0.619 vs 0.582) because a smaller cache moves less memory per step
- Coherent with the cache full: at an 83K-token prompt it pulled three buried
  facts out of the middle of a haystack and reasoned over them

Max context that loads, MTP + vision, `--parallel 1`:

| KV dtype | max ctx | VRAM |
|---|---|---|
| f16 | 32K | 23862 |
| q8_0 | 64K | 21880 |
| **q4_0** | **160K** | 23544 (128K at 22648 is the safe default) |
| q5_1 | — | **do not use, see below** |

> **q5_1 is a trap.** It loads, reports the best VRAM numbers of the lot, and
> answers a short prompt fine — then collapses to **18 tok/s** prompt processing
> with the GPU at 0% utilization. Only f16, q8_0 and q4_0 have CUDA
> flash-attention KV kernels; everything else silently falls back to CPU. A
> load-only test will not catch this.

### 3. MTP over a separate drafter

The MTP head ships inside the weights file, so it costs no extra VRAM and needs
no `-md`. `--spec-type draft-mtp` sets `load_mtp` on the *main* model, and the
draft context reuses those weights.

It is worth **~2.2x** decode. It also beat the DSpark sidecar drafter on every
prompt tested, while using half the VRAM — see the decisions doc.

## Vision

Vision and speculative decoding run together; this is the default. Pass an image
as a standard OpenAI `image_url` part with a `data:image/png;base64,...` URL.

```sh
./bin/vision-check.sh shot.png "What's the error on line 87?"
```

Verified on a 900x470 terminal screenshot: it read back the failing test name,
the exact line number, and both sides of the assertion. Image requests get the
speculative speedup too — 109.5 tok/s at 0.602 draft acceptance.

`NO_VISION=1` disables it, which frees ~1.1 GB. The benchmark scripts set it so
DSpark (which cannot fit the tower) gets a fair comparison; it is not meant for
normal use.

## Sampling defaults

Set explicitly so they cannot drift with an upstream change:

| | here | llama.cpp default |
|---|---|---|
| temperature | **0.7** | 0.80 |
| top_p | 0.95 | 0.95 |
| top_k | **20** | 40 |
| min_p | **0.0** | 0.05 |
| presence_penalty | 0.0 | 0.0 |
| repeat_penalty | 1.0 | 1.0 |

Qwen3-recommended values, at 0.7 for instruction following (Qwen documents ~0.6
for thinking-mode work and 1.0 for open-ended generation).

**These are only defaults — whatever the client sends in the request wins.**
Check what the server is actually using with `curl -s localhost:8080/props`.

```sh
TEMP=1.0 ./bin/serve-qwen38.sh
```

## Thinking

Qwen3.8 is a reasoning model, and llama.cpp leaves the thinking budget
**unrestricted** by default — which is how a 512-token reply ends up entirely
inside `reasoning_content` with nothing left for an answer.

Default here is **8192 thinking tokens**, with a wrap-up message injected at the
limit so the model finishes cleanly instead of being cut off mid-thought.

```sh
REASONING_BUDGET=0     ./bin/serve-qwen38.sh   # no thinking at all
REASONING_BUDGET=-1    ./bin/serve-qwen38.sh   # unrestricted (llama.cpp default)
REASONING_BUDGET=16384 ./bin/serve-qwen38.sh
```

Thoughts arrive in `message.reasoning_content`, separate from `message.content`.
**A client that reads only `content` will see an empty string** when a reply runs
out of room mid-thought.

## Benchmarking

```sh
./bin/run-spec-sweep.sh      # plain vs MTP vs DSpark, restarts the server per config
./bin/spec-bench.py --label mtp-n5   # 5 prompts x 3 reps against a running server
./bin/kv-sweep.sh            # how much context fits, per KV dtype
./bin/needle-check.py --tokens 120000   # long-context recall
./bin/determinism-check.sh   # is the server reproducible at temperature 0
```

Logs and results go to `./logs/` (gitignored). Override with `LOGDIR=`.

Two traps these avoid, both of which produced wrong conclusions before they were
handled:

- Qwen3.8 puts most tokens in `reasoning_content`. Hashing only `content`
  compares empty strings and calls it a match.
- A reply that hits `max_tokens` ends at an arbitrary point and is not comparable
  to anything.

Note that speculative decoding is **not bit-identical** to plain decoding here.
Each mode is internally deterministic at temperature 0, but batched verification
changes the floating-point reduction order, so plain and MTP fork onto different
(equally correct) wordings. Compare digests within a mode, not across modes.

## Running as a service

`systemd/llama-qwen38.service` is a user unit. Edit `ExecStart` to match where
you cloned this, then:

```sh
mkdir -p ~/.config/systemd/user
cp systemd/llama-qwen38.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now llama-qwen38
journalctl --user -u llama-qwen38 -f
```

## Known gotchas

- llama.cpp ships **no prebuilt Linux CUDA binaries** — you have to build it.
- Distro CUDA packages are often far behind your driver. `setup/00-cuda.sh` pulls
  13.2 from NVIDIA's repo; `cuda-toolkit-13-2` is toolkit-only and does not touch
  your installed driver.
- The build is pinned to a llama.cpp tag. `qwen35` architecture support and the
  `--spec-type` names are recent and master moves fast.
- Speculative decoding with a *different-vocab* draft model silently corrupts
  structured output — impressive tok/s, broken JSON and tool calls. MTP is immune
  because it shares the target's vocab by construction.

## License

MIT — see [LICENSE](LICENSE). This covers the scripts and notes in this repo
only. llama.cpp is MIT, and the Qwen3.8 weights are Apache 2.0; both carry their
own terms.