# Qwen3.8-Flash-Next GGUF tools

Run [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — the first
Qwen4-architecture model (125B MoE + 51B n-gram table, 6B active) — **across a DGX Spark
(or GX10) and a Windows gaming PC with an RTX 5090, connected by nothing but Ethernet (10 GbE in our case).**
Plus the day-0 tooling (converter fixes, streaming requantizer) to build the quant yourself.

Ready-made quant: [argyelan/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/argyelan/Qwen3.8-Flash-Next-GGUF) (98 GB, 3 shards)

## Why two boxes?

The 98 GB quant *loads* on a single 128 GB Spark — but with only a few GB of headroom,
sustained load (parallel requests, long contexts, eval runs) gets the process OOM-killed.
Pooling a second machine's GPU over Ethernet changes that. Measured effects, same
model, same quant:

| | Spark solo | Spark + 5090 (10 GbE) |
|---|---|---|
| Loads | ✅ (98 of ~105 GB usable) | ✅ (5090 takes 18/48 layers, ~29 GB) |
| Free RAM on the Spark | ~3 GB | **~25 GB** |
| Single stream | ~34 tok/s | ~29–33 tok/s |
| 8 parallel streams | ❌ OOM-killed | **140 tok/s aggregate** |
| Context per request | 8k was already tight | **65k stable** (131k total, 2 slots) |
| HumanEval / GSM8K runs | ❌ killed mid-run | ✅ 97.0 / 96.0 completed |

Why ordinary Ethernet is enough: with a MoE at 6B active parameters, only *activations* cross
the wire (a few KB per token) — weights never move after load. Generation speed is
nearly interconnect-independent; what you pay for the second box is a moderate prefill
cost at long context (600 → 315 tok/s from 15k to 58k) and reduced efficiency when many
deep prompts arrive at once (prefill is compute-shared).

![Generation speed vs context](docs/flashnext-depth.png)
![Concurrency scaling](docs/flashnext-scaling.png)

<details><summary>Measurement notes (click)</summary>

llama.cpp `/completion` timings, `n_predict=128`, `temperature=0`, prompt cache off,
unique prompts per stream. Depth sweep single-stream at 0/15k/31k/58k tokens; scaling
at zero depth with 1/2/4/8 parallel streams (server `-np 8`). Aggregate = sum of
per-stream generation rates. Burst caveat: 8 simultaneous 15k-token prompts drop to
~20 tok/s aggregate while prefill is shared — steady-state chat traffic does not look
like that. Hardware: DGX Spark (GB10, 121 GB) + RTX 5090 (32 GB), both on a 10 GbE switch (link speeds verified on both NICs).
</details>

## Two-box recipe: DGX Spark + RTX 5090 over LAN (RPC)

The 98 GB file runs solo, but with little headroom — under sustained load (evals,
parallel slots, long context) the box grazes OOM. Offloading part of the model to a
second machine's GPU via llama.cpp's RPC backend fixes that. We use an RTX 5090
(32 GB) in a Windows box over 10 GbE (a 1 GbE link should behave similarly for generation — untested) — no exotic interconnect needed, since only
activations cross the wire, not weights.

**On the worker (the 5090 box):** build `ggml-rpc-server` from the same llama.cpp
tree (PR #27742 — ggml changed, old worker binaries are incompatible), then:

```
ggml-rpc-server -H 0.0.0.0 -p 50052 -c
```

**On the main box (where the GGUF lives):**

```
llama-server -m Qwen3.8-Flash-Next-IQ4XS-NGQ5-00001-of-00003.gguf \
  --rpc <worker-ip>:50052 -ngl 99 -ts 18/30 -fa on \
  -c 131072 -np 2 --jinja --port 8000
```

What the numbers mean, learned the hard way:

- **`-ts 18/30`** — the RPC device is enumerated *first*, so the first value is the
  worker's share: 18 of 48 layers ≈ 28–30 GB on the 5090. The 51B n-gram table stays
  on the main box (it's lookup-only, never crosses the wire).
- **`-c 131072 -np 2`** — context is split across parallel slots (each slot gets
  c/np). 131k total = 65k per request. **Do not add `-ctk/-ctv q8_0`**: KV-cache
  quantization hits a GGML_ASSERT (`self_k_rot`) with this architecture and aborts.
  f16 KV is cheap here anyway — only 12 of 48 layers are full attention.
- With the offload, the main box keeps ~25 GB free — evals and long prompts run
  stably where solo operation got killed. Generation speed stays in the same
  ballpark as solo (~30 tok/s single stream); the wire is not the bottleneck for this MoE since
  only 6B params are active per token — though our link was 10 GbE; on 1 GbE expect prefill to suffer first.
- **Windows worker: check the firewall.** If you rebuild `ggml-rpc-server.exe` and
  never click the Windows Defender allow prompt, Windows silently creates *Block*
  rules for the new binary — the port shows LISTENING locally but is unreachable
  from the LAN, and llama.cpp **falls back to loading everything locally without
  failing loudly**. Verify offload with `nvidia-smi` on the worker (you should see
  ~28 GB used), not just a port check. Fix: `Get-NetFirewallRule -DisplayName *ggml*`,
  delete the Block rules, add a LAN-scoped Allow rule.

## Why this exists

Two things are broken on day 0 if you try this yourself:

1. **The official FP8 checkpoint breaks the llama.cpp converter** — the sharded 51B
   n-gram table ships one per-tensor `weight_scale`, which the converter cannot map
   (plus a lazy-tensor pitfall and INT64 hash constants that must survive type-exact).
   → `converter-fp8-fixes.patch` (apply on top of llama.cpp PR #27742)
2. **The 51B n-gram tensor breaks llama-quantize** — dequantizing it in one piece needs
   ~204 GB RAM (`std::bad_alloc`), and its 160-wide rows make k-quants (256-block)
   mathematically impossible.
   → `requant_ngram.py` — chunked streaming requantizer (Q8_0 → Q5_0) for exactly
   this tensor. Metadata is copied type-exact (the int64 n-gram hash constants matter).

## Full recipe (single DGX Spark / GX10, 128 GB)

```bash
# 1. llama.cpp with Qwen4 support + converter fixes
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git fetch origin pull/27742/head:qwen4 && git checkout qwen4
git apply ../converter-fp8-fixes.patch
cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server llama-quantize llama-gguf-split

# 2. Convert the official FP8 checkpoint (needs torch + ~200 GB disk)
python convert_hf_to_gguf.py /path/to/Qwen3.8-Flash-Next-FP8 \
  --outfile q8.gguf --outtype q8_0 --fp8-as-q8          # -> 176 GB Q8 master

# 3. Quantize the body, keep the n-gram table at Q8 for now
./build/bin/llama-quantize --allow-requantize \
  --tensor-type "per_layer_token_embd=q8_0" q8.gguf iq4xs.gguf iq4_xs   # -> 116 GB (too big for 128 GB!)

# 4. Shrink the n-gram table with the streaming requantizer -> fits
GGUF_PY=./gguf-py python3 requant_ngram.py iq4xs.gguf flash-next-solo.gguf   # -> 98 GB

# 5. Serve (conservative memory settings for a 128 GB box)
./build/bin/llama-server -m flash-next-solo.gguf -ngl 99 -fa on \
  -c 8192 -ub 256 -b 512 -np 1 --jinja --port 8000
```

Measured on a DGX Spark: **34 tok/s single stream, prefill ~550 tok/s**, math/JSON/needle gates clean.

## Hard-won warnings

- **Do not load files >~105 GB on a 128 GB GB10** — the box freezes hard (we power-cycled ours
  so you don't have to). 98 GB with small buffers is comfortable.
- Check `free -g` before loading — auto-started services love to eat unified memory.
- **Run with `-np 1`.** The default 4 parallel slots each add compute buffers — under
  concurrent load the box grazes OOM. Single-slot is stable; requests just queue.
- A guard loop that kills the server when MemAvailable drops below ~3 GB turns a freeze
  into a clean restart. Cheap insurance.
- Context: 262k native; start at `-c 8192` and raise stepwise while watching memory.
  KV is cheap on this arch (only 12 full-attention layers).
- **Tight on memory? See the two-box recipe below** — a second GPU on the LAN frees ~25 GB.

## Why GGUF?

The official FP8 checkpoint is 180 GB and the NVFP4 one is 135 GB — neither fits a
single 128 GB box, so a ~100 GB quant is the way in. Several groups ship Flash-Next
GGUFs now (check unsloth and friends for refined mixes); this repo is about the
*how* — the converter fixes and the requantizer for the 51B n-gram tensor, so you
can build your own. On multi-GPU rigs, vLLM/SGLang (model-support PR #36497) are
the better tools.

MIT. Measured and documented at [argyelan.ai](https://argyelan.ai).