# RTX 4090 under Windows 11 / WSL2 (Docker): reproduction notes

*The WSL2 half of the two-box campaign; the native half is
[ubuntu-3090.md](ubuntu-3090.md), with the earlier full 3090 reproduction in
[reproductions/native-3090.md](reproductions/native-3090.md).*

The docker path reproduces on a 4090 under Docker Desktop's WSL2 backend with **zero
repo changes**: `.env` plus `VLLM_WSL2_ENABLE_PIN_MEMORY=1`, exactly as
`docs/docker.md` prescribes. All numbers below: client-measured wall-clock (256-token
generations, greedy, median of 3), *not* harness rows; the existing 4090 harness row
in the README (#32) covers C1.

## CTX ladder (SPEC=dflash2, DFLASH_TOKENS=7)

| CTX | window | KV pool (tok) | prose | code | notes |
|---|---:|---:|---:|---:|---|
| fast | 65,536 | 68,605 | 134.7 | 144.5 | easy-formulaic 295.8 |
| long | 131,072 | 136,429 | 138.0 | 180.6 | int8 KV; code *faster* than fast here |
| huge | 245,760 | **268,169** | 122.9 | - | pool byte-identical to the reference 3090 |

- **huge at depth** (72,631-token prompt, prefix-cached): 35.6 tok/s decode, on the
  reference decay curve (32.0 @ 112k). First call ~71s incl. prefill (~1,080 tok/s).
- WSL2 tax at these workloads: none measurable; pools identical to native-Linux
  reference, speeds silicon-proportional.

## Concurrency ladder (`bench/conc_ladder.py --n 1,2,4,8 --out 256 --reps 2`, CTX=long, MAX_SEQS=8)

| N | per-stream | decode agg | kv% | preempt |
|---|---:|---:|---:|---:|
| 1 | 149.7 | 149 | 19.7 | 0 |
| 2 | 119.8 | 278 | 36.0 | 0 |
| 4 | 79.0 | **517** | 71.1 | 0 |
| 8 | 57.2 | 351* | 98.2 | 2 |

\* never reached steady state: the pool cannot hold 8 at ~4k ctx each.

## The KV_MEM pin: leave it alone (measured both ways)

Raising `KV_MEM` 5.58 → 6.53 GB on the 4090's spare headroom grows the pool to
159,643 tokens, but costs **8% C1 decode** (126.6 vs 138.0), and under 8-way load
the server **OOM-crashes on a 58 MiB activation allocation**: the headroom the pin
preserves is what concurrency spikes spend. The shipped constant transfers to the
4090/WSL2 as-is. Windows are design caps (131,072 for dflash2+long; 245,760 huge),
not VRAM accidents: `MAX_LEN` above them is clamped.

## Retest at c954724 (upstream pickup, 2026-08-27)

Rebuilt at upstream head (four new patches incl. n-gram chains #38, int4-KV #42).
Same standing config, warmed second-pass medians, CTX=long:

| build | LOOKUP | prose | code |
|---|---|---:|---:|
| 60daef8 | 1 (default) | 138.0 | 180.6 |
| c954724 | 1 (default) | 130.6 | 170.0 |
| c954724 | **0** | **140.6** | **194.6** |

**RETRACTED (same day): the "#38 costs ~5.5%" row above was a COLD-BOOT
artifact, not a regression.** The c954724 `LOOKUP=1` numbers came from the first
boot after the image rebuild (empty torch.compile/JIT caches), and the README's
"first run reads low" warning operates at *boot* granularity: a warmed **third**
boot of the identical config measured **140.4 / 184.8** (vs 130.6/170.0 on boot
one: **+8.7% boot-to-boot on the same build and flags**). Warm-boot old-vs-new is
138.0/180.6 → 140.4/184.8: **no regression**; the pickup is clean.

Consequences, which outlive the mistake:
- **Boot-to-boot variance on this stack is ~8-9%**, larger than most effects
  worth arguing about. Any A/B whose arms require a server restart (every env-var
  toggle) is a *cross-boot* comparison and needs boot-median protocol (≥2 boots
  per arm) or effects >2× the boot floor to be claimed at all.
- The `LOOKUP=0` vs `LOOKUP=1` deltas on this card are therefore **unresolved**
  (prose 140.6 vs 140.4, identical; code 194.6 vs 184.8, inside two boot
  floors). Upstream's own README numbers (+7% copy, flat prose) were taken within
  one server session and remain the best-sourced claim.
- The instrument list gains an item: **the warm-state of the serving process**
  (which boot, how many requests old, cache volume state).

## Chain mode + #42 int4-KV, tested (2026-08-27, boot-floor-aware)

**Boot floor, corrected by mechanism:** rebuild-reset caches produced the ~8.7%
"floor" retracted above; plain restarts inherit the cache volume and measure a floor
of **~0.1%** here (copy cell: 221.6 / 221.6 across boots). The floor is a property of
what state the boot inherited, not of the stack.

**`VLLM_DFLASH2_CHAIN=1`** (2 boots/arm, ~0.1% floors): copy **221.6 → 268.0
(+21%)**, triple the README's +7% on this card; prose 140.4 → 136.75 (**−2.5%**,
small but real). A per-workload flag: on for reproduction jobs, off for novel
generation.

**#42 int4-KV profile (`alternative.sh`, GPU_UTIL=0.93 per its own WSL2 note):**
window/pool **256,000 tokens on one card**, 22.0GB, serves clean. Shallow prose
118.0 (≈12% under bf16, near the claimed cost). ~~**But at 72.6k depth, cached-prefix
decode collapses to 3.6 tok/s**, ~10× under KVarN's 35.6 at the same depth, with
healthy acceptance (~30%, ~3.1 tok/step): the cliff is **step time (~860ms)**, i.e.
the verify attention over a long int4 cache. The repo's int8 long mode needed a
dedicated split-KV verify kernel for exactly this (its notes price unsplit Triton
attention at 7.4ms/layer at 128k); int4 has no analogous kernel yet.~~

~~**Mechanism CONFIRMED by the discriminating arm** (proposed by the 3090 seat: same
profile, same depth, speculation off; single-token decode is single-query
attention, so if the multi-query verify path is the cause, the cliff disappears):~~

| int4 @ 72.6k depth | decode tok/s |
|---|---:|
| SPEC=dflash2 (n7) | ~~3.6~~ |
| **speculation off** | ~~**29.0**~~ |

~~The cliff is the verify path, resolved at n=1 against a ~0.1% restart floor.~~

**STRUCK 2026-08-27 (same night, twice).** First the attribution died: we built
the missing split-KV verify kernel (6× in isolation, 43.5→7.2 ms/step at 72.6k)
and end-to-end deep decode did not move: verify was ~5% of the step, and the
spec-off arm removes the drafter as well as the verify, so it could never
attribute the difference (the 3090 seat retracted the test's design themselves).
Then the premise died: **the "3.6 tok/s deep decode" was never a decode rate.**
Under this profile the engine gets **zero prefix-cache reuse** (identical 72k
prompt, `enable_prefix_caching=True`, three requests = three full ~62s
prefills, request-level TTFT), so every "cached" deep request re-ran the full
72k prefill (~1,170 tok/s through the eager int4 chunked-prefill path) inside a
total-window measurement. Streamed with TTFT split out, the true numbers at
72.6k depth on this card:

| int4 @ 72.6k, decode-only | tok/s | accept | emitted/step | step ms |
|---|---:|---:|---:|---:|
| speculation off | **35.55** | - | 1 | 28 |
| SPEC=dflash2 n7, stock dispatch | 8.14 | .248 | 2.74 | 336 |
| SPEC=dflash2 n7 + MQ split-KV kernel | 25.23 | .302 | 3.11 | 123 |

**Corrected verdict:** #42's real deep-use blockers are (1) **zero prefix-cache
reuse under int4 × DFlash2**, isolated by ablation: int8+DFlash2 reuses fine
(3090), int4 *without* DFlash2 reuses fine (TTFT 62.1s → 3.9s on repeat); only
the combination never hits; and (2) the slow int4 chunked prefill itself
(~1,170 tok/s at depth).

**Blocker (1) is WORKED AROUND with one flag: `--prefix-match-unit 848`**
(pass via `EXTRA_ARGS` to `alternative.sh`). Root cause: the patch stack's
sliding-window prefix-cache guard (`port(kvarn-v2)`, issue #18) returns a
permanent clean miss when the drafter's SW block size (848 under int4's
halved-page geometry) is not a multiple of the hash unit (1696); the
hybrid coordinator's min turns that one group's miss into zero reuse
model-wide. The stack's own GCD hash-unit logic excludes SW groups, so the
default never resolves it; the explicit flag (=gcd of the block sizes) makes
every divisibility clause pass with no geometry change and zero pool cost.
Verified on this card: per-group hits converge at 71,232/72,591 tokens
(98.1%), **TTFT 62.0s → 4.1s on the warm repeat**, warm generation
byte-identical to cold, decode rate and acceptance unchanged. Two caveats
were raised and both resolved same-day: (1) ~~pool unchanged (253,965)~~: an
apparent +19% pool difference under the flag (301,974 vs 253,965) was traced
to boot-time free-VRAM drift, not the flag: a same-hour no-flag control boot
also reports 301,974. The flag does not touch capacity; quote any pool number
together with its boot's GPU state. (2) The byte-identical warm-vs-cold result
was confounded by the drafter's suffix-lookup history (request order alone can
change greedy output on this stack); re-verified with the lookup lane disabled
(`LOOKUP=0`): warm TTFT 4.12s, per-group hits converge at 71,232, and the warm
generation remains byte-identical to cold; reuse reproduces the cold
computation exactly. The value is per-card AND
per-profile: the unit must EQUAL the sliding-window group's block size
exactly (848 here; finer divisors get zero reuse from the guard's alignment
clause), and that block size moves with model, KV dtype, and DFLASH_TOKENS
(n=3 chooses 1616/808). The int8 profile never needs the flag (its geometry
lands 864/864, so the guard cannot fire). Deep *decode* was never 10× under KVarN. Speculation
at this depth is a net slowdown even with the experimental MQ split-KV kernel
(acceptance ~.30 × n7 loses to single-query decode): **run deep int4 work
spec-off** (35.5, matching KVarN-with-spec's 35.6). The MQ kernel (2.73× step ×
1.14 emitted = 3.10× over stock spec-on) is held unshipped pending operator
oracles at depth, a full-length exactness pass, and a shallow-context crossover
(shallow shows −16%; the dispatch needs a seq_len floor and a capture-aware
variant switch). It still does not obsolete `CTX=huge`/KVarN today.
~~Quality-at-depth remains unmeasured here on purpose: Austen needles are
memorization-contaminated (PPL 1.06), and the proper needle-recall harness run
is queued on the 3090.~~ **Measured (3090 seat, 2026-08-27): 4/4 exact needle
recall under int4 KV at 29,653 / 96,368 / 168,542 / 218,085 prompt tokens
(spec-off, MAX_LEN=256000, pool 266,520, 22,612 MiB), the same 218k depth
where KVarN 4/2-bit recalls exactly on that box. Scope stated precisely: a
needle is a RECALL test, not a quality test; a distinctive literal string is
near the easiest thing a degraded cache can still retrieve. This licenses
"int4 does not lose the plot at 218k," not "int4 is lossless at 218k"; the
latter needs a perplexity or teacher-forced comparison at depth against bf16,
which has not been run on either box. Four depths, n=1 each, one needle, one
card.** With recall clean, the MQ split-KV kernel (3.1× deep decode) and
`--prefix-match-unit` in hand, the remaining objections to #42 at depth are
performance items with known fixes, not correctness ones.

## ⚠ INSTRUMENT INVERSION (2026-08-28): read before any deep spec-on number below

Every deep **spec-on** "decode tok/s" below measured by the streaming probe
before 2026-08-28 late is **VOID: it counted SSE chunks, and under
speculative decoding on this stack one SSE chunk carries one STEP's tokens**
(~3 at depth). Spec-off numbers (1 token/step) and everything counter-based
(ladder aggregates, acceptance, TTFT, exactness) were always true. Proof:
a saved deep generation tokenizes to 455 tokens against 149 counted chunks
(3.05× = that run's emitted/step 3.06). Token-true re-measurement
(usage.completion_tokens, identity-checked steps×emitted/step≈tokens):

| deep 72.6k decode-only, TRUE tok/s | | |
|---|---:|---|
| **int4 + MQ split-KV kernel** | **70.5-75.2** | the fastest deep config on the box |
| long (int8) | 64.0-67.7 | |
| huge (KVarN) | 59.9-62.7 | old "35.6" void (unknown instrument) |
| any profile, spec-off | ~35.5 | |
| int4 stock dispatch | 22.3 | |

**Inverted verdicts:** speculation at depth WINS ~2× (never was break-even:
the q=8 step costs 43ms vs q=1's 28ms, +54% for ~3× tokens); ~~run deep int4
work spec-off~~ → **spec-on is the deep mode**; the MQ kernel's true e2e is
~3.3×; the deep crown moves to int4+kernel (held behind its promotion gates;
this measurement is not the default-flip gate). The shallow spec-on figures
(102-138 class) from the same probe are also step-contaminated and the
shallow crossover awaits token-true re-measurement. The mixed-load and
politeness sections' *decode* columns are step rates (~×3 for tokens);
their TTFT/starvation findings are timing-based and stand. Lesson kept:
**know what one unit of the thing you are counting actually is**. Chunks,
steps, tokens, requests, and windows were five different quantities in this
document's history.

## The whole ladder, one table (2026-08-27, single 4090 unless noted)

| profile | KV format | window | pool (tok) | shallow prose | deep 72.6k decode | windows @72.6k | notes |
|---|---|---:|---:|---:|---:|---:|---|
| fast | bf16 | 65,536 | 68,605 | 134.7 | - (window < 72.6k) | - | code 144.5, easy 295.8 |
| long | int8 | 131,072 | 136,429 | **138.0** | 25.3 (spec-on) | 1.9 | code 180.6; the daily driver |
| huge | KVarN 4/2-bit | 245,760 | 268,169 | 122.9 | **35.6** | 3.7 | on the reference decay curve (32.0 @ 112k) |
| int4 (#42) | int4 per-token-head | **256,000** | 266k-302k† | 118.0 | **35.55** spec-off · 25.23 spec-on+MQ-kernel · 8.14 spec-on stock | 3.7-4.2 | recall 4/4 to 218k (3090); needs `--prefix-match-unit 848`; only boots at n=7 |

† Pool numbers are **boot-state-scoped** (this card measured 253,965 and
301,974 under identical config, hours apart: free-VRAM drift); the 3090
reports 266,520 at MAX_LEN=256000, the patch header 314,915. Quote a pool with
its boot. Full-window concurrency is ~1.0-1.2× everywhere: these are
single-user profiles; "windows @72.6k" is the honest concurrency-depth column.

Deep prefill is the shared pain, not a differentiator: huge ~1,080 tok/s,
int4 ~1,170 tok/s at 72.6k (~60-70s first call). Deep-cached calls need a
verified cache: huge's 35.6 sits on the reference decay curve (independent
corroboration); int4's requires the prefix-match-unit flag; long's is
threadchip-verified (0→60.4% hit rate).

**Long at depth, measured 2026-08-27 (the last empty cell):** 25.31 / 25.33
tok/s across two runs (spec-on n7, acceptance .22-.24, cache verified: warm
TTFT 1.31s, engine hit rate 49.4%). So at 72.6k the daily driver is the
SLOWEST of the three: statistically identical to int4 spec-on+kernel (25.2)
and 29% behind huge (35.6). The consistent physics: deep verify steps are
KV-bandwidth-bound, and huge's 4/2-bit cache reads roughly half the bytes of
int8 per step. Long spec-off at depth remains unmeasured (same physics
predicts ~35). One measurement scar worth its line: **the chat template is
part of the prefix**. A request with `enable_thinking` toggled renders a
different prompt prefix and misses the cache entirely (61s re-prefill,
observed); pin template kwargs when benchmarking cached depth.

**Reading of the tradeoffs:** with kvarn installed (both our boxes), **huge
is the depth default, now confirmed against the daily driver**: fastest deep
decode (35.6), slightly better shallow than int4, no flag, no n-brittleness. int4's case is (a) **stock-vLLM
machinery**: it needs no kvarn install, only 3 small patches, which makes it
the *portable* 256k profile and the upstream-relevant one; (b) +4% window /
up to +13% pool; (c) it's where the MQ split-KV kernel and any future
int8-IMMA prefill work land. Run int4 spec-off at depth (25.2 spec-on <
35.5 spec-off even with the kernel: acceptance ~.30 × n7 loses to q=1).

## Three-way concurrency ladder (2026-08-27, all MAX_SEQS=8, ~4k prompts, out=256, reps=2)

Same hour, same harness (`bench/conc_ladder.py`), sequential boots, each phase
branded by its boot's pool line (long 136,429 / huge 268,169 / int4 250,486).

| N | long per-stream/agg | huge per-stream/agg | int4 per-stream/agg |
|---|---|---|---|
| 1 | 107.0 / 106 | 134.9 / 133 | 129.0 / 127 |
| 2 | 69.0 / 151 | 116.1 / 249 | 120.1 / **259** |
| 4 | 41.8 / 208 | 62.5 / 330 | 81.5 / **452** |
| 8 | 35.1 / 190* | 46.2 / **303*** | 54.4 / 242* |

C1 TTFT: long 1.8s · int4 2.7s · huge 5.5s (17.9s at C8: the KVarN prefill
tax, which drops huge's end-to-end aggregate to 34-66, worst of the three;
int4's e2e is best from C2 up at 99-121). \*C8 rows are tail measurements:
no instant had all 8 decoding; long preempted twice at 98% kv, int4 hit 99%.
ms/pass: int4 flattest (25.9→31.6, healthy batching), long steepest
(34.5→70.9). **Caveats:** this long boot ran 29% under its own earlier
documented C1 (107 vs 149.7, identical config): the boot floor applies to
ladders too, so treat C1 ordering as unconfirmed; the C2-C4 gaps (1.5-2×)
exceed any observed boot wobble. kv% is harness-internal (block-granularity),
comparable within a run only.

~~**Practical read: long for single-user shallow, huge for single-user depth,
int4 for concurrency (C2-C4).**~~ **CONTAMINATED AND CORRECTED same evening
(caught by Michael: "I thought we were pulling ~500 at C4?").** The long/huge
rows above were measured with a forced `MAX_SEQS=8` that overflows their
CUDA-graph reservations: start_qwen.sh sizes capture for its own defaults,
long=4 and huge=2, and documents the overflow mode ("bigger batches run
piecewise instead of captured: slower, alive"). Rerun at designed configs:

Cells: decode-aggregate (per-stream) / TTFT.

| N | long (MAX_SEQS=4) | huge (MAX_SEQS=2) | int4 (MAX_SEQS=8) |
|---|---|---|---|
| 1 | 143 (145.6) / 1.9s | 134 (135.3) / 3.3s | 127 (129.0) / 2.7s |
| 2 | 281 (121.8) / 2.6s | 227 (105.2) / 6.3s | 259 (120.1) / 2.9s |
| 4 | **517** (82.5) / 4.2s | 228* (101.4) / 10.9s | 452 (81.5) / 4.7s |
| 8 | **470*** (75.5) / 8.2s | 191* (80.8) / 19.7s | 242* (54.4) / 8.5s |

\* not steady state: no instant had all N decoding. The CAUSE differs per
profile and the `run`/kv% columns in the JSONs separate them: huge C4/C8 =
**admission cap** (MAX_SEQS=2; run=2, kv% 44.7, pool half empty; extra
streams queue, TTFT 10.9s→19.7s, throughput pinned at ≈C2 by construction);
long C8 = the same cap at 4; int4 C8 = genuine **pool pressure** (kv% 99.4).

Reading the two throughput columns: they measure different windows and
legitimately disagree in both directions. Per-stream is a stream's own
first-to-last-token average, dragged down by the phase where its decode
interleaves with other streams' chunked prefills; decode-aggregate opens only
after EVERY stream has its first token (all prefills done) and reads the
server counter during pure joint decode. Long C4: joint decode is 517
(~129/stream) while lifetime per-stream averages 82.5; aggregate is server
capacity, per-stream is user experience. long's C4 now three-times confirmed
(517 here, 517 documented, 533 on the standing stack). ms/pass at designed
configs: long flat 23.8→27.8 (captured).

**Corrected read: long is the concurrency king outright**: best aggregate at
every N, zero preemptions, even its over-cap C8 (470) beats the field. int4
is second and keeps two distinctions: only profile admitting >4 resident, and
the 256k window (long caps at 131k). huge is a 1-2 user depth instrument.
Standing lesson: **MAX_SEQS is a graph-budget choice, not a free knob: a
concurrency number is only comparable at the capture config it was captured
under.** Ladder JSONs (both passes) kept with the bench record.

## big.LITTLE mixed load: one whale + minnows (2026-08-28, CTX=long, MAX_SEQS=4)

The uniform ladder above can't see the real serving hazard. Mix one 72.6k
"whale" (salted → real prefill) with salted 4k "minnows", launched together:

| scenario | whale TTFT | minnow TTFT (min/med/max) | minnow decode | wall |
|---|---:|---|---|---:|
| C2: whale+1, cold | 65.6s | 2.8s | **1.2 tok/s** (starved) | 69s |
| C4: whale+3, cold | 62.6s | 63.5/68.2/70.2s | 22-40 (after waiting) | 72s |
| C8: whale+7, cold | 68.6s | 2.2/**75.9**/81.6s | 0.9/14.5/36.3 | 83s |
| **C4: whale+3, whale CACHED** | **4.9s** | **5.5/6.9/11.2s** | 15-35 (immediate) | **13.5s** |

**A deep prefill monopolizes the engine for its full ~65s**: chunked prefill
either rations an admitted minnow to ~1 tok/s (each pass = a 2,020-token whale
chunk + one minnow token) or queues minnows behind the entire prefill. Decode
coexistence afterward is fine. So the serving constraint is **prefill
head-of-line blocking, not decode concurrency**; the fix is the prefix
cache: with the whale's prefix warm, the same scenario collapses 72s → 13.5s
and every stream's first token lands inside ~11s. Pin your whales' prefixes.

## The politeness knob: `--max-num-batched-tokens` under mixed load (2026-08-28)

Same cold-whale C4 mix as above, CTX=long, sweeping the per-pass token budget:

| budget | whale TTFT | minnow TTFT | minnow decode during whale prefill | wall |
|---|---:|---|---|---:|
| 2048 (default) | 62.6s | 63-70s (queued) | 22-40 only after | 72.2s |
| 1024 | 69.6s | **4.2-7.2s** | 6.8-14.3 | 73.0s |
| 512 | 72.2s | **4.5-7.6s** | **14.2-21.8** | 75.4s |

At 2048 a whale chunk fills the whole budget, so minnow prefills queue behind
the entire whale; at ≤1024 the scheduler interleaves, so minnows are live in
~5s. Whale decode is unaffected (25.4 everywhere). **Mixed serving: budget
512 + pinned whale prefixes (with the cache, the whale itself starts in ~5s;
see the cached-whale row above); dedicated single-user depth: keep 2048.**
Caveat for the boot ritual below: smaller budgets slow ALL prefill, so the
warm-pair ratio compresses (7.7× at 2048 → ~2.8× at 512); the ½ threshold
still held.

## Cache-dead boots (observed 2026-08-27/28): add a warm-pair check to the boot ritual

One CTX=long boot served correctly but **never produced a prefix-cache hit**
(0.0% across 32 windows; three identical-prompt pairs all re-prefilled ~62s),
with verifiably healthy 864/864 block geometry. A config-identical restart hit
normally (warm TTFT 3.5s; per-group trace converging at 71,712). Cause
unknown: the dead boot was recycled before deep inspection; treat as a real,
rare, nondeterministic boot state. ~~Detection is cheap: send a ~2k-token
prompt twice and require the second TTFT < half the first.~~ **CORRECTED
(3090 seat caught it failing on its first healthy boot): a fixed-size probe
is structurally blind when it spans ≤1 block: reuse is (blocks−1)×B, so 2k
tokens at B=1696 (int4) reuses NOTHING and a healthy boot reads dead; at
B=864 it's one reusable block, a coin flip. Size the probe from the block
size the engine chose, out of its own boot log:**

```bash
BLK=$(grep -ohE 'Setting attention block size to [0-9]+ tokens' server.log | grep -oE '[0-9]+' | tail -1)
NTOK=$((BLK * 10))   # 9 reusable blocks ≈ 90% — unmistakable either way
```

Verified both boxes: 3090 int8 cold 7.58s → warm 0.99s (7.7×); 4090 int8
cold 4.94s → warm 0.64s (7.7×). Restart on failure. Until root-caused, no
"deep-cached" number should be trusted from a boot that hasn't passed the
sized pair check. (Third instance in one day of a test tuned to a constant
it didn't read: the constant is never portable and always in the boot log.)

## Batch mode on the 4090 (2026-08-28): the other wing

First boot of the repo's `batch` profile here (fp8 KV, no speculation, engine
block 800, pool 175,515 @150k, `GPU_UTIL=0.93`; **0.972 never fits under
WSL2's reserve on a desktop-attached card**; the MAX_LEN defaults are
fp8-geometry-specific, don't carry int4 numbers over). Token-true ladder,
distinct 4k prompts:

| N | 1 | 2 | 4 | 8 | 16 |
|---|---:|---:|---:|---:|---:|
| decode agg | 55 | 101 | 195 | 384 | **676** |
| per-stream | 54.5 | 49.7 | 42.4 | 33.8 | 23.3 |
| ms/pass | 18.3 | 19.8 | 20.5 | 20.9 | 23.7 |

Textbook batch scaling: tokens/pass ≡ N, ms/pass essentially flat C1→C16,
zero preemptions (the profile's 64-way graph budget is pre-sized: no
piecewise trap). **Doctrine: the spec wing (single+DFlash) wins up to
~C4-C8 (long C4 = 517 agg); batch takes over around C10 and scales to 64**
(README: ~1,094 at C64 on a 3090). At C1 batch costs 2.7× vs spec: spec IS
the single-user profile.

**Retention under batch (prediction falsified, informatively):** with NO
drafter groups, 3 contexts at 85% naive utilization still fully evicted. The
retention tax's center of mass is the **mamba state** (equal-sized mamba
pages per attention block, by construction ≈ 2× per cached context), with
the drafter adding the rest of spec-mode's ~2.7×. Universal for this hybrid
model: **cached seats ≈ pool ÷ (2-2.7 × context) on every profile** (one
warm deep whale per box); a CPU offload tier (the in-tree #33 patch /
LMCache, noted for investigation) is the only door to more. *(That door is
now open; see the offload section below.)*

## TP=2 (2×4090) under WSL2 (2026-08-27)

Boots clean (PYNCCL; no NCCL env tweaks needed). **Capacity play, not a speed play
on this platform**: KV pool **593,574 tokens** at CTX=long (4.53× concurrency at
131k: four full windows resident at once), but C1 decode drops 22-33% (prose
107.3, code 121.6 vs 138.0/180.6 single-card) and N=4 aggregate falls to 358 vs the
single card's 517. The PCIe all-reduce under WSL2 inverts #40's native-3090 result
(+16-35%). Recorded as a **platform-dependence datapoint**, not an explanation.
The #40-era `start_qwen.sh` TP calibration (auto-skip of the single-card KV_MEM
pin) worked exactly as documented. One caveat: the desktop compositor's ~1.7GB on
the second card means `GPU_UTIL=0.90` there, not 0.93.

## CPU offload tier under WSL2: the OffloadingConnector fix (2026-08-28)

The in-tree #33 OffloadingConnector (CPU tier over a pinned `/dev/shm` mmap)
crashed WSL2 with an illegal memory access on the first real eviction, after a
misdirection cascade that blamed two innocent async bystanders (the mamba-state
copy, then the block zeroer: `CUDA_LAUNCH_BLOCKING` serializes **kernels**, not
async copies, so the fault surfaced on whatever synchronized next).

**Root cause:** the Triton `swap_blocks` kernel dereferences raw **host** virtual
addresses of the `cudaHostRegistered` region. Under native-Linux UVA the host VA
happens to equal the device VA, so it works (by coincidence, not contract). WSL2
registers successfully but maps the region at a **different device address**, so
the kernel faults. Registration succeeding was mistaken for device-address
equivalence; `cudaHostGetDevicePointer` (the API's own answer for exactly this
case) was never called.

**The fix** (`patches/offload-wsl2-devptr.patch`, applied in the stack): after
registration, ask for the device pointer, carry the delta on each CPU view, and
add it where `compute_sub_block_ptrs` builds addresses. Delta is 0 on native
Linux: no behavior change. Every failure branch **raises** rather than warning:
a platform where the device pointer is unavailable refuses to start instead of
proceeding into silent-unsafe zero-copy. That makes the fix more falsifiable
than a warn-and-continue shape: a live engine with the patch applied is itself
evidence the query ran and succeeded.

**Loader gotcha** (cost one review round): standard pip installs ship only
*versioned* cudart sonames at absolute wheel paths, so `CDLL("libcudart.so")`
throws on most native boxes. The patch resolves from the process image first
(`CDLL(None)`: torch has already dlopen'd cudart), then system sonames, then a
glob over the torch wheel's `nvidia/*/lib/`.

**Falsification bracket, both platforms:**

| Arm | WSL2 / 4090 | Native / 3090 |
|---|---|---|
| With fix | 3×72.6k whales primed, all recheck **warm 4.2-4.3s**, 0 IMAs | 52.4 tok/s, token sha byte-identical to baseline, **805 MB** moved through the tier live |
| Fix removed | IMA returns (count 3), first prime dies | returns to baseline exactly |
| Loader | resolves tier 1 (also carries unversioned soname) | resolves tier 0 (`CDLL(None)`), rc=0, **delta=0** |

On WSL2 the delta line prints (`delta=-133564122513408 … translating kernel
addresses`); on native it must be **absent** (both observed). First time the CPU
tier carried state on either box: ~590k page-tokens held beyond the 252k GPU
pool; a 72.6k whale primes in ~62s and rechecks warm in 4.3s.

**Operator notes:** the 8 GB `/dev/shm` backing file **outlives the process**:
confirmed empirically on the 3090 (still resident after a teardown kill), and
because shm is RAM-backed the stranded file **holds those gigabytes for as long
as the box sits idle**. Cleaning at startup (`start_qwen.sh` does) recovers it
for the next run, but on a small-RAM box the held memory is the next boot's
OOM, so clean at shutdown too. Docker needs `--ipc host`: the 64 MB default shm
makes the region's `madvise(MADV_POPULATE_WRITE)` fail with EFAULT at boot.

Status: validated on both boxes for correctness and round-trip. The
production-form hardening (typed two-pointer separation instead of a delta
attribute, transfer records, restored-state checksums) is tracked as follow-on
work.

**Three agents, measured (2026-08-28):** with the tier armed, three
simultaneous agents each on its own 72.6k context fire repeatedly with
**zero recomputes**: unqueued tier restores in 5-7s, the LRU-rotated seat
~16s restoring under load, against 62s cold. Those 4-7s rechecks are tier
restores, *not* GPU-warm hits: the tier-off control (K=2 round-robin at
72.6k = **0/2**, K=3 = 0/3 on a 300,583-token pool) shows the GPU holds
about one such context warm; real per-context cost exceeds 150k tokens at
72.6k depth, a >2.07× multiplier. A corrective banner line computed from
the engine's own spec bytes (constant mamba + linear attention) predicted
2/2 and was falsified by that same control; it is parked on the
`banner-capacity-experiment` branch, unmerged, while the gap mechanism
(drafter group pages · per-group block rounding · prefix-retention
granularity at the 848 hash unit) stays open.