# Cache-aware expert dropping

`--drop-cold-experts F` skips a routed expert when it is **not in the cache** *and* the router
weighted it below `F × (1 / top-k)` — that is, below `F` of the uniform share each of the `k`
selected experts would get if the router split its mass evenly. Off by default.

It is the second lossy knob in the engine, after
[turbo top-k](../README.md#trading-quality-for-speed), and it exists because the first one
spends quality in a place it does not have to.

## Why cache state belongs in the decision

`--n-expert-used k` drops the routing's tail unconditionally: slot 7 and slot 8 go, whether or not
they were already sitting in RAM. But an expert that is already resident costs **no flash read** —
and on a streamed decode, flash reads are what the token is waiting for
([decode is I/O-bound](benchmarks.md)). Dropping a resident expert pays quality for nothing.

Turn that around and the policy writes itself: **spend quality only where it buys I/O**. Keep every
resident expert however small its weight; consider dropping only the ones that would cost a read,
and only when the router says they barely matter.

## What it costs and what it buys

Replayed over the committed route traces (`docs/bench-data/2026-07-15-route-trace/`), decode phase,
threshold at the uniform share (`F = 1.0`):

| policy | flash reads avoided | router weight discarded |
|---|---|---|
| `--drop-cold-experts 1.0` | **66%** | **9.5%** |
| `--n-expert-used 5` | 23% | 10.6% |
| `--n-expert-used 3` | 59% | 36.8% |

(Qwen3-30B-A3B at k=6; Gemma-4-26B-A4B is within a point and a half on both columns: 67.4% / 8.2%. On gpt-oss-120b at k=2 the
policy matches `--n-expert-used 1`'s read saving while discarding 25% of the weight mass instead of
42%.)

At a comparable quality cost the cache-aware policy avoids roughly **three times** the reads. The
reason is visible in the third column of the trace: about 80% of decode routings are cache hits, and
the policy leaves every one of them alone.

`F` is a curve, not a switch. At `F = 0.75` the same model trades 4.4% of the weight mass for 37% of
the reads — still better than `--n-expert-used 5` on **both** axes.

These are replay numbers, and they were an **upper bound** only where dropping is light — see the
measurement below, where at full strength reality beat the prediction.

## Measured on device

Qwen3.6-35B-A3B (top-k 8 of 256), in-app, cache 3000 MiB, one variable changed:

| `F` | tok/s | flash read | routings dropped |
|---|---|---|---|
| off | 2.549 | 248 GiB | 0 |
| 0.50 | 2.564 | 231 GiB | 2.7% |
| 0.75 | **3.938** | 163 GiB | 14.2% |
| 1.00 | **4.702** | 48 GiB | 28.4% |

Per-token bootstrap intervals separate every pair **except off vs 0.50**, which overlaps: at half the
uniform share the policy finds almost nothing to drop and buys nothing. That is also the useful
negative control — the machinery costs nothing measurable when it is not firing.

Run order was 1.00, off, 0.50, 0.75, so the two fastest cells are the first and the *last*: thermal
drift would have made the last the worst. Full data, caveats and the run-order argument:
[bench-data/2026-07-22-drop-cold-experts](bench-data/2026-07-22-drop-cold-experts/findings.md).

**Quality checked, no loss detected.** 15 GSM8K questions through the same configuration: 12/15
with dropping off, **13/15 at every threshold including 1.0**, where 28% of routings are discarded.
Twelve of the fifteen questions give an identical final answer in all four cells; the variation sits
on two questions and flips in both directions rather than worsening with the threshold. Reply length
is flat, so the model neither rambles nor truncates. Details, per-question replies and the grading
rule: [bench-data/2026-07-22-drop-quality](bench-data/2026-07-22-drop-quality/findings.md).

Decoding is greedy, so there is no sampling noise: every difference between cells is *caused* by
the policy, which is what makes twelve identical answers a real statement. But 13 against 12 is
**not** an improvement — perturbing a problem the model already got wrong can land either side of
the right answer, and at 6.7 points per question this sample cannot establish the sign of the
effect. It rules out a collapse, not a subtle cost.

## Two properties worth knowing

**A routing is never emptied.** The largest weight in a routing is always at least the uniform
share, so at `F ≤ 1.0` the top expert can never fall below the threshold. `validate()` rejects
`F > 1.0` for that reason, and the implementation additionally pins the top-weighted expert, so the
guarantee does not rest on the bound alone.

**It requires the expert cache.** With `--cache-mb 0` every expert reads as a miss, so the policy
would stop being cache-aware and become an unconditional weight cut — which is what
`--n-expert-used` already does, without claiming to consult residency. `validate()` rejects the
combination, the same way it rejects `--prefetch` without a cache.

**It changes what `--prefetch` means.** Speculation is normally output-neutral by construction. Here
residency is an *input* to the policy, so a correct guess un-drops an expert that would otherwise
have been discarded: prefetch depth becomes an output-affecting setting. The decision point also
settles pending speculation a few nodes after it was issued, which shortens the overlap window the
prefetch exists for — treat the two as interacting, not composable.

**Prefill is excluded by default.** With a cold cache almost every expert is a miss, and the same
threshold discards ~42% of the weight mass instead of ~9%. Prefill is compute-bound anyway, so there
is little to win. `--drop-in-prefill` arms it for experiments.

## Narrow routings are a different regime, and the engine says so

The threshold is a fraction of the uniform share `1/top-k`, so what it removes depends on how wide
the routing is. At top-k 8 — where all the evidence here was collected — `F = 0.75` means "below
9.4% of the routing", a tail trim. At top-k 4 the same `F` means "below 18.8%", and at top-k 2 it
means "below 37.5%", which on a miss discards the entire minority expert. That is closer to halving
the routing than to trimming it, and nothing in this document measured it.

So the engine warns once at load when dropping is armed on a routing of four experts or fewer:

```
bmoe: WARNING drop-cold-experts=0.75 with top-k 2 — the threshold is 37.5% of the routing here,
against 12.5% at the top-k 8 this was measured on. Expect it to discard much more, and check
output quality on your own task.
```

It warns rather than clamping or refusing. The engine cannot know whether that trade is acceptable
for a given model and task, and silently adjusting a number the caller chose would be worse than a
loud caveat. `MoeStreamConfig::drop_low_topk_warn` is an **evidence** boundary, not a physical one:
nothing in the streaming path reads it, it only decides whether the caller is told.

The app shows the same warning inline under the setting, computed from the width the loaded model
reports at `BMOE_READY` — and only once a model is loaded, since guessing would be worse than
staying quiet. gpt-oss is the case to watch: it routes 4 of 128.

## The output is no longer reproducible

This is the real novelty, and the reason the flag is off by default and named the way it is.

`--n-expert-used` is lossy but **deterministic**: same prompt, same config, same tokens. Dropping is
lossy and **state-dependent** — what gets discarded depends on what the cache happened to hold,
which depends on everything decoded before it. The same prompt can produce different text across
runs, and a benchmark cell is noisier because the drop rate itself varies.

The greedy byte-identity gates therefore do not cover the policy's output, and cannot: there is
nothing stable to compare against. They cover the machinery instead (see below).

## How it is implemented

The decision needs the **final** router weights, and those are produced several graph nodes after
the topk node where the streamer normally loads. So with the policy armed, `load_layer()` is
postponed from the topk node to the terminal node of the layer's weight chain — the last node before
the expert matmul consumes either the ids or the weights.

Which node is terminal depends on the model's gating (`_norm`, `_softmax`, `_scaled`, or none), so
the hook **learns** it from the graph instead of carrying a per-architecture table: the first graph
of a run records the chain, and dropping starts from the second. A layer whose shape has not been
seen yet simply loads at its topk node, undropped. That costs a run its first token's dropping and
nothing else, and it keeps [hard rule 4](../AGENTS.md) — no model-specific constants in the
streaming path.

At the decision point two edits happen, both before anything reads them:

1. the dropped slot's **weight is zeroed**, and with `drop_renorm` (default on) the survivors are
   scaled so the routing keeps its original total mass;
2. the dropped slot's **id is repointed** at the routing's top-weighted expert.

The second edit is not cosmetic. An expert the engine declines to read may sit in a
reserved-but-uncommitted slot, and `mul_mat_id` would still touch it. Pointing the slot at an expert
that is certainly resident makes the kernel read valid memory and multiply it by exactly zero. It
costs a duplicate matmul — the right trade on a decode bound by flash rather than arithmetic.

Renormalisation matters more than it looks: without it the layer's expert output is systematically
scaled down by the discarded mass, a perturbation of the residual stream the model never sees in
training. `--drop-no-renorm` exists to A/B that claim.

Cost of the extra barriers: the policy asks for each layer's weight nodes, a handful more
synchronisation points per MoE layer on tensors of a few floats. The same asks a route trace makes.

## Measuring it

The engine reports what the policy actually did, which the flag alone cannot tell you — the
threshold is fixed, the drop rate is not:

```
moe-drop: <dropped>/<routed> routed experts dropped (<pct>%), threshold <F> x uniform
```

The route trace gains a `dropped` column: `weight` and `residency` stay as the **router** produced
them, `dropped` records what the policy then did, and `expert_bytes` is 0 for a dropped routing
because it costs no read. That is enough to replay a real run against the offline model and check
whether the upper bound held. See [telemetry.md](telemetry.md).

## Gates

`bmoe_moe_gates` covers the machinery, not the policy's output:

- **G8a** — with a threshold below any weight the router can produce, nothing is dropped and the
  output is **byte-identical** to the undropped stream. This proves the deferral and the learned
  terminal node are transparent, separating "the plumbing is correct" from "the policy is lossy" —
  a regression in the first would otherwise hide behind the expected difference. **G8a'** asserts
  the count separately (`experts_routed > 0`, `experts_dropped == 0`), so "a weight happened to fall
  under the threshold" fails legibly instead of as a mysterious byte mismatch.
- **G8b** — at full strength against a cache small enough to be evicting constantly, so dropped
  experts really do land on slots the cache has released. Generation still completes: the id
  repointing means no matmul ever reads reserved-but-uncommitted memory. (The gates deliberately do
  *not* run this with the cache off — there the shared-slot path has no uncommitted memory, so the
  safety property the repointing exists for would go untested.)
- **G8c** — forcing top-k to 1 makes every routed expert the top one, so dropping must be a no-op at
  any threshold and the output must match the undropped k=1 run byte for byte. This pins both the
  top-expert guarantee and the fact that the threshold is taken against the **effective** top-k
  discovered at runtime — a hardcoded width would not survive the override.

## Defaults, and where the numbers do and do not come from

The **CLI defaults it off**, and will keep doing so: the byte-identity gates need a deterministic
default, and an instrument should not quietly change the thing it measures.

The **app ships it at 75%**, under **Speed / quality → Drop cold experts**, with rungs 50 / 75 /
100 as percentages of the uniform share. It is disabled there in mmap mode and with the cache off,
the same two conditions `validate()` enforces.

75% rather than 100% is the deliberate choice: it is where the measured curve turns, taking the
larger part of the throughput win (+55%) for half the discarded routings (14% against 28%). With
quality unquantified, the conservative end of a measured range is the defensible default.

What is still owed before this is recommended beyond that:

- a published decode A/B against `--n-expert-used` at matched tok/s, with the device state recorded
  the way [benchmark-method.md](benchmark-method.md) requires;
- a quality comparison at that matched speed — the whole thesis is that this knob buys the same
  throughput for less damage, and only a side-by-side can support it;
- a re-run of the replay against a real traced run with the `dropped` column, to see how far the
  static upper bound overstated the win.

The [`layer-lfu` entry in the roadmap](roadmap.md) is the standing reminder for why the third one
matters: it simulated exactly as predicted and was ~30% slower in reality.