# EXPERIMENT: LLMs on the Xiaomi TV Box

> Paths below are relative to `llm-experiments/` (this folder). Device-wide logs live one level up: `../CHANGELOG.md`, `../rollback.sh`, `../REPORT.md`.

**Date:** 2026-11-03. **Device:** Xiaomi TV Box S 3rd Gen (`MiTV-AFMU0`), Amlogic S7D, 4× Cortex-A55 @ 2.5 GHz, 2 GB RAM, Android 14.
**Goal:** get a real, pretrained language model to produce a token on the box. Use the fewest parameters possible; speed doesn't matter.
**Result: ✅ it works.** A **260,032-parameter** Llama-architecture model writes complete short stories on the TV box at **~640 tokens/s**, using about 1 MB of weights.

## 1. The model: `stories260K`

The smallest published pretrained transformer LM I know of is Karpathy's **stories260K** ([karpathy/tinyllamas](https://huggingface.co/karpathy/tinyllamas)). It's a real Llama-2-style model trained on the TinyStories dataset, the smallest of the `llama2.c` reference models (260K / 15M / 42M / 110M).

| Hyperparameter | Value |
|---|---|
| `dim` (embedding width) | 64 |
| `hidden_dim` (FFN) | 172 |
| layers | 5 |
| attention heads / KV heads | 8 / 4 (grouped-query attention) |
| vocab | 512 (custom SentencePiece BPE, `tok512.bin`) |
| context length | 512 tokens |
| output head | tied to the embedding table |
| **total parameters** | **260,032** |
| weights file | 1,056,540 bytes, fp32 |

Where the parameters go:

| Part | Params | Share |
|---|---|---|
| Token embedding (512 × 64, also the output head) | 32,768 | 12.6 % |
| Attention (Wq, Wk, Wv, Wo) × 5 layers | 61,440 | 23.6 % |
| SwiGLU FFN (W1, W2, W3) × 5 layers | 165,120 | 63.5 % |
| RMSNorm weights | 704 | 0.3 % |

For scale: GPT-2 small (124M) is ~480× bigger and Llama-2-7B is ~27,000× bigger.

## 2. Runtime: `llama2.c`

[`karpathy/llama2.c`](https://github.com/karpathy/llama2.c) `run.c` (commit `350e04f`) is the whole Llama-2 forward pass, the BPE tokenizer and the sampler in one 973-line C file with no dependencies. **I reviewed the source before running it:** it only `open`/`mmap`s the model and tokenizer files and prints to stdout. No network, no exec, no file writes.

## 3. Discovery: a 64-bit kernel under a 32-bit Android

Android reports only `armeabi-v7a` (32-bit userland, as in `../REPORT.md`), and `uname -m` says `armv8l`. But the kernel is `5.15.153-android14` **64-bit**: a statically linked **aarch64** binary runs fine from `adb shell`. So I built both:

```sh
brew install zig        # zig 0.17.0, used as a cross-compiler
# musl doesn't pull in stdint.h implicitly like glibc does → -include stdint.h
zig cc -target arm-linux-musleabihf -mcpu=cortex_a55 -O3 -static -include stdint.h -o bin/llama2c/run-armv7 third_party/llama2.c/run.c -lm
zig cc -target aarch64-linux-musl                     -O3 -static -include stdint.h -o bin/llama2c/run-arm64 third_party/llama2.c/run.c -lm
```

Both are ~1.7 MB static ELF files and need nothing from Android's libc.

## 4. Deploy and run

```sh
D=/data/local/tmp/tinyllm
adb -s <tv-ip>:5555 shell mkdir -p $D
adb -s <tv-ip>:5555 push bin/llama2c/run-armv7 bin/llama2c/run-arm64 models/llama2c/stories260K.bin models/llama2c/tok512.bin $D/
adb -s <tv-ip>:5555 shell "cd $D && chmod 755 run-* && ./run-arm64 stories260K.bin -z tok512.bin -t 0 -n 0 -i 'Once upon a time'"
```

(`-t 0` = greedy/deterministic, `-n 0` = run to the full 512-token context or until the model emits BOS, `-z` = tokenizer.)
Local files: source in `third_party/llama2.c/`, binaries in `bin/llama2c/`, model + tokenizer in `models/llama2c/`.
SHA-256: `stories260K.bin` `b0a507e7…2696`, `tok512.bin` `037cb335…c312`, checked identical on the Mac and on the TV.

## 5. Output (verbatim, from the TV)

**Greedy, prompt "Once upon a time".** armv7 and arm64 give byte-identical text:

> Once upon a time, there was a little girl named Lily. She loved to play outside in the park. One day, she saw a big, red ball. She wanted to play with it, but it was too high.
> Lily's mom said, "Lily, let's go to the park." Lily was sad and didn't know what to do. She said, "I want to play with your ball, but I can't find it."
> Lily was sad and didn't know what to do. She said, "I'm sorry, Lily. I didn't know what to do."
> Lily didn't want to help her mom, so she said, "I'm sorry, mom. I didn't know what to do." Her mom said, "Don't worry, Lily. We can help you."
> Lily and her mom went to the park to play. They played together and had fun. After a while, Lily's mom came into the room and saw the park. She was so happy and said, "Thank you, Lily! You are a good friend."

**Sampled, temperature 0.8, seed 42, no prompt:**

> Once upon a time, there was a little girl named Lily. She loved to play outside in the sun. One day, Lily's mom told her to teach him to cut one toys and bring it to the sunflower. Lily was scared and scared.
> Lily wanted to make a popcorn spoiled a big, red treasure. She was sad because she had started to draw and it was dead. […] The cake was happy to have a friend.

**Prompt "The cat", greedy, 60 tokens:**

> The cat and a boy were playing in the park. They liked to play with their toys and run around the park. They liked to play with their toys and seek.

**The bare minimum, "one token out"** (empty prompt, `-n 2`): `Once upon`

What 260K parameters get you: correct grammar, punctuation, quotes, and a named character kept across sentences. You also get loops ("didn't know what to do"), logic slips ("Lily's mom came into the room and saw the park") and, with sampling, nonsense phrases ("a popcorn spoiled a big, red treasure").

## 6. Numbers

| Metric | armv7 (32-bit) | arm64 (64-bit) |
|---|---|---|
| Speed, 3 runs, greedy full story | 615 / 616 / 655 tok/s | 673 / 644 / 678 tok/s |
| Best observed (short prompt) | — | ~900–1000 tok/s |
| Binary size | 1.67 MB | 1.81 MB |

The arm64 build is ~5–10 % faster (more NEON registers). Both use one core: no OpenMP, and the model is too small for threads to help.

**Memory.** Weights 1.06 MB (mmapped) + KV cache 5 × 512 × 32 × 2 × 4 B = 0.66 MB + a few KB of activations. The kernel reported `VmHWM` 868 kB mid-run (RssAnon 768 kB, RssFile 100 kB). Linux updates RSS counters lazily, so the true total is closer to ~1.8 MB. Either way, the box's ~800 MB of free RAM is untouched.

## 7. Footprint and undo

- On the TV: ~6 MB in `/data/local/tmp/tinyllm`, owned by the `shell` user. **No app, package, or setting changed.** The only thing that runs in the background is the optional web server (section 8) while it is started.
- On the Mac: `zig` (brew, ~212 MB) and this folder (`llm-experiments/`).
- Undo: `adb -s <tv-ip>:5555 shell "pkill -f llm-server; rm -rf /data/local/tmp/tinyllm"` (also in `../rollback.sh`). `brew uninstall zig` is in `../rollback.sh --mac`. Logged in `../CHANGELOG.md` 2026-11-03.

## 8. Using it from the TV itself: web page

![Tiny LLM page on the TV, mid-story](screenshots/tinyllm_04_done.png)

`tiny-llm-web/` turns the experiment into something you can use from the couch:

- **`server.c`** (~230 lines) `#include`s the same reviewed `run.c` and adds a single-threaded HTTP server. The model loads **once**, the process stays at ~1.9 MB RSS, and it serves:
  - `GET /`: the page
  - `GET /info`: model config as JSON
  - `GET /gen?prompt=…&t=…&delay=…`: a token stream. Each piece is followed by a marker byte: `0x1d` for a prompt token, `0x1f` for a generated one. A final `0x1e` is followed by stats JSON.
  - If the page starts a new story, the old request is aborted, `send` fails, and the old generation stops.
- **`index.html`**: a TV-layout page with a 5 % safe area (the Samsung crops the edges), big type, preset openings (no typing needed), Imagination = Steady / Creative / Wild (temperature 0 / 0.8 / 1.3) and Speed = Typewriter (60 ms per token, so you can watch it) / Full speed. Prompt text is blue and the newest token amber. The footer shows the live token count, then tok/s, seed and why it stopped. Arrow-key focus navigation is built in, but this browser uses an on-screen mouse cursor, so you just point and click with the remote.
- **`start.sh`**: builds `llm-server` (zig, static aarch64) if needed, pushes it, (re)starts it detached with `setsid`, and opens the page in the TV browser. `./start.sh stop` stops it.

**How to open it:** run `tiny-llm-web/start.sh` from the Mac, or on the TV open the browser (`com.internet.tvbrowser`) and go to **`http://<tv-ip>:8080/`**. Bookmark it with the ☆. Phones and the Mac on the same Wi-Fi can open the same URL.

**Gotchas found:**
- That browser treats `http://127.0.0.1:8080/` (and `localhost`) as a **Google search**, so use the LAN IP. If the router gives the box a new IP, the bookmark changes too.
- `adb shell "cmd &"` hangs, because adb waits on the background process's file descriptors. Fix: `(setsid cmd </dev/null >log 2>&1 &) </dev/null >/dev/null 2>&1`. The server keeps running after `adb disconnect` (verified).
- It does **not** survive a TV reboot, since nothing on Android can autostart a shell-user process without root. After a reboot, run `start.sh` again.
- It listens on port 8080 on the home network with no password. All it can do is write children's stories, but stop it (`./start.sh stop`) if that bothers you.

Measured through the server: "The little dog", greedy → 217 tokens, ending naturally, **731 tok/s** compute (forward passes / CPU time).

## 9. Going bigger: real chat models with llama.cpp

Same box, same trick (static aarch64 binary), but now with real instruction-tuned models.

### Build

- **Runtime:** [llama.cpp](https://github.com/ggml-org/llama.cpp) release **v0.8.0** (commit `8642541`, 28 Oct 2026). I couldn't review it line by line like `run.c`, so I built the pinned official release with your approval.
- **Why build it at all:** the official `android-arm64` release build **can't run on this box**. It needs Android's 64-bit loader, and this 32-bit Android has none (`/system/bin/linker64` and `/system/lib64` don't exist). A fully static musl build sidesteps that.
- **Compiler:** CMake + Ninja, with zig as the cross-compiler through a two-line wrapper (`zigcc/`). The wrapper drops ggml's `-march=armv8.2-a+dotprod+fp16`, which zig rejects, and passes `-mcpu=cortex_a55` instead, which enables the same dotprod + fp16 instructions.
- **CMake options:** `-DGGML_NATIVE=OFF -DGGML_OPENMP=OFF -DLLAMA_OPENSSL=OFF -DBUILD_SHARED_LIBS=OFF -DCMAKE_EXE_LINKER_FLAGS=-static`.
- **Output:** `llama-cli`, `llama-server`, `llama-bench`, stripped with `llvm-strip` from 125–177 MB to 9–15 MB each. They're kept in `bin/llama-tv/` (with `SHA256SUMS`); the source and build tree were deleted afterwards to save Mac disk.

### Models (in `/data/local/tmp/llm`, checksums match Hugging Face)

| Model | Released | Params | File | SHA-256 |
|---|---|---|---|---|
| Gemma 3 270M-it, Q4_K_M ([unsloth](https://huggingface.co/unsloth/gemma-3-270m-it-GGUF)) | Aug 2025 | 268 M | 253 MB | `b1baabd6…8e39` |
| Qwen3.5-0.8B, Q4_K_M ([unsloth](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF)) | Mar 2026 | 752 M (text) | 533 MB | `bd258782…c517` |

### Speed (`llama-bench`, 4 threads)

| Model | Prompt processing | Generation | vs stories260K |
|---|---|---|---|
| stories260K (llama2.c, fp32) | — | ~650 tok/s | 1× params |
| Gemma 3 270M | 70 tok/s | **21.9 tok/s** | ~1,000× params |
| Qwen3.5-0.8B | 31 tok/s | **6.7 tok/s** | ~2,900× params |

Gemma generation with 1 / 2 / 4 threads: 10.8 / 18.6 / 21.9 tok/s. At 4 threads it's limited by memory bandwidth, not compute.

### Answers (verbatim, from the TV, greedy unless noted)

| Prompt | Gemma 3 270M | Qwen3.5-0.8B (thinking off) |
|---|---|---|
| "Why is the sky blue? 3 sentences" (t=0.7) | Rayleigh scattering, short, a bit circular | Fluent, mostly right; one slip ("Rayleigh scattering … at the Earth's surface") |
| "17 sheep, all but 9 run away, how many left?" | "There are 17 sheep left." ✗ | "18" ✗ |
| Haiku about a TV box that learned to think | "A tiny box, / A world of stories untold, / Learned to think, small and bright." | "A tiny box, / With eyes that glow, / And a mind that grows." |
| Capital of Australia, one word | "Sydney" ✗ | "Canberra" ✓ |

Takeaway: both models are real assistants that follow instructions and keep to the format. Neither can reason under 1B params, and trick questions fool both. Qwen knows more facts and Gemma is 3× faster.

### Memory: the real limit

- The box has ~650–770 MB free with its normal apps running.
- Qwen3.5-0.8B peaked at **~960 MB RSS**. That counts the 533 MB weights file as mmapped pages, plus the context and compute buffers.
- To make room, Android's low-memory killer **closed 7 cached background apps** (YouTube recommendations, TTS, Assistant search, Play instant-app installer, …). Android restarts these on demand, and nothing in the foreground was touched.
- A 1B+ model (Llama 3.2 1B, Gemma 3 1B ≈ 800 MB at Q4) would start pushing out things you'd notice. ~0.8B at Q4 is about the ceiling for this box.
- (AirPlay Server was found not running, but it died at 10:11, three hours before these runs, so it's unrelated.)

### Reproduce

```sh
adb -s <tv-ip>:5555 shell 'cd /data/local/tmp/llm && ./llama-cli -m Qwen3.5-0.8B-Q4_K_M.gguf -t 4 -c 2048 -n 300 --reasoning off -st -p "Your question" </dev/null'
```

`-st` = single turn then exit. Leave it out for an interactive chat in `adb shell`. `--reasoning on` lets Qwen think first, but at ~6 tok/s that takes minutes.

## 10. Bigger than RAM: streaming weights from flash (Qwen3.5-2B)

The idea: run a model that **doesn't fit in free RAM** by leaving the weights memory-mapped from flash. The kernel pages them in as each layer is used, evicts them when memory runs out, and reads them back on the next token.

**Model:** Qwen3.5-2B Q4_K_M ([unsloth](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF)), 1,280,835,840 bytes, SHA-256 `aaf42c8b…9223` (matched on Mac and TV). That's 1.28 GB against ~0.85–0.9 GB free RAM.

**Run:** `-nr` (`--no-repack`) is the key flag. With repacking on, llama.cpp copies weights into anonymous RAM and the run would OOM; without it they stay file-backed and can be evicted and re-read.
```sh
sh llm-stream-monitor.sh q2b ./llama-cli -m Qwen3.5-2B-Q4_K_M.gguf -t 4 -c 512 -n 60 --temp 0 -st -nr --reasoning off \
  -p "What is the capital of Australia, and why was it chosen over Sydney? Two sentences."
```
`tools/llm-stream-monitor.sh` samples `/proc/vmstat` `pgpgin` (flash reads), the process's major page faults, RSS and MemAvailable every 2 s. `/proc/diskstats` is not readable as the shell user.

**Answer** (verbatim, capped at 60 tokens):
> The capital of Australia is Canberra, which was chosen because it was designed to be a neutral, non-capital city that avoided the political tensions of the two competing colonial capitals, Sydney and Melbourne.
> Canberra was selected as the capital in 1908 after a decade of planning, with the…

The answer is correct, and a clear step up from the 0.8B.

| Metric | Value |
|---|---|
| Speed | prompt 1.8 tok/s, **generation 0.1 tok/s (~8 s per token)** |
| Wall time | 529 s (8 min 49 s) for the prompt + 60 tokens |
| **Read from flash** | **83.8 GB** for one short answer |
| Flash throughput | ~200 MB/s sustained (peaks ~290 MB/s) |
| Major page faults | ~265,000 |
| Process RSS | flat at ~700–840 MB the whole run |
| MemAvailable | flat at ~850–870 MB, **no app kills** (unlike the 0.8B run, §9) |

**What happened:** a forward pass reads every weight once, in order. The file is bigger than what the kernel can keep cached, so in a sequential scan like that, eviction always throws out the part that will be needed soonest. In practice **the whole 1.28 GB model is re-read from flash for every token**: 60 tokens × 1.28 GB ≈ 77 GB, plus the prompt pass, ≈ 84 GB measured. Speed is set purely by the eMMC: 1.28 GB ÷ ~200 MB/s ≈ 6–8 s per token, which matches. The CPU mostly waits.

The upside: because the weights are file-backed, the page cache gives them up under pressure. The model **shrinks itself instead of killing other apps**, which makes streaming a gentler way to run something big than a model that fits.

**Stopped at 2B on purpose.** Qwen3.5-4B (2.74 GB, bigger than the box's total 1.9 GB of RAM) was downloaded and checksum-verified (`00fe7986…11a4`) but deleted unrun at your call. By the same arithmetic it would be ~14 s per token, with ~2.7 GB read per token.

**Flash wear:** this is all reads. eMMC wear comes from writes, so streaming doesn't wear the flash out.

## 11. Peak-performance push for Qwen3.5-2B

**Goal (your call):** make Qwen3.5-2B as fast as this box allows. You can edit frameworks or write our own. **Baseline:** 0.1 tok/s, streamed from flash (§10).

### 11.1 Where the bytes go

`tools/gguf_info.py` reads a GGUF header, locally or over an HTTP range request. Qwen3.5-2B Q4_K_M breaks down as:

| Part | Size | Read per token |
|---|---|---|
| `token_embd` 248,320 × 2048, Q6_K | **417 MB (33 %)** | **all of it**: embeddings are tied, so the output head multiplies the whole table |
| 24 layers (Q4_K 514 / Q5_K 208 / Q6_K 127 MB) | 853 MB | all of it |

The input side only needs one 1.7 KB row per token. Only the output head forces the full 417 MB read.

### 11.2 Experiments run so far

| # | Experiment | Result |
|---|---|---|
| E1 | **Pruned LM head**, 32,801 rows (ids < 32768 + 33 control tokens), via a llama.cpp patch + `tools/gguf_prune_head.py` | **0.1 → 4.76 tok/s** generation, prompt 14.9 tok/s (`llama-bench`, 4 threads). The model is RAM-resident: flash reads drop to ~0 after load. |
| E1b | Same, quality check | ✗ "Canberra" became "the Australian Capital City (ACQ)": `ĠCanberra` is token **66463**, outside the kept ids (likewise `ĠCairo` 50779, `ĠNile` 73583, `ĠXiaomi` 69025). Ranking tokens by id is too crude. |
| E2 | Weight repacking on vs off (`--repack 1/0`) | 4.76 vs **0.40** tok/s. Repacked weights live in anonymous RAM, which Android doesn't drop like file cache. Never use `-nr` here. |
| E3 | Threads 1/2/3/4 | 2.37 / 3.98 / **4.98** / 4.49 tok/s: memory-bandwidth bound at ~4.5 GB/s. So **speed ≈ 4.5 GB/s ÷ bytes per token**, and 3 threads is best. |
| E4 | Vocab coverage from real text (Python port of the Qwen3.5 tokenizer, matches llama.cpp token for token; 1.5 M-token corpus: Wikipedia, 4 Gutenberg books, code, docs) | Held-out coverage with ~33K rows: **97.9 %** when picked by frequency vs **91.4 %** by id. All 42,167 tokens seen in training text: 98.7 %. A hybrid set (frequent + low ids) is next. |
| E5 | Which quants get fast repacked kernels on an A55 (dotprod, no i8mm) | Q4_0, IQ4_NL, Q4_K, Q5_K, Q6_K, Q8_0: yes. IQ4_XS, Q3_K, Q2_K: no. Per-token bytes with a 33K head: Q4_K_M 908, Q4_K_S 845, IQ4_NL 840, Q4_0 842, IQ4_XS 800, Q3_K_M 734 MB. |
| E6 | **MTP head** (multi-token prediction, for speculative decoding) | unsloth's GGUFs drop it, but the original checkpoint has it (15 `mtp.*` tensors). `tools/fetch_mtp.py` range-reads only those tensors from the 4.5 GB safetensors file, applies the converter's norm +1, and quantizes to Q8_0: **64.7 MB**. Norm scales check out against the main layers. The llama.cpp patch also scatters pruned logits in the MTP graph. **Not run yet.** |
| E7 | Live web page (`tv-llm-ui/`) | Works: tokens stream with confidence colors, top-3 alternatives, live tok/s, model/config/box panels. **The TV browser (~300 MB) plus the 2B doesn't fit** (95 MB free, load 22, so I stopped it). The 0.8B runs next to the browser at 4.7 tok/s on screen. You chose to watch the 2B from a phone/Mac. |
| E8 | Server benchmark (`bench.py`, 4 fixed prompts, greedy, 128 tokens) | 3.4–3.6 tok/s on 3 prompts, then **0.38 tok/s** on the 4th: Android respawned background services (Projectivy jobs, Play Services, launcher services, Xiaomi quick settings) and pushed the model out (118 MB free, 424 MB/s flash). Short `llama-bench` runs never showed this. |
| E9 | **Disabled 7 background apps** (your request; list + undo in `../CHANGELOG.md` / `../rollback.sh`) and shrank buffers (ctx 1024, micro-batch 128) | Idle: model process 1,021 MB, 299 MB free, flash 0 MB/s. **During your generation from the page it thrashed again** (83 MB free, 1,171 MB/s flash, load 21), so I stopped the server at your request. |

### 11.3 What this means

At ~1 GB the Q4_K_M 2B is **too big for sustained use** on this box, even with apps disabled: Android's own floor plus whatever it respawns leaves ~1.0–1.15 GB at best. **The model has to drop to ~750–800 MB to stay resident.** Next steps, in order:

1. **Leaner weights:** IQ4_NL or Q4_K_S (−60 MB, still fast kernels), or Q3_K_M (−120 MB, slower kernels). Measure speed, memory over time, and answers.
2. **Frequency-chosen head** (~40K rows, ~67 MB) so "Canberra" survives; E1b gets re-checked.
3. **MTP speculative decoding** on top: ~1.5× expected if ~70 % of drafts are accepted, since generation is bandwidth-bound.
4. Every run is logged to `bench/results.jsonl` (prompt, text, speed) and summarized here.

### 11.4 Tools added in this push

| File | What it does |
|---|---|
| `tools/gguf_info.py` | GGUF header reader (local file or URL range request) |
| `tools/gguf_prune_head.py` | adds `output.weight` (kept rows, raw bytes) + `output_ids.weight` |
| `tools/vocab_select.py` | Qwen3.5 tokenizer port + coverage stats + kept-id lists |
| `tools/fetch_corpus.py` | Wikipedia + Gutenberg corpus |
| `tools/fetch_mtp.py` | range-fetch + convert the MTP tensors |
| `tools/tv-free-ram.sh` | force-stop heavy apps before a run |
| `tools/llm-stream-monitor.sh` | flash/RSS sampler |
| `tv-llm-ui/` | `start.sh`, `index.html`, `boxstats.sh`, `bench.py` |

The llama.cpp patch (`third_party/llama.cpp`, v0.8.0 + 4 files) adds the `LLM_TENSOR_OUTPUT_IDS` tensor and a `set_rows` scatter in the qwen35 main and MTP graphs. Patched binaries live in `/data/local/tmp/llm/patched/` on the TV.

### 11.5 Round 2: pushing to the limit (2026-11-03, evening)

Goal: exhaust every way to make the 2B faster on this box without breaking its answers. Every run is listed here; raw logs are in `bench/raw/`, benchmark runs in `bench/results.jsonl`.

| # | Experiment | Result |
|---|---|---|
| E10 | **What can the RAM deliver?** `tools/membw/membw.c`: NEON streaming read of a 256 MB buffer, 1–4 threads | **6.7 GB/s on 1 core, 8.3–8.4 GB/s on 2–4 cores.** llama.cpp only reached ~4.5 GB/s effective (E3), so generation is **not** memory-bound yet: there is up to ~1.85× headroom in compute. Raw: `bench/raw/E10-membw.txt` |
| E11 | **Where does a token's time go?** Added a per-op profiler to ggml (`GGML_OPPROF=1`: thread 0 times every graph node, including the barrier wait, and prints a table at exit). Q4_K_M + 32K head, 3 threads, tg32 | Q4_K matmuls 34 % at **6.2 GB/s** (close to the RAM limit) · **Q6_K matmuls 22 % at only 3.4 GB/s** (slow kernel; Q4_K_M keeps Q6_K for some layers *and* the head) · Q5_K 18 % at 4.7 GB/s · ~900 small ops per token (norms, copies, DeltaNet, conv) ~20 %, i.e. ~50–80 µs each, far more than their math: mostly thread sync / scheduling · embedding lookup (`GET_ROWS q6_K`) 5 %, page faults on the memory-mapped table. Raw: `bench/raw/E11-opprof-q4km-h32k-t3.txt` |
| E12 | **Thread options** (`llama-bench` tg48 ×3, Q4_K_M + 32K head): default, `--poll 0/100`, pinning (`-C 0xE`/`0x7 --cpu-strict 1`), `-t 4`, `--prio 2` | All within noise: 4.0–4.9 tok/s, ±0.2–0.8. Pinning is slightly *worse* (4.4–4.5): Android's own threads then compete with ours on fixed cores. No option removes the per-op overhead, so it's real work + sync, not polling. (Partly overlapped a Wi-Fi transfer; re-checked later on the final model.) Raw: `bench/raw/E12-thread-options.txt` |
| E13 | ⚠ **Incident: kernel panic + reboot.** First `llama-quantize` run on the TV | llama-quantize's default slab buffer is 8 GB, so converting the 1 GB BF16 embedding table (508 M values) wanted a ~2 GB float buffer on a 1.9 GB box: memory ran out and the **kernel panicked** (`sys.boot.reason=kernel_panic`, uptime reset). The 4 GB input survived (hash re-checked); the files written in the seconds before the crash were lost. Fix: `--max-buffer-size 64` (MiB slabs) in `tools/tv-quant.sh`, and every heavy job now runs under `tools/tv-guard.sh`, which samples MemAvailable every second and kills the job below 180 MB. With both, quantizing holds ~770 MB available. |
| E14 | **All-Q4_K mix, quantized on the TV from BF16** (`tools/tv-quant.sh q4k`: `--pure` Q4_K for every matrix, embedding table and head; unsloth imatrix) vs unsloth's Q4_K_M, both with the same id-ranked 32K head | **4.82 → 5.59 tok/s generation (+16 %), 14.5 → 17.4 tok/s prompt (+20 %)**, file 1.23 → 1.05 GiB. The Q6_K/Q5_K time is gone: all matmuls now run the fast Q4_K kernel (5.2–6.6 GB/s). Quantizing took 166 s on the TV (min 701 MB free). Raw: `bench/raw/E14-q4km-vs-q4k-pure.txt` |
| E15 | **Benchmark artefact:** the embedding lookup still costs ~10–13 ms per token in `llama-bench` | `llama-bench` feeds random token ids from the whole 248K vocab, so nearly every token reads a cold row of the memory-mapped embedding table from flash. Real text reuses frequent tokens that stay cached. Added `LLAMA_BENCH_COMMON_TOKENS=N` (draw from ids 256…256+N) to the patched llama-bench; `tools/tv-eval.sh` now uses N = 4000 and 5 repetitions (run-to-run noise on this box is ±5–10 %). |
| E16 | ⚠ **Perplexity at context 512 ran the box out of memory** (guard killed all 4 runs at 76–147 MB free; no crash) | `llama-perplexity` keeps logits for every position: 512 × 248,320 × 4 B = 508 MB on top of the model. Switched to context 128 × 40 chunks (127 MB of logits, 2,560 scored tokens). |
| E17 | **Which tokens should the head keep?** All-Q4_K model with three heads, quantized identically: B = 32,801 rows by token id (old), C = **H45** (45,654 rows: frequency-picked from English/code + ar/zh/ja/es/fr/de/ru Wikipedia, plus all ids < 16384), D = **H65** (65,668 rows, bigger budgets). *Misses* = held-out tokens the head cannot produce at all | Speed (common tokens): B **6.18**, C 6.13, D 6.03 tok/s; prompt 17.4 / 17.2 / 16.8. Doubling the head costs only ~2 %. Misses on multilingual text: **B 50.9 % → C 19.7 % → D 12.6 %**; on English **B 10.0 % → D 1.3 %**. ⚠ *Corrected in E29:* this row first said D missed 1.3 % of multilingual tokens; the guard had killed a different run for C (English) and D (multilingual) than I assumed, and the 1.3 % was D's English figure. PPL values are not comparable across heads (each skips different positions). Conclusion: the id-ranked head was badly hurting non-English text; a ~65K frequency head is nearly lossless on English and much better on other languages, for ~2 % speed. Raw: `bench/raw/E17-heads-B-C-D-raw.txt` |
| E18 | **Which 4-bit kernel is fastest on the A55?** Q4_0 and IQ4_NL "pure" variants (same 4.5 bits/weight, same 32K head) vs Q4_K | Generation / prompt tok/s: **Q4_0 6.47 / 19.7** · Q4_K 6.18 / 17.4 · IQ4_NL 5.74 / 10.6 (noisy). Q4_0 has the simplest decode, so it's ~5 % faster to generate and 13 % faster on prompts. **But quality:** English perplexity Q4_0 **9.157** vs Q4_K **8.890** (+3.0 %, same positions). Raw: `bench/raw/E18-kernels-q40-iq4nl.txt` |
| E19 | **Where does the RAM go?** `tools/memprobe.sh` (llama-server + one real chat request, `/proc/<pid>/status`), Q4_K + H65; plus a loader patch that drops the source pages of every tensor copied into the repacked buffer (`madvise(MADV_DONTNEED)` + `posix_fadvise(POSIX_FADV_DONTNEED)`; `LLAMA_KEEP_MAPPED_COPIES=1` = stock) | Stock: RSS 1,092 MB = **832 MB anonymous** + 259 MB mapped file pages; MemFree 38 MB after load. Patched: RSS 985 MB = 851 MB anonymous + **134 MB** file; **MemFree 205 MB** after load. MemAvailable is about the same (~385 MB → ~340 MB after a request) because clean page cache already counted as "available". **The real cost is the ~850 MB of anonymous memory** (771 MB repacked layers + 76 MB head + buffers), plus 44–74 MB already pushed to zram swap. Only fewer weight bytes, or fewer background apps, can fix that. Raw: `bench/raw/memprobe-D-*.txt` |
| E20 | **Quality reference: how much does 4-bit cost?** English perplexity of the BF16 (full precision) model, same 32K head, streamed from flash (130 s per 128-token chunk, so stopped after 15 of 40 chunks); others compared at the same chunk | Running PPL at chunk 15: **BF16 8.785 · unsloth Q4_K_M 9.036 (+2.9 %) · all-Q4_K 9.262 (+5.4 %) · Q4_0 9.664 (+10.0 %)**. Final (40 chunks): Q4_K_M **8.687**, all-Q4_K 8.890, Q4_0 9.157 (Q4_K_M streamed with `-nr` so it fits). So all-Q4_K gives up ~2.3 % perplexity versus Q4_K_M for +16 % speed and −180 MB; Q4_0's extra +5 % speed costs twice that again. **Q4_K stays the base format**; a "Q4_K + a few upgraded sensitive tensors" mix is queued as a middle ground. |
| E21 | **3-bit: Q3_K_M (stock) and Q4_K with 3-bit FFN up/gate** | Generation / prompt: Q3_K_M **4.12 / 7.2**, Q4_K+Q3_K-FFN **4.66 / 8.8** (vs all-Q4_K 6.18 / 17.4). Q3_K has no repacked kernel on this CPU and runs at only **1.7–2.0 GB/s** of weights. Side effect: tensors that aren't repacked stay memory-mapped file pages, so min free RAM during the runs was 680–800 MB instead of ~350 MB. **3-bit needs a repacked kernel**; that is now on the custom-kernel list (it would cut ~25 % of the weight bytes). Raw: `bench/raw/E21-q3k.txt` |
| E22 | **2-bit sweep** (§11.6): layers at IQ2_M / IQ2_XXS / Q2_K / Q2_0 with the imatrix; embedding table, head and MTP block stay Q4_K; same 32K head. IQ2_XXS was quantized by the TV and IQ1_M/IQ1_S by the Mac (`tools/mac-lowbit.sh`, 104 s on the Mac vs several minutes on the TV) | Gen / prompt tok/s, file, final PPL (BF16 ≈ 8.8, Q4_K 8.89): **IQ2_M** 3.12 / 4.96, 823 MiB, 12.32 · **IQ2_XXS** 4.16 / 5.57, 751 MiB, 20.23 · **Q2_K** 4.46 / 7.23, 857 MiB, 12.20 · **Q2_0** 4.56 / 7.32, 712 MiB, **377,511 (collapsed)**. All are slower than Q4_K (no repacked kernels) and none is much smaller (the Q4_K embedding/head/MTP parts are ~350 MB). Answers: IQ2_M and Q2_K are fluent but say Australia's capital is "**Canterbury**" and loop; IQ2_XXS says "Sydney" and degenerates into "I'm sorry. I'm sorry…" / `self = self`. **Post-training 2-bit breaks this 2B.** Raw: `bench/raw/E22-lowbit-2bit.txt` |
| E23 | **1-bit sweep**: IQ1_M (1.75 bits), IQ1_S (1.56), Q1_0 (1.125, binary ±1; the Bonsai format) on the 24 layers | **IQ1_M**: 3.32 / 5.30 tok/s, PPL **35.4**, answers are "**Answer** **Answer** **Answer**…". **Q1_0**: the fastest generation in this whole push, **7.40 tok/s, prompt 24.0** (it has a repacked dotprod kernel, and the layers are only ~190 MB), but PPL **75,679** and both answers empty: binary weights applied after training destroy a normal model, as expected. **IQ1_S** (re-run after the fix below): 4.16 / 6.87 tok/s, PPL **70.5**, answers "Answered by the answer is the answer… 1. 1. 1.". The first copy failed to load. The Mac→TV hand-off had fetched the file while llama-quantize was still writing it (591 MB of 731 MB). Fixed in `tools/mac-lowbit.sh` (write to a temp name, publish when finished, check the size on the TV before renaming); redone and re-queued. **Conclusion: post-training 1-bit is unusable; the only route to 1-bit is a model trained for it** (§11.7). Raw: `bench/raw/E23-lowbit-1bit.txt` |
| E24 | **Middle-ground mixes**: all-Q4_K with the most sensitive tensors upgraded (`tools/tv-mixes.sh`, same 32K head). M1 = Q4_K + `attn_v` at Q6_K + `ffn_down` of the first and last 4 layers (0–3, 20–23) at Q5_K; M2 = M1 + `ssm_out` (the DeltaNet output projections) at Q5_K | Gen / prompt tok/s, file, final English PPL: all-Q4_K 6.18 / 17.4, 1,091 MB, 8.890 · **M1** 5.82 ± 0.84 / 17.1, 1,104 MB, 8.832 · **M2** **6.09** ± 0.20 / 15.2 ± 2.0, 1,113 MB, **8.805** · unsloth Q4_K_M 4.82 / 14.5, 1,260 MB, 8.687. M2 recovers ~42 % of the gap to Q4_K_M for +22 MB and no measurable generation-speed loss (the upgraded tensors are only ~4 % of the bytes). The prompt numbers are noisy (±2). **M2 is the new quality/speed pick**; queued again with the H65 multilingual head (`tools/tv-heads2.sh`). Raw: `bench/raw/E24-mixes.txt` |
| E25 | **Models trained for 1-bit / ternary** (§11.7, `tools/tv-others.sh`): speed, peak RSS, English perplexity → **bits per byte** (lower is better; each model's own token count of the 22,031-byte `eval-en.txt`), two greedy answers | Gen / prompt tok/s · max RSS · PPL · bits/byte: **Bonsai-1.7B (Q1_0)** 6.81 / 28.8 · 514 MB · 21.59 · 1.158 — **Bonsai-4B (Q1_0)** 3.17 / 11.4 · 1,090 MB · 17.21 · 1.073 — **Ternary-Bonsai-1.7B (Q2_0)** 4.33 / 7.6 · 556 MB · 17.13 · 1.071 — **BitNet b1.58 2B4T (TQ2_0)** 4.60 / 8.2 · 794 MB · 66.42 · 1.546. Reference, measured with the full 248K vocabulary (nothing skipped, E29): **Qwen3.5-2B all-Q4_K = PPL 10.478 = 0.909 bits/byte**, clearly better than all four (the earlier 0.845 estimate skipped the 10 % of tokens the 32K head can't produce). Answers: all three Bonsai models answer **Canberra** correctly and write a correct, documented `is_prime` (their reasons for choosing Canberra are vague); BitNet loops ("chosen over Sydney due to its being chosen over Sydney…") and writes broken code. **Verdict:** trained 1-bit models are usable, unlike post-training 1-bit (E23), but none beats Qwen3.5-2B at Q4_K: Bonsai-1.7B is ~10 % faster with half the RAM but clearly weaker; Bonsai-4B is about as good as the 1.7B ternary but half the speed. Raw: `bench/raw/E25-trained-lowbit.txt` |
| E26 | **Why don't fewer bits mean more speed?** Op profiles of every format measured so far, turned into weights per second | Matmul throughput in **billions of weights/s** (3 threads): Q4_K **11.0** (6.2 GB/s) · Q1_0 **12.1–13.4** (1.7–1.9 GB/s) · TQ2_0 **12.3** (3.2 GB/s) · Q2_0 8.3 · Q5_K 6.8 · Q6_K 4.1. Every repacked/simple format lands at **~11–13 G weights/s ≈ 1.5 weights per cycle per core**, whatever the bit width: the A55 matmuls are limited by **decode + dot-product instructions**, not by memory. That's why Bonsai-1.7B (1/4 the bytes of our 2B) is only 10 % faster, and why the 1-bit Qwen (E23) only reached 7.4 tok/s. Consequences: (1) fewer bits only help if the kernel's per-weight work drops too; (2) the Q4_K kernel itself is the lever: at the RAM limit (8.4 GB/s, E10) it would do ~15 G weights/s, **+35 %** headroom. The kernel micro-benchmark (`tools/kbench`) is next in the TV queue. Source: the `[opprof]` tables in `bench/raw/E11`, `E14`, `E25` |
| E27 | **Can a better Q4_K kernel get closer to the RAM limit?** `tools/kbench` on the TV: llama.cpp's 8x4 repacked gemv ("ref") vs an A55-scheduled variant ("v2": all 16 loads issued first, 8 independent dot-product accumulators, bias folded per super-block), each with software prefetch at 0–8 KB ahead; a 54 KB matrix (stays in cache) and a 27 MB one (from DRAM) | Cache-resident: ref **4.05 GB/s on 1 core, 11.4 on 3**; v2 3.9 / 10.6. From DRAM: ref **3.37 / 6.69 GB/s**, v2 3.24 / 6.83. Software prefetch never helps (−2 % to −25 %): the A55's hardware prefetcher already streams these sequential reads. So the stock kernel runs at **81 % of the measured 8.3 GB/s RAM ceiling** on 3 cores, and its pure compute rate (11.4 GB/s) is only 1.4× the RAM rate, so a perfect overlap would gain at most ~20 % and my rescheduling gained nothing. **The Q4_K kernel route is closed.** Same queue: final English PPL **Q3_K_M 9.64**, **Q4_K + Q3_K FFN 9.25** (vs Q4_K 8.89), slower and worse, which also closes 3-bit (a Q3_K repack kernel would cut bytes 25 % but adds decode work on a core that's already near its compute limit). Raw: `bench/raw/E27-kbench-and-q3-ppl.txt` (includes the op profile of the current best model: matmuls 74 %, GATED_DELTA_NET 6.2 %, RMS_NORM 5.8 %, CPY 3.9 %, GET_ROWS f32 2.9 %) |
| E28 | **Fused one-pass DeltaNet kernel** (patch, `patched3/`): the CPU `GATED_DELTA_NET` kernel made 5 passes over each head's 64 KB state per token (copy, decay, k-dot, update, q-dot); it now makes one (row j only needs its own delta, so k-dot, update and q-dot happen while the row is in L1). The new state is also written straight into the recurrent cache (`src[6]` = cache view) instead of a separate 1 MB/layer copy. Switches back to stock: `GGML_GDN_STOCK=1`, `LLAMA_GDN_COPY=1` | Op profile (Q4_K + H65, tg32): `GATED_DELTA_NET` **614 → 399 µs/call (−35 %)**, the 2,376 state `CPY`s per 33 tokens (224 ms) are gone, total profiled time **5.27 → 4.91 s (−6.9 %)**. llama-bench alternated stock/new: tg64 6.31 / **6.71** and 6.15 / 6.09 (±0.1–0.8, this box's noise), pp128 18.06 → **18.65**. Accuracy: the direct state write alone is bit-identical (PPL 8.7254 = stock); the fused kernel gives PPL 8.7356 (+0.12 %, ±9 % stderr) and different greedy text. `tools/gdn-test/gdn_test.c` (2,000 recurrent steps, same NEON code vs a double-precision reference) shows the fused update is **3× more accurate** than stock (4.3e-8 vs 1.35e-7 max error: FMA, fewer roundings), so the difference is rounding propagated through a 4-bit network, not a bug. Note: the per-op profiler now also reports compute-only time; the earlier "~100 µs per RMS_NORM" (E27) was barrier/scheduling noise in that run (15–16 µs per call here). Raw: `bench/raw/E28-gdn-ab-and-vkprobe.txt` |
| E29 | **Head follow-ups** (`tools/tv-heads2.sh`): on-TV head re-cutter check; full-vocab baseline; H65 perplexity on both texts; M2 mix with the H65 head | `bin/gguf_recut` (C, runs on the TV) gives a **byte-identical** H45 file to `gguf_build.py` (SHA-256 a50eadc5…), so heads can now be re-cut on the box in seconds. H65: English PPL 9.957 with 1.31 % misses, multilingual 9.100 with **12.58 %** misses (corrects E17). **M2 + H65** (`w-m2-h65.gguf`, 1,205 MB): 5.82 / 16.1 tok/s, English PPL **9.859** (−1.0 % vs Q4_K + H65 on the same positions), multilingual **9.068** (−0.35 %). The full-vocab run (248K tied head, repacked) **ran out of memory** (69 MB left, guard kill); re-run streamed (`-nr`, `tools/tv-fullvocab.sh`, 16 min): **PPL 10.478, 0 misses = 0.909 bits/byte**, the fair number for comparing with other models (E25). Raw: `bench/raw/E29-heads-baseline.txt` |
| E30 | **The GPU** (Mali-G310, your request: "fastest speed on GPU, up to 2B"). The box's userland is 32-bit and its only GPU driver is `/vendor/lib/egl/libGLES_mali.so` (32-bit, also the Vulkan ICD), so GPU code must be a 32-bit Android build: Android NDK r28c, `armeabi-v7a`, llama.cpp's Vulkan backend (`bin/vulkan32/`), shaders compiled with Homebrew `glslc` 2026.4 (the NDK's is too old for the int8 dot-product path). `tools/vkprobe` first: Vulkan works from the adb shell (no root; `/dev/mali0` is world-writable), **Mali-G310, Vulkan 1.3.247, fp16, int8 + integer dot product, subgroup 16, 1,910 MB unified heap, no cooperative matrix; GPU read bandwidth 7.5–8.0 GB/s** (same DRAM as the CPU, 8.3), 93 µs per submit round trip | ⚠ **Kernel panic #2**: the first run offloaded the whole 2B (`-ngl 99`) under the old 1-second guard; the box rebooted (CHANGELOG 03:10), files pushed seconds before were lost. Since then every GPU run goes through `tools/tv-gpu.sh` (5 samples/s, echoed live to the Mac, kills multi-process jobs by name). **Where GPU-path memory goes** (0.8B, 497 MB file): llama.cpp's default 512-token compute buffers 299 MB GPU + 272 MB pinned host (fix `-ub 64`); CPU-side weights put in a pinned Vulkan host buffer, 361 MB (fix `--no-host 1`); the 32-bit CPU backend repacking its weights, +430 MB (fix `--repack 0`); the tied 248K embedding copied again as the output layer, 208 MB; Mali shader compiles, a ~100–250 MB peak in warm-up. **Tuning, 0.8B tg16** (CPU, 64-bit: **8.30 tok/s**): GPU default 2.05 · forced int-dot mat-vec 2.01 · 4-subgroup workgroups 1.63–1.66 · K-quant rows/workgroup 1/4/8: 2.08/2.01/1.98 · **int-dot + 4 rows: 2.25** · plus DeltaNet with 4/8/16 lanes per column (new `GGML_VK_GDN_LANES`; the Mali default packs 64 floats of state per lane): 2.36/2.52/**2.53** (`GATED_DELTA_NET` 3.76 → 0.80 ms/call). Best GPU = **2.53 tok/s = 30 % of the CPU**. **Ceiling test**: a minimal hand-written 4-bit × int8 mat-vec shader (`tools/vkgemv`, coalesced 16-byte loads, `dotPacked4x8AccSatEXT`, subgroup reduce) peaks at **4.3 GB/s (7.6 G weights/s)** on large matrices and 1.3–2.5 GB/s on the 2B's layer shapes, vs **6.7 GB/s** for the CPU kernel (E27): this GPU can't beat the CPU at generation even with a perfect kernel. **Prompts** (Gemma 3 270M, the only model whose batched GPU path fits): GPU pp64 38.5 / tg32 8.3 vs CPU 72.9 / 22.0, also slower. **2B**: doesn't fit at any split (`tools/tv-gpu-2b.sh`, 0/8/16/20 GPU layers: all stopped by the guard, lowest 121 MB free); the GPU path needs the same ~1.1 GB of weights plus ~300 MB of driver/compile/staging overhead, on a box with ~1.0 GB available. Projected from the 0.8B per-op profile it would run at ~1.4 tok/s vs 6.0–6.7 on the CPU. A CPU+GPU hybrid over llama.cpp RPC (64-bit CPU process + 32-bit GPU server, `tools/tv-rpc-hybrid.sh`) fails in the RPC handshake on this box, and couldn't help anyway: layers run in sequence, so moving them to a 3× slower device only adds time. **Verdict: the GPU route is closed**: the Mali-G310 is ~3× slower than the CPU per weight and the RAM can't hold the 2B on its path. Raw: `bench/raw/E30-*` (sweeps, per-op GPU profiles, vkgemv, memory traces) |
| E31 | **MTP speculative decoding** (the 2B's built-in next-token head, blk.24, already in our files): `llama-server --spec-type draft-mtp --spec-draft-n-max N`, `tools/mtp-bench.sh` (4 fixed chat prompts, greedy, 128 tokens, real HTTP requests), `patched3` (fused DeltaNet), Q4_K + H65 | **No MTP: 6.58 tok/s** generation, 17.3 prompt (best real-chat number so far; E14 was 5.6–6.2 in llama-bench). **n_max=1: 4.50** (draft acceptance 0.76) · **n_max=2: 1.59** (0.59) · **n_max=3: 2.88** (0.48). The draft is often right (the prime-number answer: 97 % accepted), yet every setting is slower. Why: on this core each extra token in the verification batch costs nearly a full pass (E26: matmuls are compute-bound, so batching 2 tokens saves little), plus the draft pass itself, plus K snapshots of every DeltaNet state. Some runs also hit memory pressure (one 0.77 tok/s answer). Raw: `bench/raw/E31-mtp.txt` |
| E32 | **Cheaper 2–3 token batches** (patch, `patched5/`): stock llama.cpp runs the 1-row gemv once per row for batches of 2–3 tokens; they are now padded to one 4-row gemm tile (each weight block decoded once). `GGML_REPACK_PAD4=0` = stock | llama-bench, 2B Q4_K + H65, 3 threads, tok/s: pp1 5.98 → 6.10 · pp2 **7.25 → 7.33** (no gain: a 4-row tile costs about two 1-row passes here, and stock pp2 already costs only 1.65× pp1) · pp3 **6.99 → 10.48 (+50 %)** · pp4 13.6 → 13.8. Helps any 3-token batch (MTP n_max=2 verification). MTP re-measured with it: see below. Raw: `bench/raw/E32-pad4.txt` |

**Plan from E10/E11:** (1) an all-4-bit mix with no Q6_K/Q5_K, quantized on the TV from the original BF16 weights, which is both smaller and runs on the fast kernel; (2) cut the per-op sync overhead; (3) a frequency-chosen, multilingual head; (4) MTP speculative decoding; (5) look at the embedding page faults.

**New tooling for round 2:**
- `tools/gguf_build.py`: streaming GGUF rewriter. Adds a pruned head from an id list (`--head-ids`), extra tensors (`--add`, e.g. the MTP layer) and metadata (`--set`) in one sequential pass, so `curl HF | gguf_build.py | adb exec-in` writes a modified 3.9 GB model to the TV without storing it on the Mac (~4 GB free). Verified: rebuilding the old 32K-head file gives an identical size, tensor table and head bytes.
- `tools/fetch_mtp.py --bf16`: keeps the MTP matrices as raw BF16 so llama-quantize quantizes them along with the model.
- The llama.cpp patch gains `GGML_OPPROF` (ggml-cpu.c) and pruned-head-aware perplexity (`llama-perplexity`: positions whose correct token is outside the kept vocab are counted as *misses* and skipped, so PPL is the pruned model's own renormalized distribution).
- BF16 checkpoint streamed straight to the TV (`/data/local/tmp/llm/Qwen3.5-2B-BF16.gguf`, 3,775,709,216 bytes, SHA-256 `dc11be2c…19f1`, matches Hugging Face). Note: a two-way pipe (TV → Mac → TV over adb) only does ~1 MB/s on Wi-Fi; one-way HF → Mac → TV does ~8 MB/s.

### 11.6 Queued: the 1 / 1.5 / 2-bit sweep (your request)

**Research first:**
- **Q1_0** (1.125 bits/weight, binary weights ±1 with one scale per 128) is PrismML's Bonsai format, made for models *trained* to be 1-bit ([PrismML formats](https://docs.prismml.com/download/formats), [Bonsai-8B GGUF](https://huggingface.co/prism-ml/Bonsai-8B-gguf)). It's merged into llama.cpp, and our v0.8.0 even has a repacked dotprod kernel for it. Applied after the fact to a normal model, expect gibberish; it's measured anyway for completeness.
- **IQ1_S / IQ1_M** (1.56 / 1.75 bits) and **IQ2_XXS…IQ2_M** (2.06–2.7 bits) are llama.cpp's post-training codebook quants and need an importance matrix (we have unsloth's). There's no published perplexity for Qwen3.5-2B at these sizes. A Qwen3.5-2B study found no layer tolerates 1-bit without training for it ([study](https://huggingface.co/daksh-neo/qwen3.5-1bit-quantization-study)); 2-bit is reported workable with tuning ([write-up](https://dev.to/genrius_ai/how-we-fit-a-27b-model-on-a-12-gb-laptop-gpu-14eo)); IQ2_XXS loses noticeably even at 7B ([guide](https://tonisagrista.com/blog/2026/quantization/)).
- **TQ1_0 / TQ2_0** (ternary) only fit BitNet-trained models. Skipped.
- None of the IQ1/IQ2/Q2_K types has a repacked kernel on this CPU (dotprod, no i8mm), so they run the slower plain NEON dot products. The trade is far less RAM (the layers drop from 771 MB at 4.5 bits to ~350 MB at 2 bits, ~190 MB at 1.1 bits) against speed and quality.

**Plan** (`tools/tv-lowbit.sh`, queued to start when the Q4_K_M / Q3_K jobs finish). Variants in order: IQ2_M, IQ2_XXS, Q2_K, Q2_0, IQ1_M, IQ1_S, Q1_0. Each one:
1. Quantize the 24 layers from BF16 with the imatrix. The embedding table, head and MTP block stay Q4_K (no imatrix data for them).
2. Speed (`tv-eval.sh speed`: tg64 common tokens, pp128, op profile).
3. English perplexity, same 32K head as B/E, so it compares directly with BF16 8.785 / Q4_K 9.262 at chunk 15.
4. Two greedy answers (Canberra question, prime function) to check whether the output is still coherent.

Results go to `eval/lowbit-summary.txt` on the TV, then here.

### 11.7 Trained 1-bit and ternary models (your call: not limited to Qwen)

Post-training 1–1.75 bits can't be expected to work on Qwen3.5-2B (§11.6), but models *trained* at those widths exist:

| Model | Bits | File | Notes |
|---|---|---|---|
| [PrismML Bonsai-1.7B](https://huggingface.co/prism-ml/Bonsai-1.7B-gguf) | 1 (Q1_0, ±1 weights) | 248 MB | Qwen3 architecture, 28 layers; Q1_0 has a repacked dotprod kernel in our llama.cpp |
| [PrismML Bonsai-4B](https://huggingface.co/prism-ml/Bonsai-4B-gguf) | 1 (Q1_0) | 572 MB | Qwen3, 36 layers: 4B parameters in less RAM than our 2B's head + MTP |
| [PrismML Ternary-Bonsai-1.7B](https://huggingface.co/prism-ml/Ternary-Bonsai-1.7B-gguf) | 1.58 (Q2_0, group 64) | 490 MB | the `_g64` file matches mainline's Q2_0 layout |
| [Microsoft BitNet b1.58 2B4T](https://arxiv.org/pdf/2504.12285v1) ([TQ2_0 GGUF](https://huggingface.co/Synapticode/bitnet-b1.58-2B-4T-tq2_0-gguf)) | 1.58 (TQ2_0) | 1.2 GB | 657 MB of it is an F16 tied embedding/head; requantized to Q4_K on the Mac (ternary layers copied byte-for-byte) |

Bonsai-8B (1.16 GB of 1-bit weights, all of which would be repacked into RAM) and Ternary-Bonsai-4B (1.07 GB) are over this box's RAM budget. Evaluation (`tools/tv-others.sh`): speed, memory, English perplexity converted to **bits per byte** (each model's own token count of `eval-en.txt`; Qwen3.5 uses 5,905 tokens for its 22,031 bytes), and the same two prompts.

Work is split between the machines (your call): the Mac quantizes the slow IQ2_XXS / IQ1_M / IQ1_S variants from the identical BF16 file (same SHA-256) with a native llama-quantize build (`tools/mac-lowbit.sh`) and hands them to the TV over HTTP, while the TV quantizes the fast types and runs every evaluation.

### 11.8 Final state of the 2B push (2026-11-04)

**Final configuration** (`tv-llm-ui/start.sh` defaults): `patched5/llama-server`, model `w-q4k-h65.gguf` (all-Q4_K from the BF16 weights with the unsloth imatrix, 65,668-row frequency-chosen multilingual head, MTP block kept but unused), 3 threads, context 1024, `-ub 128 -b 128`, MTP off.

| Step | Generation tok/s | Prompt tok/s | English PPL |
|---|---|---|---|
| Start of the push: unsloth Q4_K_M, id-ranked 32K head, stock kernels | 4.8 (llama-bench) | 14.5 | 8.69* |
| All-Q4_K from BF16 (E14) | 5.6–6.2 | 17.4 | 8.89* |
| + 65K multilingual head (E17, E29) | 6.0 | 16.8 | 9.96** |
| + fused DeltaNet kernel, direct state write (E28) | 6.3–6.7 | 18.7 | 9.96** |
| **Final, real chat requests via llama-server** (E31–E33) | **6.55** (llama-bench **6.69**) | **17.3** | 9.96** |
| Quality option `w-m2-h65.gguf` (E24, E29) | 6.20 | 16.4 | 9.86** |

\* 32K-head positions, \*\* 65K-head positions (not comparable across heads; each skips the tokens its head can't produce).

**Routes tried and closed** (each measured, see the E-rows):

| Route | Result | Why it doesn't help on this box |
|---|---|---|
| Fewer bits after training: 3-bit (E21, E27), 2-bit (E22), 1-1.75 bit (E23) | slower, and quality breaks below 4 bits | no repacked kernels for those types; matmuls are compute-bound (E26) |
| Models trained for 1-bit / ternary (E25) | usable, but none beats Qwen3.5-2B Q4_K | same compute-bound kernels; Bonsai-1.7B is 10 % faster but clearly weaker |
| Q4_0 / IQ4_NL (E18) | +5 % speed for +3 % perplexity / slower | |
| Better Q4_K kernel: rescheduling, software prefetch (E27) | no gain | stock kernel already at 81 % of the RAM bandwidth |
| Thread count, pinning, polling, priority (E12, E33) | 3 threads best | Android's own threads need the 4th core |
| GPU (Mali-G310, Vulkan, E30) | 30 % of CPU speed at best; the 2B doesn't fit | GPU reads 4-bit weights at ≤4.3 GB/s (CPU 6.7); its path needs ~300 MB more RAM; two kernel panics on the way |
| MTP speculative decoding (E31, E32) | slower (3.9–4.5 vs 6.55) even with 48-76 % draft acceptance and a 50 %-cheaper 3-token batch | verifying extra tokens costs almost a full pass on a compute-bound core |
| Middle-ground mix M2 (E24) | −1 % perplexity for −5 % speed | kept as an option, not the default |

**What's left, and why it's not worth it:** the remaining time per token is ~75 % Q4_K matmuls already near the RAM ceiling, ~5 % DeltaNet (now fused), and ~20 % small ops and synchronization (E11, E28). The only bigger levers would be a smaller model or a model trained for this hardware, both outside "the 2B".

## 12. Possible next rungs

- Go **smaller than 260K**: train a toy model with llama2.c's `train.py` (for example dim 16, 2 layers, ~20K params). At that size it stops being a "published LLM" and becomes a model built for the experiment.
- Put **Gemma 3 270M behind the web page** as a chat (via `llama-server`; it would hold ~300 MB while running). Qwen is too heavy to keep resident.
- Make it start on boot: a small Android app could launch the same server from its native library directory (that needs the Android SDK on the Mac).

## 13. The 4B push (2026-11-04)

Your request: "now lets try 4b models, no restriction on model just figure it out and push the limits". Same box, same rules: every run on the TV goes through `tv-guard.sh` (fast 0.2 s sampling, kill below 150 MB available), quality is English perplexity on `eval-en.txt` (ctx 128 × 40 chunks) turned into **bits per byte** (`log2(PPL) × tokens / 22,031 bytes`; Qwen3.5 uses 5,905 tokens, Qwen3/Bonsai 5,756) so different tokenizers compare fairly. Reference: **Qwen3.5-2B all-Q4_K = 0.909 bits/byte, 6.55 tok/s in chat** (§11.8).

### 13.1 The constraint

A 4B model has ~3.5-3.6 B layer weights. The 2B's final config already uses ~850 MB of anonymous RAM (E19), and the 4B runs below show the box tops out at **~0.9-1.0 GB of weights in RAM** (the rest is Android). That is **≤ ~2 bits per weight** for a 4B. Streaming from flash is no way out: flash reads at **192 MB/s** (`dd` of a 939 MB file), so a 2.7 GB Q4 4B would need ~8 s per token.

### 13.2 Experiments

| # | What | Result |
|---|---|---|
| E34 | **New repacked Q2_K kernel for ARM dotprod** (`ggml-cpu/repack-q2k-tv.inc`). Upstream repacks Q2_K only for x86 AVX-512 and RISC-V; on ARM it runs the per-row `vec_dot`, which does a horizontal add (`vaddvq`) and a scalar scale multiply every 16 weights. New layout: 8 rows interleaved 4 bytes at a time, `vdotq_laneq` against the activation, 2-bit fields masked in place (×4/×16 folded into a shift of the int32 sums), scales applied in int16. Gemv + 4-row gemm (prompt batches). `GGML_Q2K_TV=0` = stock | `tools/kbench/kbench_q2k.c` on the TV, 3 threads: **7.8 → 13.9 G weights/s** from DRAM (+78 %), 8.6 → 19.4 cache-resident (2.2×); software prefetch only hurts. Exact: `tools/kbench/q2k_check.cpp` (includes the kernel file itself) matches a double reference to 1.3e-6 for both gemv and gemm; gemm is 1.7× four gemv calls. In llama.cpp (0.8B requantized to pure Q2_K as a test model): **generation 8.47 → 11.97 tok/s, prompt 13.6 → 32.4**; perplexity 247.3 vs 246.4 on this deliberately broken test model (activation-rounding differences, amplified by a flat distribution). Raw: `bench/raw/E34-*` |
| E35 | **Qwen3.5-4B quality at the sizes that matter** (Mac, `tools/mac-4b-frontier.sh`, unsloth Q8_0 + unsloth's 4B imatrix, Q4_K embedding) | **Q8_0: PPL 8.252 = 0.816 bits/byte** (the 4B is ~10 % better than the 2B when unconstrained) · pure **Q2_K** (1,462 MiB file, ~1.12 GiB of layers): **12.351 = 0.972** · pure **IQ2_XXS** (1,222 MiB, ~0.88 GiB of layers, the largest that fits): **19.049 = 1.139**. At the bit width the RAM allows, the 4B is far worse than the 2B at Q4_K (0.909). Raw: `bench/raw/E35-4b-frontier.txt` |
| E36 | **Does the 4B Q2_K fit anyway?** (TV, `patched6` = E34 kernel) | Full 248K tied head: guard kill at RSS 1,166 MB. With the 65K head (`q4b-q2k-h65.gguf`): kill at **RSS 1,045 MB**, after `tv-free-ram.sh`. No. Raw: `bench/raw/tv-logs-4b/E36-*` |
| E37 | **Ternary-Bonsai-4B** (PrismML, Qwen3-4B architecture trained with ternary weights, `Q2_0_g64` file, 1.14 GB, sha256 9d968b04…) | Quality (Mac, full vocab): **PPL 12.902 = 0.964 bits/byte**: the best 4B that can fit, better than Bonsai-4B 1-bit (1.073, E25), still behind the 2B (0.909). Requantized to llama.cpp's ternary types TQ1_0 (1.69 bits, 810 MiB) or TQ2_0 (2.06 bits, 990 MiB): **PPL 13.934 = 0.993** for both (same values: one scale per 256 weights instead of per 64 costs +8 %). Head: id-rank 64K head misses 4.2 % / 37 % of English / multilingual tokens; a frequency-chosen head from the same corpora as H65 (`tools/head_select_tok.py`, llama.cpp's own tokenizer, 61,592 rows) misses **0.95 % / 6.7 %** (the 2B's H65: 1.3 % / 12.6 %). Pruned heads needed a port to the Qwen3 architecture (`src/models/qwen3.cpp`). TV, stock kernels: TQ1_0 + head **2.20 tok/s, prompt 2.70**, RSS 873 MB; TQ2_0 (1.04 GB mapped) thrashes the page cache (prompt 3.19, generation never finished, killed after 20 min). Raw: `bench/raw/E37-*`, `E35-4b-frontier.txt` |
| E38 | **New repacked TQ1_0 / TQ2_0 kernels** (`ggml-cpu/repack-tq-tv.inc`, same 8-row × 4-byte interleave; TQ1_0 keeps ggml's base-3 decode `(x·3^k mod 256)·3 >> 8`; `GGML_TQ_TV=0` = stock) | `tools/kbench/kbench_tq.c`, DRAM, 3 threads: **TQ1_0 9.5 → 14.4 G weights/s** (4T 17.0), **TQ2_0 11.9 → 19.1** (4T 24.0); exact vs a scalar reference. On the TV: Ternary-Bonsai-4B TQ1_0 **2.20 → 3.17 tok/s, prompt 2.70 → 7.91**; TV perplexity 13.394 vs Mac's stock 13.378 (rounding). Raw: `bench/raw/E38-*` |
| E39 | **Lossless ternary in less RAM**: load the original Q2_0_g64 file and repack it at load time into base-3 packing (5 weights/byte) that **keeps the per-64 scales**: 480 bytes per 8 rows × 256 instead of 576 = **1.875 bits/weight**. Needed: a `get_alloc_size` for the repack buffer (5/6 of the file size) and type-aware row offsets in the repack mat-mul (`repack_rows_bytes`). Aborts if a Q2_0 tensor holds the 4th code (+2); `GGML_Q2_0_TERNARY=0` = stock | The file is purely ternary (0 of 1.0 G bytes use +2; within a 256 block the 4 group scales differ by ~7 % median, what TQ1_0 rounds away). Mac A/B on the same file: **PPL 12.3729 (stock Q2_0) vs 12.3711 (new path)** with the 61,592-row head. TV: stock path thrashes (1.18 GB mapped: prompt 2.07, killed); new path **3.22 tok/s, prompt 5.60, peak RSS 996 MB**. Side finding: the 61,588-row heads of E37/E38 weren't repacked at all (rows not a multiple of 8); padded to 61,592 rows, TQ1_0 goes **3.17 → 3.51 tok/s** (prompt 7.82, RSS 945 MB). Raw: `bench/raw/E39-*` |
| E40 | **Threads and op profile** of the lossless 4B | 2 threads 2.54, **3 threads 3.20**, 4 threads 2.43 tok/s (Android needs the 4th core, as for the 2B). Profile per token: **MUL_MAT 90 %** (~15 G weights/s, what the kernel bench predicts), embedding lookup (GET_ROWS on the mapped Q2_0 table) **4.5 % = 14 ms/token** (page faults), flash attention 1.5 %. Raw: `bench/raw/E40-*` |
| E41 | **Embedding as a lazy, randomly-accessed mapping** (Qwen3/Qwen3.5 mark `token_embd` `TENSOR_READ_LAZY` when the head is pruned; active with `-lzm on`) | GET_ROWS 14.2 → 3.0 ms/token, but generation 3.11-3.13 vs 3.14-3.22 tok/s in alternating runs: **no gain**, left off (the default `auto` keeps stock behaviour). Raw: `bench/raw/E41-lazy-embd.txt` |
| E42 | **Unrolled 4-row kernel for the lossless format** (one 64-group at a time, its 16 (chunk, trit) pairs written out instead of a table) | Prompt **5.60 → 7.20 tok/s**, generation unchanged (3.21); Mac perplexity identical (12.3711). Raw: `bench/raw/E42-q20-gemm.txt` |
| E43 | **Real chat through llama-server** (`tv-llm-ui/start.sh`, `MODEL=tb4-q20-hml.gguf BIN=patched11`, `bench.py`, 4 prompts, greedy) | **3.07 tok/s generation, 7.2 tok/s prompt**, server RSS 963 MB, 193 MB left. Answers: Canberra correct (reason vague: "central… neutral"), TV-remote explanation correct, correct documented `is_prime`, Mars facts fluent but two errors (Olympus Mons "11× Everest", it's ~2.5×; overstates evidence of microbial life). Multilingual perplexity 14.35 (6.7 % misses). Raw: `bench/results.jsonl` (label `E43-tb4-q20-chat`) |

### 13.3 Where the 4B ends up

| Model on this TV | Bits/weight in RAM | Quality (bits/byte, lower is better) | Generation (tok/s) | Prompt (tok/s) | Peak RSS |
|---|---|---|---|---|---|
| **Qwen3.5-2B all-Q4_K + H65** (final 2B, §11.8) | 4.5 | **0.909** | **6.55** chat | 17.3 | ~0.99 GB |
| Qwen3.5-4B pure Q2_K | 2.6 | 0.972 | doesn't fit (E36) | | > 1.05 GB |
| Qwen3.5-4B IQ2_XXS | 2.06 | 1.139 | (not run: quality already out) | | |
| Bonsai-4B 1-bit (E25) | 1.125 | 1.073 | 3.17 | 11.4 | 1.09 GB |
| **Ternary-Bonsai-4B, lossless, new kernels** (`tb4-q20-hml.gguf`, `patched11`) | 1.875 | **0.964** (0.948 on the pruned head's positions) | **3.07** chat / 3.21 bench | 7.2 | 1.0 GB |
| Ternary-Bonsai-4B TQ1_0, new kernels (`tb4-TQ1_0-h8.gguf`) | 1.69 | 0.993 | **3.51** bench | 7.8 | 0.95 GB |
| Ternary-Bonsai-4B, stock llama.cpp | 1.69-2.25 | | 2.20 (TQ1_0) or thrashes (Q2_0, TQ2_0) | 2.7 | |

**Verdict.** The best 4B on this box is **Ternary-Bonsai-4B**, made usable by three new kernels and a lossless in-RAM format: from "thrashes" / 2.2 tok/s to **3.1-3.5 tok/s with 2.7-2.9× faster prompts**. It still loses to the 2B: about half the speed and ~6 % more bits per byte. A normal 4B (Qwen3.5-4B, the strongest one) cannot be squeezed into ~0.9 GB without collapsing (IQ2_XXS: 1.139 bits/byte). Other 4B-class models (Phi-4-mini, Gemma 3 4B, SmolLM3, Llama 3.2 3B) face the same ~2-bit wall with weaker starting points, so they weren't downloaded. Bonsai-8B (1.16 GB of 1-bit weights) is over the RAM budget.

**Not tried, with reasons:** speculative decoding with Ternary-Bonsai-1.7B as the draft (both models together need ~1.3 GB, and E31/E32 showed verification costs almost a full pass on this CPU); 4 threads (measured slower); software prefetch (measured slower in both kernel benches).

**Useful beyond the 4B:** the Q2_K and TQ1_0/TQ2_0 kernels speed up any model in those formats on Cortex-A55-class ARM chips (dotprod without i8mm): Q2_K +41 % generation / 2.4× prompt, TQ1_0 +44-60 % generation / 2.9× prompt.

**To use it:** `MODEL=tb4-q20-hml.gguf BIN=patched11 tv-llm-ui/start.sh` (quality), or `MODEL=tb4-TQ1_0-h8.gguf BIN=patched11 tv-llm-ui/start.sh` (+10 % speed, ~3 % worse bits/byte). The default stays the 2B.

## 14. The 9B push: streaming weights and a 1-bit kernel (2026-11-04)

Your request: "let's now try 9b models, and do the best in optimizing streaming and kernels and so on". Same rules (every TV run under `tv-guard.sh`, kill below 180 MB available; quality = bits per byte on `eval-en.txt`).

### 14.1 The constraint, measured

- **RAM:** with nothing running, MemAvailable is ~1.07-1.2 GB, and Android keeps ~350 MB of it free on purpose (its watermarks: the page cache never grew past ~880 MB in any run). One process can hold **~0.96 GB of anonymous memory** before the guard trips (E46). A 9B model is 2-2.8 GB at 2 bits, so **every 9B needs streaming**.
- **Flash** (`tools/kbench/flashbench.c`, E45): O_DIRECT reads 159 MB/s at 128 KB × 1 thread, **278 MB/s at 8 MB × 4 threads**; buffered reads 235-251 MB/s; the kernel's own page-fault path ~150-200 MB/s. So a token that must stream X MB costs at least X / 278 MB/s.

| Candidate (all 8-9 B) | Weights in our format | Must stream per token at ~0.9 GB resident | Quality (Mac, bits/byte) |
|---|---|---|---|
| Bonsai-8B (1-bit Q1_0, Qwen3-8B arch) | 1.01 GB | ~50-120 MB | 0.991 |
| Ternary-Bonsai-8B (Q2_0_g64, repacked 1.875 bits) | 1.69 GB | ~0.8 GB | 0.919 |
| Qwen3.5-9B (unsloth UD-IQ2_XXS, 65K head + MTP block) | 2.36 GB of mat-mul weights | ~1.4-1.7 GB | **0.885** |

### 14.2 Experiments

| # | What | Result |
|---|---|---|
| E44 | **New 1-bit kernel** (`ggml-cpu/repack-q1-tv.inc`): Q1_0 (sign bits + fp16 scale per 128) repacked into **bit planes**, 8 rows × 4 weights per 16-byte vector; `vtstq_u8` against `1<<k` turns plane k straight into a 0/-1 mask for `vdotq_laneq`, and the dot is `d·(−2T − S)` with S from the activation bsums. Two NEON ops per 16 multiply-adds, no table lookups (upstream expands every byte through a 256-entry table). Same bytes as the plain blocks. Gemv + 4-row gemm vs Q8_K; 4 accumulator chains. `GGML_Q1_TV=0` = upstream | `tools/kbench/q1_check.cpp` (includes the kernel file): exact to 1.2e-7 vs a double reference for gemv and gemm. TV, DRAM-sized matrix: **9.3 / 17.4 / 24.5 / 27.0 G weights/s at 1/2/3/4 threads** (2 chains: 20.8 at 3 threads). llama.cpp, Bonsai-4B: **generation 3.02 → 4.22 tok/s (+40 %), prompt 9.4 → 11.0**; perplexity 10.0139 vs upstream 10.0162 (6 chunks). Raw: `bench/raw/E44-*`, `tv-logs-9b/` |
| E45 | **Bonsai-8B quality, and does it fit?** (prism-ml/Bonsai-8B-gguf, 1.16 GB) | Mac: **PPL 13.877 = 0.991 bits/byte** (multilingual 19.63); with the 61,592-row head (cut from its untied `output.weight`, new `gguf_build.py` path): 13.25 / 15.58 on the kept positions (0.95 % / 6.7 % misses). TV: repacked weights need 1,012 MB of anonymous RAM; the guard stopped the load at 949 MB (MemAvailable 177 MB). **About 60-120 MB too big.** Raw: `bench/raw/E44-mac-ppl-b8.txt` |
| E46 | **Repack cache on flash** (`GGML_REPACK_CACHE=<prefix>` in `repack.cpp`): repacked weights go to a `MAP_SHARED` file instead of anonymous RAM; the first run writes it (tensor by tensor, written back and released), later runs map it read-only and skip the repack. Load in seconds; file-backed pages can be dropped under pressure instead of the process being killed | Works (rebuilt cache is byte-identical, `cmp`). But **0.11 tok/s**: with the weights as page cache, the cache holds only ~750 MB of the 1,012 MB working set (the ~350 MB Android keeps free counts against it), so every token re-reads it all. Plain page cache is the wrong tool; it only stays as the cache format. Raw: `bench/raw/tv-logs-9b/b8-cache*` |
| E47 | **Hybrid resident + streamed, v1** (`ggml-cpu/stream-tv.inc`, `GGML_REPACK_RESIDENT_MB`): the first N MB of the cache are copied into anonymous RAM, the rest is a *stream zone*. A hook at every MUL_MAT records the order zone tensors are used (the trace); a prefetch thread reads ahead in trace order into the page cache; the previous zone tensor is dropped after use (`MADV_DONTNEED` + `POSIX_FADV_DONTNEED`) so the stream never pushes anything else out | Bonsai-8B, resident MiB / window → tok/s: 800/128 0.94 · 850/16 0.99 · 850/128 1.39 · 900/128 1.85 · **920/128 2.24** (95 % of zone bytes already cached when used) · 940 and fully resident: guard kill. At 850 only 67 % was cached in time: prefetch can't keep up because flash is the limit. Raw: `bench/raw/E46-b8-stream-sweep.txt` |
| E48 | **Streaming v2: O_DIRECT into anonymous memory.** The zone's address range is remapped to empty anonymous pages; 3 reader threads `pread` 8 MB chunks with `O_DIRECT` straight into each tensor's own addresses (the 278 MB/s path, no page cache, no 4 KB faults), in trace order up to a window ahead; every compute thread waits until its weight is completely loaded, so a wrong prediction only costs time, never correctness; pages freed after use. `GGML_STREAM_MODE=cache` keeps v1's page-cache behaviour | **Bug found and fixed:** first results were garbage (Bonsai-8B PPL 80,946; uniform logits). `GGML_STREAM_VERIFY=1` (compares each tensor with the file when it is used) showed only the 4 KB pages two tensors share were wrong, and only with >1 reader: two concurrent O_DIRECT reads into one page corrupt it. Fix: shared edge pages are read once with a plain `pread` and kept; O_DIRECT covers whole pages only. After the fix: 0 mismatches; streamed Bonsai-8B PPL **8.1719 vs Mac 8.1632** (6 chunks; first 3 identical to the non-streamed TV run). Speed: Bonsai-8B 850/64 **1.52**, 880/32 1.66, 900/16 1.83 tok/s (the window now costs RAM, so less fits than in v1's 920/128 = 2.24); **Ternary-Bonsai-8B 0.21 → 0.31 tok/s** (796 MB streamed per token at ~250 MB/s; 4 readers 0.30; prompt 3.4). Raw: `bench/raw/E48-*` |
| E49 | **Ternary-Bonsai-8B** (prism-ml, Q2_0_g64, 2.31 GB) with the lossless 1.875-bit repack (E39) + 61,592 head | Mac: **PPL 11.457 = 0.919 bits/byte** (multilingual 13.92): almost the 2B's 0.909. Repack cache 1.69 GB. TV: **0.31 tok/s** generation, **3.45 tok/s** prompt (one streaming pass serves the whole batch). Raw: `bench/raw/E47-*`, `E48-tb8-sweep-fixed.txt` |
| E50 | **Qwen3.5-9B** (unsloth UD-IQ2_XXS: IQ2_XXS / IQ3_XXS / IQ2_S / Q4_K layers, Q5_K head, 3.19 GB), rebuilt for the TV in one pipe (`gguf_build.py`: 65,668-row head cut from the untied Q5_K head = 185 MB, + the model's own MTP block fetched from the BF16 safetensors and quantized to Q4_0 by `fetch_mtp.py --q4` = 137 MB, `block_count` 33, `nextn_predict_layers` 1): 2.81 GB | Mac: **PPL 9.859 = 0.885 bits/byte** (multilingual 11.32): **the best quality of anything tried on this box**. Greedy answer (same on the Mac): "The capital of Australia is **Sydney**": better perplexity, but a 2-bit 9B still gets facts wrong. TV, stock mmap: 0.08 tok/s. Streaming the mapped (non-repacked) weights needed a loader path (`LLAMA_STREAM_RESIDENT_MB`, `LLAMA_STREAM_PIN`: head + MTP pinned first) and a second bug fix: zones must stop at any tensor that is not a quantized 2-D weight (norms and conv kernels between the matrices were left as empty pages; found when the server answered "/****!**??"). After the fix: fluent, 0 `GGML_STREAM_VERIFY` mismatches. **Real chat (llama-server, `bench.py`, 4 prompts × 24 tokens, 600 MiB resident + head/MTP pinned, ~1.6 GB streamed per token): 0.16 tok/s, prompt 0.8 tok/s**, RSS 762 MB; answers correct in chat (Canberra, IR remote, `is_prime`, Mars). (A pre-fix llama-bench run at 800 MiB resident gave 0.18 tok/s / prompt 0.82: same I/O, wrong maths.) The server needed `-ctxcp 0 -cram 0`: by default it keeps up to 32 copies of the recurrent DeltaNet state (50 MB each for the 9B) and an 8 GB RAM prompt cache, which the guard killed at ~1 GB RSS. Raw: `bench/raw/E49-*`, `E50-q9-stream-mtp*.txt`, `bench/results.jsonl` |
| E51 | **MTP speculative decoding on the streamed 9B** (`--spec-type draft-mtp`, the model's own next-token block, pinned in RAM). On the 2B (E31) it lost because extra tokens cost almost a full compute pass; here a pass is dominated by flash reads, so verifying 2 tokens costs about the same as 1 | **n_max=1: 0.16 → 0.27 tok/s (+69 %), draft acceptance 91 %**, RSS 822 MB. n_max=2: guard kill at RSS 966 MB (one more DeltaNet snapshot per draft token); not retried with less resident weight at your stop. Raw: `bench/raw/E50-q9-stream-mtp.txt` |
| E52 | **Compact 1-bit scales: Bonsai-8B in 0.96 GB.** Bonsai's per-128 scales are all positive and within 0.44-1.0× of their row's maximum (range reads of 5 tensors from Hugging Face, `q1scales.py` in the session scratchpad), so each is stored as 1 byte: `d = dmin_row + code × dstep_row`, two fp16 row parameters per row in a 32-byte header per 8 rows. 17 bytes per 128 weights instead of 18 (`repack-q1-tv.inc`, now the default for Q1_0) | `q1_check`: exact (3e-7) vs a reference using the stored scales; scales vs fp16 originals **mean 0.07 %, max 0.38 %** (fp16's own rounding is ~0.05 %). Repack cache **1,012 → 962 MB**. Fully resident: prompt 5.06 tok/s, then guard kill at 176-179 MB available (1-4 MB short, also after `tv-free-ram.sh`). Near-full: **920 MB resident + 42 MB streamed: 2.21 tok/s** (direct mode) · 944 MB + 18 MB in page-cache mode: **2.23 tok/s** · 920 MB cache mode: 1.95. Raw: `bench/raw/E51-q1-compact-check.txt`, `E52-b8-compact.txt` |
| E53 | **Bonsai-8B in real chat** (llama-server, `bench.py`, 4 prompts × 64 tokens, KV cache q8_0 + flash attention, `-cram 0`, ctx 1024; the f16 KV cache alone would be 151 MB) | **839 MB resident (123 MB streamed, page-cache mode): 1.21 tok/s generation, 4.9 tok/s prompt**, RSS 900 MB. 912 MB resident (cache mode) and 922 MB (direct): guard kills at ~1.14 GB RSS, so ~840-880 MB is the chat ceiling. Answers: Canberra correct ("chosen to be the seat of government"), correct TV-remote explanation, correct documented `is_prime`, Mars facts fluent. Raw: `bench/raw/E53-b8-chat.txt`, `bench/results.jsonl` |

### 14.3 Where the 9B ends up

| Model on this TV | In RAM / streamed per token | Quality (bits/byte) | Generation (tok/s) | Prompt (tok/s) |
|---|---|---|---|---|
| **Qwen3.5-2B all-Q4_K + H65** (default, §11.8) | 0.99 GB / 0 | **0.909** | **6.55** chat | 17.3 |
| Ternary-Bonsai-4B, lossless (§13) | 1.0 GB / 0 | 0.964 | 3.07 chat | 7.2 |
| **Bonsai-8B 1-bit, compact scales** (`b8-q1-hml.gguf`, cache `b8c`) | 0.84 GB / 0.12 GB | 0.991 | **1.21 chat**, 2.2 bench | 4.9 |
| Ternary-Bonsai-8B (`tb8-q20-hml.gguf`, cache `tb8`) | 0.89 GB / 0.80 GB | 0.919 | 0.31 bench | 3.45 |
| **Qwen3.5-9B UD-IQ2_XXS + MTP** (`q9-iq2-h65-mtp.gguf`) | 0.63 GB / ~1.6 GB | **0.885** | 0.16 chat, **0.27 with MTP** | 0.8 |
| Qwen3.5-9B, stock llama.cpp (plain mmap) | page cache / everything | 0.885 | 0.08 | 0.65 |

**Verdict.** Your 1-2 tok/s target is reached only by **Bonsai-8B** (8.2 B parameters, trained 1-bit): **1.21 tok/s in real chat, 2.2 tok/s in llama-bench**, thanks to the new bit-plane kernel (+40 % on its 4B sibling), compact 8-bit scales (−50 MB) and the streaming engine for the last ~0.1 GB. It is the weakest of the three in quality (0.991 bits/byte, below the 2B's 0.909). The genuinely better 9B, **Qwen3.5-9B** (0.885, the best quality ever measured on this box), is flash-bound: ~1.6 GB of its 2.4 GB of weights must be read for every token at ≤ 278 MB/s, so even with streaming (2×) and MTP speculation (+69 %) it stays at **0.27 tok/s**. Reaching 1 tok/s with it would need ~1.2 bits per weight, where post-training quantization breaks normal models (E23, E35). **For everyday use the 2B stays the best trade-off.**

**What's reusable beyond these models:**
- **Q1_0 bit-plane kernel** (any Bonsai 1-bit model on dotprod-only ARM).
- **Repack cache:** `GGML_REPACK_CACHE` loads in seconds, and `GGML_REPACK_RESIDENT_MB` sets how much stays in RAM.
- **Streaming engine:** trace-ordered prefetch, parallel O_DIRECT reads into the weights' own addresses, drop-behind, and per-weight completion waits. It's enabled with `GGML_REPACK_RESIDENT_MB` for repacked weights or `LLAMA_STREAM_RESIDENT_MB` / `LLAMA_STREAM_PIN` for plain mapped ones, and tuned with `GGML_STREAM_MODE`, `GGML_STREAM_WINDOW_MB` and `GGML_STREAM_READERS`.
- **`GGML_STREAM_VERIFY`:** compares each streamed weight with the file, the check that found both bugs.

**Not tried, with reasons:**
- **MTP with 2+ drafted tokens at a lower resident size.** Stopped at your request.
- **A draft model for Ternary-Bonsai-8B.** A 1.7B draft costs ~200 MB of RAM, which would add ~0.8 s of streaming to every pass.
- **Lower-bit requantization of Qwen3.5-9B.** The Mac's 4 GB of free disk can't hold the BF16 source, and IQ1 quality collapses (E23).
- **4 threads.** Slower on this box (E40).

**To use them** (`tv-llm-ui/start.sh`, new `SENV=` passes the streaming settings):
- Bonsai-8B, about 1.2 tok/s:
  `MODEL=b8-q1-hml.gguf BIN=patched16 EXTRA="-ctk q8_0 -ctv q8_0 -fa on -cram 0" SENV="GGML_REPACK_CACHE=/data/local/tmp/llm/cache/b8c GGML_REPACK_RESIDENT_MB=800 GGML_STREAM_MODE=cache GGML_STREAM_WINDOW_MB=64" tv-llm-ui/start.sh`
- Qwen3.5-9B with MTP, about 0.27 tok/s:
  `MODEL=q9-iq2-h65-mtp.gguf BIN=patched16 EXTRA="-nr -ctxcp 0 -cram 0 -ub 16 -b 64 --spec-type draft-mtp --spec-draft-n-max 1" SENV="LLAMA_STREAM_RESIDENT_MB=600 LLAMA_STREAM_PIN=output.,blk.32. GGML_STREAM_WINDOW_MB=64" tv-llm-ui/start.sh`
- The default stays the 2B.

## 15. The 0.8B speed push (2026-11-04 → 05)

Your request: "go back to 0.8b models, push to the max speed possible, research the latest and try new things, be creative and don't limit yourself to Qwen". Same rules as before: every TV run goes through `tv-guard.sh`. Speed is `llama-bench` tg64 with 3 threads on frequent tokens (`LLAMA_BENCH_COMMON_TOKENS=4000`, as in E15) plus pp128. Quality is English perplexity on `eval-en.txt` (ctx 128 × 40 chunks), turned into bits per byte with each model's own token count. **Starting point:** Qwen3.5-0.8B Q4_K_M (unsloth), stock head, `patched16`: **7.47 tok/s**, prompt 28.3 (`bench/raw/E55-tv-speed.txt`).

### 15.1 Research: what's new at ~0.3-2B (Nov 2026)

Found on Hugging Face (`api/models`, model cards):
- **Liquid AI:** LFM2.5 (230M / 350M / 1.2B, with "QAD" Q4_0 files distilled for Q4_0), LFM2-700M, LFM3-1.7B (Oct 2026, "tri-hybrid"), and the d1 decision models (not generative, skipped).
- **PrismML:** **Ternary-Bonsai-2-2B** (Sep 2026, Qwen3.5-2B made ternary with a folded Hadamard rotation, 0.52-0.55 GB).
- **Others:** Gemma 3 1B, Llama 3.2 1B, Qwen3-0.6B, Granite 4.0 H 1B (hybrid Mamba-2), Falcon-H1-Tiny (90M-0.6B), Falcon-E-1B (BitNet; only an `i2_s` bitnet.cpp file, skipped), MobileLLM-R1 (reasoning-only, skipped).

Our llama.cpp v0.8.0 already supports every architecture above, including `lfm3`.

### 15.2 Experiments

| # | What | Result |
|---|---|---|
| E54 | **Screening of other small models** (`tools/screen-small.sh`: each GGUF streams from Hugging Face straight to the TV, runs speed + perplexity + one greedy answer, then is deleted; stock files, stock full heads, `patched16`) | Generation / prompt tok/s, bits/byte, max RSS: **Qwen3.5-0.8B Q4_0** 8.15 / 31.8, **1.024**, 750 MB · **LFM2.5-350M** QAD-Q4_0 **23.7** / 104, 1.805, 298 MB (answer correct; perplexity climbs on the later code-heavy chunks) · **LFM2-700M** Q4_0 11.5 / 52.8, 1.169, 523 MB · **LFM2.5-1.2B** QAD-Q4_0 8.26 / 31.6, 1.133, 783 MB (answer correct) · **Qwen3-0.6B** Q8_0 9.35 / 50.6, 1.059, 798 MB · **Gemma 3 1B** Q4_K_M 5.66 / 11.1, 1.260, 1,038 MB · **Llama 3.2 1B** Q4_0 7.04 / 29.5, 0.998, 979 MB · **Ternary-Bonsai-2-2B** Q2_0 5.88 / 15.8, 1.124, 718 MB · **Granite 4.0 H 1B** Q4_0 (~1.5 B parameters, hybrid Mamba-2): **0.916 bits/byte**, nearly the 2B's 0.909, but the guard stopped its speed run (RSS 925 MB) and its answer (killed while loading); only the perplexity run, with a lower kill threshold, finished · **LFM3-1.7B** QAD-Q4_0: doesn't fit (every run killed at 1.0-1.08 GB RSS). An adb drop stopped the first attempt at Gemma; it was rerun. Summary: `bench/raw/E54-screen/summary.txt` |
| E55 | **Qwen3.5-0.8B rebuilt with the 2B recipe** (Mac): unsloth BF16 → `gguf_build.py` 65,668-row head (`data/head-ids-ml60k.txt`, same tokenizer as the 2B) → `llama-quantize --pure` with unsloth's 0.8B imatrix, one type everywhere | English PPL (Mac, head positions, 1.31 % misses): BF16 12.358 · Q8_0 12.368 · Q6_K 12.454 · Q5_K 12.626 · **Q4_K 13.241 (+7.1 %)** · Q4_0 14.168 (+14.6 %). TV gen / prompt tok/s: **Q4_K 12.14 / 33.1** (RSS 566 MB) · Q4_0 12.50 / 35.6 · Q5_K 9.21 / 26.1 · Q6_K 6.78 / 20.8 · Q8_0 8.23 / 29.4 (RSS 925 MB). **7.47 → 12.14 tok/s (+63 %)**: the stock file spent 43 % of each token on Q6_K matmuls, mostly the 248K-row head. Small models lose more to 4 bits than the 2B did (+7 % vs +2.3 %). Q4_K stays the format: Q4_0 is 3 % faster for twice the loss. Raw: `bench/raw/E55-*` |
| E56 | **DeltaNet reads its state from the cache by id** (patch, `patched17`): stock llama.cpp gathers each layer's 1 MB recurrent state into a temporary with GET_ROWS before the GDN op, every token. Now `build_rs` returns a view of the cache, and the GDN op gets the row ids (`src[7]`) and the cache (`src[8]`), reading row `s_copy[seq]` at compute time and updating it in place (row j of the state only needs its own row). `LLAMA_GDN_GATHER=1` = stock | **11.94 → 12.88 tok/s (+8 %)**; stock gather on the same binary 11.86. Perplexity and greedy text identical on the Mac (13.2409). Raw: `bench/raw/E56-byid.txt` |
| E57 | **Where the remaining time goes**: a node-by-node trace (`GGML_OPPROF=2`: per-thread compute time and start time of every node of one graph, `tools/optrace.py`), `LLAMA_UBATCH_PROF=1` (graph build / inputs / compute), `GGML_CPU_PROF=1` (plan vs node loop), and a barrier microbenchmark (`tools/kbench/barrier.c`) | Per token, ~73 ms: **Q4_K matmuls ~71 %** (52 ms; FFN and QKV at ~7 GB/s, the 65K head 7.7 ms at 4.9 GB/s, `z` 4 GB/s because it runs right after DeltaNet's 1 MB state write-back), DeltaNet 7 %, ~830 small nodes 12 %. Overheads are small: graph build 0.25-0.33 ms (the graph is reused), inputs 0.02 ms, plan 0.39 ms, barrier 0.25 µs idle (2.5 µs under load), start skew + gaps between nodes 1.8 ms. Two fixes: the head scatter's `FILL` of 248,320 logits ran one 1-element row at a time (1.2 ms), now one flat range per thread; for a single output row the scatter skips both transposes. **12.88 → 13.2-13.6 tok/s.** Run-to-run noise on this box is ±0.2-0.8 tok/s. Raw: `bench/raw/E56-trace17.txt`, `E57-*` |
| E58 | **Activation sparsity, simulated** (TEAL / CATS style, training-free: `LLAMA_SPARSE_SIM=a,d,g` zeroes the smallest-magnitude fraction `a` of every layer mat-mul input, `d` for `ffn_down`, in groups of `g` dims). Skipping the matching weight columns would cut bytes read | Q8_0 + H65, Mac PPL 12.368 → 25 %: 12.760 (+3.2 %) · 40 %: 14.772 (+19 %) · 50 %: 19.10 · 30 % / 60 % down: 14.06. **Grouped skipping, which a fast int8 kernel needs, collapses:** groups of 4 at 40 %: 44.4; groups of 16: 230. ✗ Not worth a kernel: at most ~15 % fewer bytes for +3 % perplexity. Raw: `bench/raw/E58-sparse-sim.txt` |
| E59 | **fp16 DeltaNet state, simulated** (`GGML_GDN_F16SIM=1`: the fused kernel rounds the stored state to fp16) | PPL ctx 128: 12.7040 → 12.7051; ctx 512: 12.7887 → 12.8018 (+0.1 %). Raw: `bench/raw/E59-gdn-f16-sim.txt` |
| E60 | **Approximate LM head** (`tools/head_approx.py`, hidden states dumped with `LLAMA_DUMP_HIDDEN`): score all tokens with a rank-r SVD of the head, re-score the top K exactly | The exact head reproduces the perplexity (12.707), so the setup is right. **Low rank can't find the token:** rank 128 / K 64 has the true top-1 in its candidates 52 % of the time (PPL 722); rank 384 / K 256, 83 % (PPL 71). ✗ The head is genuinely high-rank; the pruned head stays. Raw: `bench/raw/E60-head-lowrank-h65.txt` |
| E61 | **fp16 DeltaNet state, real** (`LLAMA_STATE_F16=1`, `patched18`): fp16 recurrent cache, a NEON fp16 path in the fused GDN kernel (read by id, convert in registers, write back in place), F16 `scale` for the state clear | Mac PPL ctx 512 12.7887 → 12.8058 (+0.13 %); greedy text stays fluent but diverges after ~2 sentences. TV: GDN 314 → 262 µs/call (−17 %), but a whole token only −0.7 % (72.7 → 72.2 ms over 65 graphs). Plain tg64 runs can't separate it from noise (f32 12.64 / 13.18, f16 13.41 / 12.92). Kept as an option, off by default. Raw: `bench/raw/E61-state-f16.txt` |
| E62 | **Granite 4.0 H 1B with a pruned head** (2026-11-05; IBM hybrid: 36 Mamba-2 + 4 attention layers, 1.46 B parameters, tied 100,352-token embedding). Pruned-head port to `src/models/granite-hybrid.cpp` (load `output_ids`, lazy `token_embd`, scatter after the logit scale; `patched19`). Head: `tools/head_select_tok.py` with Granite's tokenizer, specials from 100256; 48K rows (`data/head-ids-granite-48k.txt`) miss 1.15 % of held-out English + code and 0.87 % of multilingual tokens (32K rows: 5.0 % / 1.8 %, rejected). Quantized on the Mac from IBM's Q8_0 (the 2.9 GB BF16 doesn't fit on the Mac's free disk) with unsloth's Granite imatrix: `tools/rebuild-models.sh g1` | **Quality (Mac, full vocab, bits/byte):** Q8_0 0.906 · all-Q4_K 0.947 (+11.8 % PPL) · all-Q4_K + Q6_K embedding 0.925 · **all-Q4_K + Q8_0 embedding 0.919 (+3.5 %)**, kept: the tied table is the sensitive part, and unsloth's imatrix has no data for it. With the 48K head: PPL 11.51 on 98.65 % of positions; greedy answer correct. **TV (`g1-q4k-e8-h48.gguf`, 986 MB, SHA-256 `6bc4e656…fca0ac4`): 5.29 tok/s** (tg64, `-ub 32`), max RSS 950 MB. It only fits after `tools/tv-free-ram.sh`; with the default ubatch (49 MB compute buffer) or without freeing RAM the guard stops it at 792-913 MB RSS. Per token (205 ms): Q4_K matmuls 121 ms at 6.1 GB/s (≈735 MB of weights, about 2× the 0.8B's), SSM_SCAN 22 ms (2.5 GB/s), state CPY 20 ms, Q8_0 head 12 ms, small ops ~25 ms. **Verdict: no gain.** The Qwen3.5-2B is better on both axes (0.909 bits/byte, 6.55 tok/s in chat, more memory headroom). An in-place Mamba-2 state write (as E56 did for DeltaNet) would save at most ~20 ms (→ ~5.8 tok/s); not pursued. Raw: `bench/raw/E62-granite-*.txt` |

### 15.3 Where the 0.8B stands (so far)

| Qwen3.5-0.8B on this TV | Generation (tok/s) | Prompt (tok/s) | English PPL |
|---|---|---|---|
| Stock unsloth Q4_K_M, full head | 7.47 | 28.3 | not measured* |
| All-Q4_K + 65K head (E55), `patched16` | 12.14 | 33.1 | 13.24** |
| + state read by id (E56), `patched17` | 12.88 | | 13.24** |
| **+ flat FILL, transpose-free scatter (E57)** | **13.2-13.6** | ~33 | 13.24** |
| + fp16 state (E61, optional) | +0.7 % | | 13.26** |

\* the stock unsloth Q4_0 file (full vocabulary, TV) has PPL 14.14 = 1.024 bits/byte (E54); \*\* 65K-head positions (1.31 % misses), Mac; not comparable with full-vocabulary numbers.

**What limits it now:** about 71 % of each token is Q4_K matmuls, already at ~7 GB/s on the big matrices (the kernel's DRAM ceiling measured in E27 is 6.7-8.3 GB/s). The remaining small ops and sync are ~15 % together. A big step past ~14 tok/s needs fewer weight bytes per token. E54 found no model in this size class that does that without losing quality: the only faster one, LFM2.5-350M (23.7 tok/s), is much weaker (1.805 bits/byte), and Ternary-Bonsai-2-2B runs at 5.9 tok/s with stock kernels. The interesting find was **Granite 4.0 H 1B: 0.916 bits/byte**, close to the 2B's 0.909. E62 built it with a pruned head and all-Q4_K: it barely fits and runs at 5.29 tok/s, slower than the 2B at slightly lower quality, because it has about twice the 0.8B's weight bytes per token.

**Not done yet:** a real-chat run through llama-server (sampling over the full 248K vocabulary is not in the llama-bench numbers).

**adb from the Mac:** on 2026-11-05 `adb connect` began failing with "No route to host" while `ping` and `nc -z <tv> 5555` worked: macOS Local Network privacy refusing the adb daemon (built-in tools are exempt). Workaround used for E62, no settings changed: a localhost relay through `nc` (`mkfifo f; nc -l 127.0.0.1 15555 < f | nc <tv> 5555 > f &`), then `adb connect 127.0.0.1:15555` and `ADB_SERIAL=127.0.0.1:15555`. The proper fix is allowing the terminal app under System Settings → Privacy & Security → Local Network.

**New environment switches** (all off by default; the patch is `patches/llama.cpp-v0.8.0-tvbox.patch`):
- `LLAMA_GDN_GATHER=1`: stock state gather instead of the by-id read (E56).
- `LLAMA_STATE_F16=1`: fp16 recurrent state, Qwen3.5 only, no MTP (E61).
- `GGML_OPPROF=2` (+ `GGML_OPPROF_GRAPH=n`): node-by-node trace of one graph; `LLAMA_UBATCH_PROF=1`, `GGML_CPU_PROF=1`: phase timings (E57).
- `LLAMA_SPARSE_SIM=a,d,g`, `GGML_GDN_F16SIM=1`, `LLAMA_DUMP_HIDDEN=<file>`: quality simulations (E58-E60).

**To use it:** `tools/rebuild-models.sh 08` builds `q08-q4k-h65.gguf` (SHA-256 `901f3838…8523a6b`) and its padded-head copy `q08-q4k-h65p8.gguf` (§17), then `MODEL=q08-q4k-h65p8.gguf tv-llm-ui/start.sh` (default build `patched25` since §17: 16.8 tok/s). The default model stays the 2B.

## 16. The 2B, round 3: scheduling and the last small ops (2026-11-05)

**Goal (your call):** push the default model, Qwen3.5-2B (`w-q4k-h65.gguf`), to the maximum. Starting point: the UI still ran `patched5` (§11.8). All numbers below are back-to-back A/B runs on the same day (llama-bench tg64/tg128, 3 threads, `LLAMA_BENCH_COMMON_TOKENS=4000`), because the box itself was slower than on 11-04: `patched5` measured 5.79 tok/s in llama-bench today vs 6.69 then (the audio HAL uses ~19 % of a core all the time). Before this, the TV was cleaned up: Granite, the stock 0.8B, Gemma 270M, the four `o-*` screening files and `tb4-TQ1_0-h8` were deleted (TV free flash 0.56 → 4.9 GB); the models in the README table, `w-m2-h65` and the 2B BF16 work file stay.

### 16.1 Experiments

| # | What | Result |
|---|---|---|
| E63 | **Re-baseline and trace.** `patched5` vs `patched19` (everything from §15), op profile, GGML_OPPROF=2 node trace (`tools/optrace.py`, parser fixed for the newer trace columns) | llama-bench 5.79 → **6.25** (+8 %); chat (`tools/mtp-bench.sh`) 5.65 → 5.81 with one outlier answer (server eval time matches the client, so sampling and HTTP cost nothing measurable). Per token 166 ms: Q4_K matmuls 137 ms (82 %, 6.2 GB/s), DeltaNet 5.6, embedding GET_ROWS 4.9 (page faults on the lazily mapped `token_embd`: with a 1-token pool it drops to 0.75 ms/call, but tg64 doesn't move, so ≤ 2 %), small ops ~20. **The trace's finding:** in matmuls the slowest thread sums to 166 ms vs a 138 ms mean, i.e. ~28 ms per token of idle threads waiting for a straggler (thread 1: 157 ms vs ~129 ms for the others). Tried and closed: more, smaller chunks (`GGML_CHUNK_MULT` 16 / 32 per thread: 6.12-6.25 / 5.70 vs 6.37 at the stock 4: short chunks break the prefetcher's sequential streams); pinning to cores 1-3 or 0-2 (`taskset`: 5.7-5.9 vs 6.2 unpinned); a 4th core (`KB_SWEEP=1 kbench`, Q4_K gemv from DRAM, best of 3 runs: 3.7 / 5.7 / 6.6 / 7.3 GB/s for 1-4 threads, so at most ~10 % more bandwidth). Also found: single-row element-wise ops run on thread 0 only (ggml splits them by rows): SWIGLU 97 µs on thread 0 vs 3.6 µs on the others. Raw: `bench/raw/E63-*` |
| E63b | **Single-row ops split by columns** (`patched21`): SWIGLU, SILU, the generic unary ops (SIGMOID, …) and same-shape binary ops (ADD, …) split their columns between threads when there is one row of ≥ 1,024 values | Greedy text identical to `patched20`. llama-bench tg128 ×5, three alternating rounds: **6.26 → 6.39 (+2.0 %)**, 3/3 wins. Raw: `bench/raw/E63-colsplit-ab.txt` |
| E64 | **Guided self-scheduling of matmul rows** (`patched22`, default since `patched23`): for 1-4-row matmuls each grab takes remaining / (2 × threads) row groups (at least 1/48 of the matrix), so the first chunks are long sequential streams and only the last ones are short | Mac PPL bit-identical (19.2111 both ways). 2B: **6.39 → 6.54 (+2.4 %)**, 3/3 wins. Trace: matmul time per thread **124.4 / 123.7 / 124.9 ms** (was 130 / 157 / 128); total imbalance 33.5 → 6.5 ms. 0.8B: +1.4 %. Prompt batches stay on the stock scheduler (pp128 was too noisy to call, ±2). `GGML_CHUNK_GUIDED=0` = stock. Raw: `bench/raw/E64-*` |
| E65 | **fp16 DeltaNet state on the 2B** (`LLAMA_STATE_F16=1`, E61) | **6.43 → 6.59 (+2.5 %)**, 3/3 wins (the 2B moves 18 MB of state per token). English PPL (TV, 65K-head positions): 9.9374 → **9.9523 (+0.15 %)**, same 1.31 % misses. Raw: `bench/raw/E65-*` |
| E66 | **4 threads again**, now with guided scheduling | 5.09 / 5.18 vs 6.53 / 6.52 with 3: still much worse. The straggler problem isn't only inside matmuls: ~850 nodes per token each end in a barrier, and a preempted 4th thread stalls every one of them. Closed. Raw: `bench/raw/E66-threads-guided.txt` |
| E67 | **Real chat, final** (`tools/mtp-bench.sh`, now with `SENV`): `patched5` vs `patched23` + `LLAMA_STATE_F16=1`, two rounds each | **2B chat: 5.83 / 5.96 → 6.57 / 6.53 tok/s (+11 %)**, prompt 15.4 → 16.1; server eval 149-159 ms/token. **0.8B** (tg128, `patched18` vs `patched23` + fp16 state, 3 rounds): **13.45 → 14.20 (+5.6 %)**. Raw: `bench/raw/E67-*` |

### 16.2 Where the 2B stands

| Qwen3.5-2B, same day, back to back | llama-bench tok/s | Chat tok/s |
|---|---|---|
| `patched5` (UI default until now) | 5.79 | 5.90 |
| `patched19` (§15: state by id, flat FILL, transpose-free scatter) | 6.25 | 5.81* |
| + column-split single-row ops (`patched21`) | 6.39 | |
| + guided matmul scheduling (`patched22/23`) | 6.54 | |
| **+ fp16 DeltaNet state (new default)** | **6.59** | **6.55** |

\* one slow answer (5.18) pulled this run down.

**What limits it now:** with the tail gone, each thread streams Q4_K weights at ~6.85 GB/s, the RAM ceiling `kbench` measures for 3 threads; matmuls are 90 % of the critical path, the rest is ~14 ms of small ops. Every format we have runs at ~11-13 G weights/s on these cores (E26), so fewer bits don't help either. What's left is a few % each, and none of them is free: (1) a smaller head (`data/head-ids-ml40k.txt`: ~52 M fewer weights per token, about +3 %, more tokens outside the head: a quality trade); (2) embedding rows taken from the resident head instead of faulting them in from flash (≤ 2 %; needs an inverse of the 8-row repack, since `output.weight` is repacked); (3) fusing the remaining small ops (DeltaNet conv concat + SSM_CONV, norms: ~2-3 ms).

**New environment switches:** `GGML_CHUNK_GUIDED=0` (stock matmul scheduling), `GGML_CHUNK_MULT=n` (stock scheduler, n chunks per thread). `tv-llm-ui/start.sh` now defaults to `BIN=patched23` with `SENV=LLAMA_STATE_F16=1` (an explicit `BIN` gets no default environment; `SENV=` turns fp16 state off).

## 17. The 2B, round 4: the head, the embedding and the conv step (2026-11-05, evening)

**Goal (your call):** go after the three options left at the end of §16: a smaller head, embedding rows from the resident head instead of flash, and fusing the small DeltaNet ops. Same method: back-to-back alternating llama-bench runs (tg128 × 5, 3 threads, `LLAMA_BENCH_COMMON_TOKENS=4000`, `LLAMA_STATE_F16=1`), 3 rounds, plus real chat with `tools/mtp-bench.sh`. The box's absolute speed moved a lot during the evening (the same `patched24` + 2B: 6.64 tok/s at 20:00, 7.31-7.40 at 21:30; cause not identified), so only numbers from the same block of runs are compared.

### 17.1 Experiments

| # | What | Result |
|---|---|---|
| E68 | **45K head** (`w-q4k-h45.gguf`: the 2B re-cut on the TV with `data/head-ids-ml40k.txt`, 45,654 rows = H45 of E17; `tools/rebuild-models.sh 2b45`) | Speed: **6.64 → 6.90 tok/s (+3.9 %)**, 3/3 wins; chat 6.89 vs 6.68-6.79. Quality (TV, `llama-perplexity`, ctx 128, 40 chunks): tokens the head cannot produce **English 1.31 % → 2.22 %, multilingual 12.58 % → 19.72 %** (the PPL on the remaining positions, 9.66 / 7.72, isn't comparable: the smaller head skips more of the hard positions). 2 of the 4 greedy chat answers change. Same picture as E17: **kept as an option, not the default** (`MODEL=w-q4k-h45.gguf tv-llm-ui/start.sh`). Raw: `bench/raw/E68-70-ab.txt`, `E68-69-ppl.txt` |
| E69 | **Embedding rows from the resident head** (`patched24`; `src/llama-graph.cpp`, `LLAMA_EMBD_FROM_HEAD=0` = off). With a tied embedding the head's rows are byte copies of `token_embd` rows, so a custom op reads a token's row from the head (in RAM) when the token is in it, and from the flash-mapped `token_embd` otherwise. Checked once per head: 8 rows compared byte for byte with `token_embd` (an untied head, the 9B's, turns it off). Offline check: all 65,668 head rows of the 0.8B file are byte-identical to their `token_embd` rows | Output bit-identical: TV English PPL **9.9523** with both builds, same 4 chat answers. Speed: **no measurable change in llama-bench** (6.57 → 6.55, noise). Worth noting: on the Mac the stock path itself is slightly non-deterministic (PPL 17.8258 / 17.8319 on two identical runs), the head path gives the stock path's usual value. In real chat `patched24` (E69 + E70) gained +2.9 % vs +1.3 % in llama-bench, so it may help when the page cache is under pressure; I can't separate it from E70 there. Raw: `bench/raw/E68-70-ab.txt` |
| E70 | **Fused one-token conv step** (`patched24`; `src/models/qwen35.cpp`, `LLAMA_CONV_FUSED=0` = off). For one token, the DeltaNet causal conv was 4 graph nodes per layer: state gather, CONCAT (old 3 columns + new one, transposed), CPY (shifted state back to the cache) and SSM_CONV, ~110 µs per layer for a 4-tap filter. Now one custom op reads the 3 old values per channel at row `s_copy[0]`, writes the shifted window to the destination row, and sums the taps in SSM_CONV's order (NEON: `vld3q`/`vld4q` de-interleave 4 channels at a time; SILU stays separate). A scalar first version was *slower* on the Mac (20 vs 16 µs: a 4-deep dependent FMA chain per channel) | Greedy text identical. Mac per call 15.6 → 4.5 µs. TV llama-bench: **6.55 → 6.64 (+1.3 %)**, 3/3 wins. **Real chat (E69 + E70): 6.54 / 6.55 → 6.68 / 6.79 tok/s (+2.9 %)**, same answers. **0.8B: 14.05 → 14.72 (+4.8 %)**, 3/3 (fixed per-node costs weigh more on the smaller model). Raw: `bench/raw/E68-70-ab.txt`, `E70-chat-2b.txt`, `E70-08b.txt` |
| E71 | **The head was never repacked.** The CPU backend only repacks Q4_K matrices whose row count is a multiple of 8; the 65,668-row head (65,668 mod 8 = 4) stayed in the plain mmap buffer and ran on the generic `vec_dot` kernel, not the interleaved 8x4 kernel every other matrix uses (repack buffer 736 MiB vs 808 MiB with the head). Fix: 4 more ids (`data/head-ids-ml60k-p8.txt`, 65,672 rows), re-cut on the TV (`w-q4k-h65p8.gguf`) | `patched24`: **7.40 → 7.60 tok/s (+2.7 %)**, 2 clear wins and one noisy tie. But a repacked head interleaves its rows, which switched E69 off |
| E72 | **Embedding rows from a repacked head** (`patched25`): the inverse of `make_block_q4_Kx8` rebuilds a row's plain Q4_K blocks (d, dmin, the 6-bit scales re-packed, the 4-bit quants de-interleaved) before dequantizing. The interleave (4 bytes on the A55, 8 on chips with i8mm) is found by the check, which rebuilds 8 rows and compares them with `token_embd`. The CPU backend refused the op at first (an op with a source in the repack buffer must be supported by the repack type, which only declares MUL_MAT): custom ops are now exempt, they read raw memory. Also the 0.8B with the padded head (`q08-q4k-h65p8.gguf`) | Mac: greedy text identical, PPL 17.8178 in all four runs (on / off × 2). TV: **2B 7.31 → 7.73 tok/s (+5.8 %)**, 3/3 wins (with the embedding path off: 7.72, so again no measurable effect in llama-bench); **0.8B 16.10 → 16.81 (+4.4 %)**, 3/3. **Real chat: 7.41 → 7.70 tok/s (+3.9 %, median of 4 runs each, 3/4 wins)**, prompt 17.2 → 17.8; one `patched25` run at 7.05 had part of the model evicted (RSS 892 MB vs ~1,020 MB). The 4 greedy answers are identical in all 8 runs. **New default** (`patched25` + `w-q4k-h65p8.gguf`). Raw: `bench/raw/E71-head-pad8-ab.txt`, `E72-*` |

### 17.2 Where the 2B stands

| Qwen3.5-2B (65K head unless noted) | llama-bench tok/s | Chat tok/s |
|---|---|---|
| *block A (20:00)* `patched23` (§16 default) | 6.55 | 6.55 |
| + fused conv step, embedding from head (`patched24`) | 6.64 (+1.3 %) | 6.74 (+2.9 %) |
| 45K head on `patched24` (option, E68) | 6.90 (+3.9 %) | 6.89 |
| *block B (21:30, box faster)* `patched24` | 7.31 | 7.41 |
| **+ head padded to 65,672 rows → repacked, rows rebuilt for the embedding (`patched25`, new default)** | **7.73 (+5.8 %)** | **7.70 (+3.9 %)** |

Chained, the round is worth about **+7 % in llama-bench and in chat over §16's build** (1.013 × 1.058 and 1.029 × 1.039), with bit-identical output. The 0.8B gained **+9.5 %** (E70 +4.8 %, E72 +4.4 %; 16.8 tok/s in block B). What's left is the same as in §16: matmuls stream weights at the RAM ceiling and are ~90 % of a token; the embedding path didn't show up in llama-bench at all.

**Notes on method** (they cost time this round): `tv-guard.sh` writes the command's output to its log file, not stdout, so a pipe after it gets nothing; Android's `mksh` has a built-in alias `r` (history re-run), so a shell function named `r` silently never runs; `ps -A` there shows the process name (`sh`), not the script, so watchers must use `ps -A -o ARGS`; `llama-perplexity` on the 2B needs the guard at `LIMIT_MB=130` (its 127 MB logits buffer takes MemAvailable to ~150-170 MB), the 180 MB default stops it.

**New environment switches:** `LLAMA_EMBD_FROM_HEAD=0` (stock embedding GET_ROWS), `LLAMA_CONV_FUSED=0` (stock conv nodes). `tv-llm-ui/start.sh` now defaults to `BIN=patched25` and `MODEL=w-q4k-h65p8.gguf`.

## 18. The 4B, round 2: arXiv and GitHub, then a kernel that spends RAM on decode time (2026-11-06)

**Goal (your call):** "a round very aggressive and deeper search especially around latest arXiv papers and GitHub to optimize the 4B, then trickle down learnings to the 2B and 0.8B". Same rules: TV runs through `tv-guard.sh` with `am kill-all` first, llama-bench tg64 × 3 with 3 threads and `LLAMA_BENCH_COMMON_TOKENS=4000`, three alternating rounds, then real chat with `tools/mtp-bench.sh`. **Starting point:** Ternary-Bonsai-4B (`tb4-q20-hml.gguf`), 0.964 bits/byte, **3.40 tok/s** in llama-bench and **3.17-3.28** in chat on today's build.

### 18.1 Research

Four scans: post-training compression at 1-3 bits, inference tricks that touch fewer weight bytes, GitHub (llama.cpp upstream, PrismML, Arm, MNN, torchao) and new models. Numbers below are as reported by the papers or PRs, except where marked measured; nothing in this list was reproduced unless an experiment below says so.

| Lead | What it reports | Verdict for this box |
|---|---|---|
| **GSQ** (arXiv 2604.18556), refines an existing GGUF's codes in place | Qwen3-4B Q2_K: 48.3 vs Unsloth's 35.0 (task score; bits not stated); Llama-3.1-8B 2.13 bits 68.6 vs 73.7 full precision | Most promising quality lead (no kernel change), but about 40 GPU-hours for a 4B by the scan's estimate, and Q2_K doesn't fit. **Not run:** this Mac is an 8 GB M1, and renting GPUs is your call |
| ICLR 2026 "QAT for ultra-low-bit reasoning LLMs" (`yasu0001/ReasoningQAT`), Double Binary Factorization (2505.11076), AF1 (2609.06161), GAMMA (2605.18475), D²Quant (2602.02546) | 2-bit results well above GPTQ / AWQ | Same: all need a GPU. DBF would also need a new graph op |
| **R2R** token routing (2505.21600), ECHO (2609.17241) | 11-15 % of tokens go to the large model; 2.4-2.9× on GPU | Both models must fit in RAM at once; here they don't. Not tried |
| Olica FFN pruning (2506.08436), Prox channel sparsity (2607.27591), HNSW head (2608.27460) | Olica: PPL 12.63 → 15.35 at 20 % pruned; Prox needs a low-bit proxy copy | Costs more quality than the bytes are worth here; the head idea became E74 |
| llama.cpp PRs: #27478 / #22022 huge pages, #30715 JQ3_K / JQ4_K, #15550 `--target-bpw`, #22836 Sherry 1.31-bit | JQ4_K cuts KLD 15-20 % at Q4_K's size; #15550 allocates bits per tensor | Huge pages tested (E73). **JQ4_K is the best open lead for the 2B and 0.8B's quality** (needs a new 8×4 kernel, speed on the A55 unknown). Sherry has no 4B model |
| New models | Ternary-Bonsai-2-4B, Falcon-E-3B ternary, LFM3-1.7B, Granite-SWASH 3B (mixture of experts, 20 % of weights active per token) | Measured in E75 |

Dead ends the scans flagged, so they aren't retried: LMTLS5's ternary Qwen3.5-4B (+0.8 % perplexity, but 6.4 GB, it decomposes each matrix at 3.5× rank: no compression), QTIP / Proteus / CurveTQ (per-weight decode, too slow), EAGLE-3 and other speculation (verification is still a full pass, as E31 found), layer skipping (no CPU results, no results on DeltaNet state).

### 18.2 Experiments

| # | What | Result |
|---|---|---|
| E73 | **Transparent huge pages** for the resident weights (`madvise(MADV_HUGEPAGE)` on 2 MiB aligned buffers; never tried here, and the kernel allows it). `tools/kbench/thpcheck.c` | **✗ Unobtainable.** With 1.07 GB free: **0 of 128** huge pages, 8 of 128 on a second run. In a real 2B load: 2 MiB of ~900 MB, compaction failing 839 of 842 times, and without root it can't be forced. The patch was removed. Raw: `bench/raw/E73-thp.txt` |
| E74 | **1-bit sketch of the LM head** (`tools/head_sketch.py`): score all rows with sign(W) × row norm, re-score the top K exactly | Simulated on the 0.8B: K = 256 has the exact top-1 **99.92 %** of the time and 97.6 % of the top 10, reading 8.6 MB instead of 37.8 (the low-rank attempt of E60 reached 83 %). **Parked:** with the 1-bit kernel's measured 24.5 G weights/s it saves about 2 ms per token at most (~3.7 % on the 0.8B, 1.7 % on the 2B, under 1 % on the 4B), and it changes which tokens can be sampled. Raw: `bench/raw/E74-head-sketch.txt` |
| E75 | **New models**, quality on the Mac (English, full vocabulary, bits/byte) | **Ternary-Bonsai-2-4B Q2_0 0.987** (PPL 12.84; SHA-256 `64a5183b…33d4`), Falcon-E-3B DPO TQ2_0 **1.394** (PPL 19.87, 7,121 tokens), LFM3-1.7B QAD-Q4_0 **1.258** (PPL 27.06), Granite-SWASH 3B Q4_K 1.119 (1.7 GB) / Q2_K 1.878 (no imatrix; its GGUF needed a small loader fix for split shared-expert tensors, and it's a base model with no chat template). Only the first is competitive, and it's slightly below Ternary-Bonsai-4B's 0.964. Pruned head ported to tied Hadamard models (`src/llama-model.cpp`): English PPL 12.07 with the same 1.31 % misses as the 2B's head, multilingual 11.88 (12.58 % misses). TV: 3.25-3.32 tok/s vs Ternary-Bonsai-4B's 3.29-3.49 in the same block. Raw: `bench/raw/E75-*` |
| E76 | **Where the ternary kernel's time goes, and two new in-RAM layouts** (`ggml-cpu/repack-q20c-tv.inc`). Kernel benches show the base-3 kernel is **compute-bound**, unlike Q4_K: TQ1_0 does 23 G weights/s cache-resident and 13.5 from DRAM, and its 3.1 weights/cycle is what ~5 NEON ops per 16 weights predicts on an in-order core that issues about one Q-form op per cycle (mul, ushr, uhadd, ushr, sdot). Five decode variants (`tools/kbench/kbench_t3.c`) all tie, since each still takes ~5 ops. A 2-bit field takes 2 (and, sdot). So: **g128** keeps the base-3 packing but one scale per 128 weights (Ternary-Bonsai-2's adjacent scales are identical in all 202 tensors, `tools/q20_stats.py`): 1.75 bits/weight. **f2** stores ternary values in 2-bit fields with the same scales: 2.125 bits/weight | Ternary-Bonsai-2-4B, TV, mean of 3 rounds: g64 **3.27** → g128 **3.44** → 11 layers f2 3.75 → 16 layers 3.83 → 21 layers 3.98-4.03 → 26 layers 4.13 → all 32 **4.33** → + head **4.46 tok/s (+36 %)**. Exact: Mac PPL 12.0733 for g128 and every f2 mix (12.0690 for the per-64 layouts; merging scales changes float rounding). **But real chat collapsed:** 0.28 tok/s, prompt 3.7, and the next run ended with a closed connection and the box unreachable (§18.4). Rejected. Raw: `bench/raw/E76-*` |
| E77 | **The same on Ternary-Bonsai-4B.** Its adjacent scales differ, so two options. **f2g64** keeps all 4 scales per 256 weights (2.25 bits/weight, lossless). **Merged scales** (`GGML_Q2_0_SCALES_AVG=1`): one averaged scale per 128 weights, then g128 / f2 as above | **Merging costs almost nothing: Mac PPL 12.3711 → 12.3939 (+0.18 %), multilingual 14.3536 → 14.3729 (+0.13 %), bits/byte on the head's positions 0.9481 → 0.9488.** TV, mean of 3 rounds: g64 **3.40** · f2g64 9 layers 3.60, 18 layers 3.81, 27 layers killed by the guard in 2 of 3 runs · merged g128 **3.68** · merged f2 18 layers 4.18, 27 layers 4.49, **all 36 layers 4.81 (+41 %)**, + head 4.81 (no gain, so left out). **Real chat, 2 runs each: 3.17 / 3.28 → 4.29 / 4.33 tok/s (+33 %), prompt 6.9-7.0 → 9.2-10.0**, RSS 1.01-1.04 GB. Answers: 2 of 4 byte-identical, the other 2 differ in wording (the same two in both runs). **New best 4B.** Raw: `bench/raw/E77-*`, `bench/results.jsonl` |
| E78 | **Trickle-down** | **Doesn't reach the 2B or 0.8B.** The gain comes from the base-3 kernel being compute-bound; their Q4_K kernel is RAM-bound (E27: 6.7 of 8.3 GB/s, with rescheduling gaining nothing). Ternary-Bonsai-2-2B would get the layouts (all 152 tensors share scales), but its PPL 17.09 is far behind the 0.8B's 13.24, and the 0.8B is already faster, so it wasn't run on the TV. Raw: `bench/raw/E78-tb22-ppl.txt` |

### 18.3 Where the 4B stands

| Ternary-Bonsai-4B on this TV | Bits/weight in RAM | Bits/byte | llama-bench | Chat | Prompt |
|---|---|---|---|---|---|
| Lossless, §13 (`patched11`) | 1.875 | 0.964 | 3.21 | 3.07 | 7.2 |
| Same, today's build (`patched29`) | 1.875 | 0.964 | 3.40 | 3.17-3.28 | 7.0 |
| **Merged scales + 2-bit fields (`patched29`), new best 4B** | **2.125** | **≈0.965*** | **4.81** | **4.29-4.33** | **9.2-10.0** |
| Lossless, 18 of 36 layers in 2-bit fields (no text change) | 2.06 | 0.964 | 3.81 | not run | |
| *for comparison:* Qwen3.5-2B, the default | 4.5 | 0.909 | 7.7 | 7.7 | 17.8 |

\* 0.964 plus the +0.0007 measured on the head's positions; the full-vocabulary run wasn't repeated.

**Verdict.** The 4B is **33 % faster in real chat at almost the same quality**. The 2B stays the default, with better quality and more speed: 7.7 tok/s in chat yesterday evening, 6.65-6.89 in today's llama-bench against the 4B's 4.81 (about 1.4×). A same-day chat run of the 2B wasn't possible before the box went down. The research's main lesson for this model size: no published method beats a model trained for ternary at under 2 bits, and the good fine-tuning recipes need GPUs. The gain came from measuring where the kernel's time goes.

**The lesson:** on an in-order core, count NEON ops as well as bytes. Packing a weight tighter (1.6 instead of 2 bits) saves 20 % of the bytes but costs 2.5× the ops, and at this bit width compute is the larger cost. This is a mirror image of Q4_K, where RAM is the limit.

**Left open:** JQ4_K (quality for the 2B and 0.8B), GSQ or the ICLR fine-tune on rented GPUs, and a chat run of the lossless 18-layer mix.

**To use it:** `MODEL=tb4-q20-hml.gguf tv-llm-ui/start.sh` (it picks `patched29` and the settings below). Same model file as §13.

**New environment switches:**
- `GGML_Q2_0_SCALES=128`: one scale per 128 weights (aborts if adjacent scales differ), `GGML_Q2_0_SCALES_AVG=1`: average them instead (lossy).
- `GGML_Q2_0_F2_LAYERS=a-b`: 2-bit fields for `blk.a`..`blk.b`, `GGML_Q2_0_F2_HEAD=1`: also the pruned head.
- `GGML_Q2_0_F2G64_LAYERS=a-b`: 2-bit fields that keep the per-64 scales (lossless), without `GGML_Q2_0_SCALES`.

### 18.4 What went wrong this round

- **The box went silent at the end, and my test caused it.** After the second E76 chat attempt on `tb24-q20-h65p8.gguf`, ping, adb and the chat port stopped answering at about 18:37, and stayed silent until you power-cycled it at 00:58. It hadn't crashed: `boxstats.sh` was still running 6 hours later. Its last sample (`bench/raw/E76-outage.txt`), taken at 00:44:46, shows **28 MB available, swap free 0, `llama-server` at 780 MB RSS, load 11.84**, and that one-second loop took about 7 minutes to finish writing it. The server log shows the slide: the last request ran at 0.13 tok/s. So the 1.26 GB model left the box thrashing with memory and swap both exhausted. That the thrashing is what silenced the network is my inference, the cold boot left no kernel log I can read. The chat runs went through `mtp-bench.sh`, which has no memory guard, and nothing stopped the server. A guard there would have ended this in minutes.
- **Reverting the huge-page patch with `git checkout -- ggml/src/ggml.c` also removed an older tvbox line in that file,** because the whole patch lives as uncommitted changes on top of upstream. It looked like a crash in the new model (fp16 state assert), and I first told you so. Restored from the committed patch; no other file was affected.
- **Two Mac exactness checks were weaker than first reported:** zsh doesn't split `env $E`, so the head setting wasn't applied, and one combined run only tested g128. Rerun under bash; the numbers above are from the reruns. TV scripts weren't affected.
- **Smaller slips:** `pkill -f` killed its own adb command again (already in my notes), and a loose grep downloaded the wrong Bonsai variant twice (wasted time only, checksums verified).

## 19. The 8-9B push: 3-4 tok/s, anything allowed (2026-11-07)

**Goal (your call):** "go more advanced for the next 12 hours and squeeze the most out of 8-9B models, at least 3-4 tokens per second: extreme quantization, custom kernels, patching, custom engines, look online for research and models." Same rules: TV runs through `tv-guard.sh` (chat runs too, new this round), 3 threads, `LLAMA_BENCH_COMMON_TOKENS=4000`, real chat with `tools/mtp-bench.sh`. **Starting point:** Bonsai-8B (`b8-q1-hml.gguf`), 1.21 tok/s in chat, 2.2 in llama-bench, 0.974 bits/byte on the head's positions (0.991 with the full vocabulary).

**The result: 5.2 tok/s in real chat on Bonsai-8B (dense, 1-bit; 4.04 near a full 1,024-token context), about 4× the starting point, and 7.2 tok/s on Moon-Beam-8B-A1B (MoE, Q2_K experts paged from flash) with coherent long answers.** The 10 tok/s follow-up target is reachable only by ruining the model (top-2 experts: 10.42 tok/s, gibberish; E98). The Bonsai ceiling was a RAM-fit problem, and most of what was lost was going to Android, not to the model.

### 19.1 Experiments

| # | What | Result |
|---|---|---|
| E79 | **Is free CMA a hidden reserve?** `/proc/meminfo` shows `CmaTotal` 536 MB, and kernel watermarks don't count free CMA, which could explain the "~350 MB Android keeps free" of §14. `tools/kbench/dmaheap.c`, `anonprobe.c` | **✗ No.** Anonymous pages use CMA freely (`CmaFree` falls to 0 by 192 MB allocated). The CMA dma-heaps are denied to shell (`EACCES`, SELinux); only `system` opens, which is ordinary memory. Raw: `bench/raw/r6-logs` |
| E80 | **Headless mode** (`tools/tv-llm-mode.sh`): a small foreground activity, then force-stop both launchers, GMS and the other heavy apps. Non-persistent, except stay-awake (E83) | MemAvailable ~1000 → 1150-1300 MB. **Bonsai-8B fully resident for the first time** (962 MB, `GGML_REPACK_RESIDENT_MB=962`): 2.2 → **3.77 tok/s** llama-bench, **1.21 → 3.63 tok/s real chat**, prompt 6.2, min MemAvailable 186 MB. Same kernel, same weights. `tv.settings` costs 110-170 MB in front; `com.android.providers.calendar/.CalendarDebugActivity` costs ~50 MB |
| E81 | **LUT kernel** (`ggml-cpu/repack-q1lut-tv.inc`, Four-Russians, published as T-MAC 2407.00088 with averaging; `tools/kbench/q1_lut.cpp`). 32 rows per group; one 16-byte vector = 32 weight nibbles; one 16-entry table per 4 activations; ~8 NEON ops + 2 loads per 128 multiply-adds vs 16 + 4. **New relative to T-MAC:** tables are built straight from the float activations, normalized per 128 columns to the largest tile sum, one antisymmetric rounding, no averaging and no intermediate 8-bit activation. Loop interchange for batches: all activation rows walk one 128-column block before the next | Kernel, 3 threads from DRAM: 25.9 → 34.0 G weights/s (1.31×), cache-resident 1 thread 9.7 → 18.1 (1.87×), so memory stalls are now a real part of the time. Mac PPL 13.2394 → **13.2657 (+0.2 %)**, same bytes. llama-bench **3.27 → 4.6** (same block); chat **3.63 → 4.42**, prompt 5.1 |
| E82 | **Screensaver.** `top` during a slow run: `com.android.dreams.basic` 17 %, surfaceflinger 14 %, composer 5 %, `crtc0_commit` 3 %; `llama-bench` only 248 % of a possible 300 %. After 10 idle minutes the box starts an animated screensaver, and every run after that is slower (this explains a downward drift I first took for noise). `svc power stayon true` (disabling dreams would let the box sleep and drop off the network) | **llama-bench ~4.3 → 5.08-5.36, stable. Chat 4.42 → 5.11-5.21 (mean 5.15), prompt 5.8-6.1**, RSS 1.06 GB, MemAvailable steady at 240-255 MB |
| E83 | **Three ideas that lost** | **RMS_NORM split across threads:** no gain, removed. The op looked like 10 % in the 33-graph average, but that's the cold first graph; the steady-state node trace shows 2.9 ms of 208 ms (matmul 194 ms). **Helper thread on the 4th core** (outside the threadpool, no barriers, cancellable, nice 19 / nice 0 / SCHED_IDLE): loses, 4.5-5.04 vs 5.08-5.36, removed. **`-t 4`:** 1.75 / 2.89 vs 3.74 / 3.16 (screensaver on). **Software prefetch** (PRFM every cache line, +128...+1024 bytes): 28.9-38.9 vs 36.8-39.5 G weights/s without. Clocks are fine: 2.5 GHz through a 150-token run |
| E84 | **Moon-Beam-8B-A1B** (`tablinum/Moon-Beam-8B-A1B-GGUF-Preview`, a fine-tune of Ling-3.0-tiny, MIT, `bailingmoe3`: 7.89 B, 128 experts per layer, 8 used + 1 shared, 23 expert layers, hidden 1536, so ~0.94 B of weights touched per token vs Bonsai's 7.2 B). Routing trace (`GGML_MOE_TRACE`, `tools/moe_sim.py`, `bench/raw/E84-*`) | Mac, full vocabulary: IQ2_XXS 2.46 GB **1.0195 bits/byte** (PPL 13.43), Q4_K_M 4.94 GB (non-experts at Q6_K) **0.9009** (PPL 9.93). Routing: **45.2 % of a token's experts were used by the previous token** (paper 2608.18261 reports 44.4 % on the same router), distinct experts 104-127 of 128, and heavy skew (the top 16 experts carry 41 % of uses in layer 1, 62 % in layer 22). LRU over the whole model: 32.4 % misses at 525 experts in RAM, Belady 15.9 %. Frequency-aware policies calibrated on half the text don't beat LRU on the other half (35.0-47.9 %) |
| E85 | **Bit-width ladder, Moon-Beam** (own imatrix from `data/corpus`, requantized from the published Q8_0 because the Mac has no room for BF16; all non-experts Q4_K) | Experts **Q4_K 4.25 GiB: PPL 10.10, 0.9075** · **Q3_K 3.37 GiB: 10.50, 0.9229** · **Q2_K 2.70 GiB: 11.78, 0.9679**. All beat Bonsai's 0.991 on the same full-vocabulary basis |
| E86 | **Q8_0 attention** (`ggml_flash_attn_q8_tv`, `ops.cpp`). A 1,000-token generation fell from 4.22 to 2.50 tok/s: attention costs 0.21 ms per context token, as much as the rest of the model at n = 1000. The new path does all K dots first with `sdot` (K rows shared by 4 q heads are read once), so the softmax maximum is known up front, and skips `expf` and V work for rows below M + ln(eps) | Mac, c = 1024: stock PPL 10.8349, eps 0 10.8531, **eps 1e-4 10.8399**, eps 1e-3 10.8815, eps 1e-2 11.0268. Tensor check (`GGML_FA_CHECK=1`): no node above NMSE 1e-9 against stock, and with stock's output fed forward PPL is exactly stock's. So the +0.17 % at eps 0 is rounding chaos: **the noise floor of these PPLs is about ±0.3 %**, which also covers E81's +0.2 %. TV, 988 tokens: average 3.27 → **4.48 tok/s**, ~900 tokens in 2.50 → **4.04**, slope 0.214 → ~0.07 ms per context token, no loss on short contexts (5.16 vs 5.2) |
| E87 | **Cache-aware routing**, simulated per layer (`GGML_MOE_POOL`, `GGML_MOE_BIAS` add a bonus to experts in the pool at the routing step; arXiv 2412.00099 reports the idea on other models). Moon-Beam Q3_K, 30 experts per layer | Bias 0: 23.74 % misses, PPL 10.5037 (unchanged, as it must be). 0.02: 18.38 %, 10.5193. 0.05: 13.74 %, 10.6729. 0.1: 9.54 %, 10.8946. Raw: `bench/raw/E87-bias.txt` |
| E88 | **`tools/tv-watch.c`**: guard + box stats in one static binary with no forks. `boxstats.sh` started about 25 processes a second, each able to preempt a compute thread and stall a barrier | **4.67 → 5.16 tok/s (+10 %) from stopping boxstats alone.** With tv-watch's stats on: 5.20-5.27; full `start.sh` path 5.06. Kill and normal-exit paths both checked. `tv-guard.sh` hands off to it when present |
| E89 | **Expert pager** (`ggml-cpu/moe-pager-tv.inc`): each `*_exps` tensor's whole-page interior is overlaid with empty anonymous memory; right after a layer's routing the missing experts are read with parallel O_DIRECT straight into their own addresses, with a per-layer LRU. Kernels see an ordinary tensor. `GGML_MOE_VERIFY=1` compares every load with the file, since an unloaded page reads as zeros, not as an error | Works first time; verify clean. Moon-Beam Q3_K (plain mapped, generic `vec_dot`), TV: **bias 0 pool 30: 1.87 tok/s** (28.4 % misses), **bias 0.2: 3.85 tok/s** (7.3 %), steady token 253 ms: experts 82, flash 80, Q4_K 58. **Real chat 3.69 tok/s, prompt 4.3**, RSS 1.2 GB, answers correct |
| E90 | **Cache-aware routing, full curve** (Q3_K, pool 30) and **three ideas that lost** | Bias 0 / 0.02 / 0.05 / 0.1 / 0.2 / 0.4: misses 23.7 / 18.4 / 13.7 / 9.5 / 5.2 / 1.7 %, PPL 10.50 / 10.52 / 10.67 / 10.89 / 11.06 / **13.90 (cliff)**. **Static pruning** (route only among a layer's most used experts, lists from other text): keep 60 PPL 16.87, keep 40 33.60: ✗. **Runner-up prefetch** (load ranks 9+ for the next token): misses 5.23-5.66 % vs 5.22 %, and 24.07-25.16 % vs 23.74 % at bias 0, with up to 416,190 extra loads: ✗, runners-up don't predict the next token. **Protected ranks** (top P chosen on true scores, the rest on a strong bias): top-5 protect 2 PPL 13.42-13.56 at 2.8 % misses, protect 3 12.89-12.90 at 4.65 %, vs plain bias 0.2 12.94 at 2.31 % and 0.1 12.90 at 4.21 %: ✗ no better |
| E91 | **Fewer active experts** (`LLAMA_MOE_USED`; weights renormalize): pool 33, bias 0.2, Q3_K with 60K head | Top 8 / 6 / 5 / 4: PPL 10.59 / 11.14 / 11.86 / **14.40 (cliff)** |
| E92 | **Flash, expert-sized reads** (`tools/kbench/randread.c`) | 340 KB: 122 / 157 / 171 MB/s at 1 / 2 / 4 threads; 1 MB: 160 / 223 / 231; 8 MB: 244 / 267 / 263. A lone 1 MB read takes 6.2 ms, three parallel 340 KB reads about 5.9 ms, and misses are usually isolated, so laying out an expert's three matrices together isn't worth doing. **A miss costs about 6 ms.** Pager reader threads 2 / 4 / 8: 133 / 143 / 145 MB/s |
| E93 | **Q2_K experts with the existing repacked kernel** (E34, ~14 G weights/s vs Q3_K `vec_dot` ~5), expert pager on the repack cache: non-experts copied to RAM, experts demand-loaded from it (`repack_cache_moe_place`), GGUF's own expert bytes hole-punched (`tools/gguf-punch.c`: 2.76 GB → 477 MB on flash, so the model costs one copy of its weights). 60K head (`data/head-ids-ling-60k.txt`, `gguf_build.py`, `bailingmoe3.cpp` port): saves ~85 MB and ~15 ms | Mac, pool 43: bias 0 **11.20** (13.0 % misses); bias 0.2 top-8 **11.62** (2.22 %), top-6 **12.22** (2.32 %), top-5 **12.94** (2.31 %); bias 0.1 top-6 12.09 (4.27 %), top-5 12.90 (4.21 %). Steady token: 128 ms at top-8, 110 ms at top-5. **Warm start** (`GGML_MOE_WARM`, lists from calibration text, `data/moe-warm-moonbeam.txt`): the pool starts full, cold misses go to load time. Pool 48: guard kill at RSS 1.2 GB, so 43 is the ceiling |
| E94 | **DeltaNet state read by id** (ported from qwen35.cpp: the fused op reads the state from the cache instead of a 1 MB/layer GET_ROWS) | Steady token **105.8 → 100.2 ms**. `LLAMA_STATE_F16=1` still asserts here: that path needs the fused kernels this model's KDA attention doesn't use |
| E95 | **Non-blocking misses** (simulated: a token never waits; a top-ranked expert not in RAM is skipped for this token and loaded for later ones) | Top-5, bias 0 / 0.05 / 0.1 / 0.2: PPL 18.00 / 16.43 / 16.51 / **16.48** with 0.75 / 0.35 / 0.23 / 0.12 experts skipped per layer. **✗ Skipping 2.4 % of experts costs +27 % PPL:** the misses that survive a bias are exactly the experts whose score beats every cached one plus the bonus, the ones the router wanted most. Blocking on them is what preserves quality. Not built |
| E96 | **Non-uniform expert pools and the bias knee.** `tools/moe_alloc.py`: per-layer LRU miss curves on calibration text, slots allocated greedily by marginal benefit, tested on the eval text | By benefit vs uniform 43: top-8 13.00 → 12.34 % misses (-5.1 %), top-5 9.72 → 9.03 % (-7.1 %): about 1.5 ms per token, not built. Bias knee, Q2_K pool 43: top-5 0.25 / 0.3: PPL 13.70 / 14.33 (1.76 / 1.36 %); top-6 0.25: **12.64** (1.76 %), better than top-5 at 0.2 on both counts; top-8 0.25 / 0.3: 11.90 / 12.17 (1.69 / 1.28 %). Raw: `bench/raw/E97-alloc.txt` |
| E97 | **All non-experts at Q2_K** too (compute-bound kernel, ~120 MB more pool) | ✗ top-6 bias 0.25 pool 52: PPL 16.16 at 0.98 % misses; top-8 bias 0.2: 14.52. Attention doesn't take 2 bits. Raw: `bench/raw/E98-allq2.txt` |
| E98 | **What 10 tok/s costs.** Free-running chat, 3 prompts × 300 tokens at temperature 0.7, repeated 6-grams (`tools/rep_chat.py`), PPL on the Mac | Top-8 bias 0: 4.40 tok/s, 5.4 % repeats, PPL 11.20 · top-8 0.2: 6.85, 0.6 %, 11.62 · top-8 0.3: 7.49, 6.8 %, 12.17 · **top-6 0.25: 8.06, 0.2 %, 12.64** · top-5 0.2: 8.08, 5.4 %, 12.94 · top-4 0.2: 8.50, 10.0 %, 15.86 · top-3 0.2: 9.22, 10.1 %, 24.61 · **top-2 0.2: 10.42 tok/s, 5.2 %: gibberish** ("organized around the the main figure name of that date"). **So 10 tok/s is reachable on this box only by ruining the model.** Texts: `bench/raw/E100-repetition.txt` |
| E99 | **Long answers loop** (`tools/long_chat.py`, `tools/loop_stats.py`, `tools/degen_score.py`). PPL scores one token at a time on clean text and can't see a model falling into a loop in its own output. Metric: a run is *degenerate* if the second half of a ~1,000-token answer compresses below 0.30 (zlib; prose is ~0.40) or over 40 % of sentences reuse an earlier sentence's first three words, which also catches paraphrased loops a repeated-6-gram count misses | Degenerate long answers (16 seeds unless noted, Mac with routing simulated, TV with the real pager). **Q8_0 reference: 0/6.** Q4_K experts: 7-8/16. Q3_K: 6/16 without bias, 10-12/16 with bias 0.2 (TV: 3/8 and 9/16). **Q2_K: 12-16 of 15-16 in every configuration** (TV default: 12/13; TV top-8 bias 0: 7/7). Ruled out: the q8 KV cache (f16 loops as often), the Q8_0 attention path (stock attention loops on the TV too), the pruned head, top-6 vs top-8, temperature 1.0 / top-k 20. **Cause: Q2_K experts break long-form coherence after ~250-300 words; routing bias amplifies it** (it keeps the model on fewer distinct experts). Raw: `bench/raw/E102-*`, `E103-*`, `E104-*`, `E105-*` |
| E100 | **Sampler fixes** (Q2_K, top-6, bias 0.25) | DRY 0.8 / 1.2: "0/16 loops" by repeated 6-grams but **11/15, 8/12 degenerate**: it blocks exact repeats and the model paraphrases ("The abacus was traded. The abacus was valued. ...", `bench/raw/E106-dry2-texts.txt`). Presence penalty 0.6: 7/9. **Repeat penalty 1.15 over 256 tokens: 1/15 (Mac), 1/13 on the TV**, texts coherent. Code (TV, 6 seeds): with the penalty the module is well structured but has penalty typos ("OSErrrors", "Bom milk"); without it the code collapses into `print("30. Run a todo") ... print("38. Run a todo")`. Now the Moon-Beam default |
| E101 | **MTP speculation** (the published 0.55 GB draft head, `-md`, `--spec-type draft-mtp`), Mac, deployed routing + penalty | n=1: **37.9 % accepted, 1.38 tokens per pass**, 42.4 → 38.0 tok/s; n=2: 25.1 %, 1.50, 29.4 tok/s. ✗ A second token costs ~1.5× a pass on the TV, so this loses |
| E102 | **Qwen** | The smallest official Qwen MoE is Qwen3.6-35B-A3B (3 B active, 3× Moon-Beam's): far beyond this box at 10 tok/s. Dense Qwen3-8B is what Bonsai-8B is, at 1 bit |
| E103 | **Pre-gated expert prefetch** (rewritten pager: page states empty / pending / filled, edge pages buffered, urgent reads first; `GGML_MOE_PREFETCH_K`, `GGML_MOE_PREFETCH_AHEAD`). Layer L+a's router applied to layer L's input. Simulated on real generation: one layer ahead, top 6 predicts **97.8 %** of pool misses; two ahead 94.2 % (98.5 % with 12) | First versions: guard kills at 130 / 108 MB (prefetches and asynchronous warm-start reads piled up past the pool; fixed: a prefetch only takes an evictable slot, warm start synchronous). Plain top-k then read 61.7 GB vs 11 GB and was slower (6.5 vs 7.4 tok/s). **Decision-aware** (sigmoid + the model's selection bias + the cache bonus, as the router will decide): 932 prefetches for 1,460 misses, verify clean. Real chat, 400 tokens: **6.93 → 7.20 tok/s**, flash waits ~32 → ~20 ms per token. 4 threads: 6.51 |
| E104 | **Where a real chat token goes** (TV, server, `LLAMA_UBATCH_PROF`, 400 tokens, penalty on): 144 ms, all of it graph compute | Q4_K non-expert matmuls ~50 ms (RAM-bound), flash waits ~32 ms (4.7 misses × 6.8 ms; varied text misses more, 3.2 %), Q2_K experts ~26, DeltaNet + attention ~18, the rest ~18. Raw: `bench/raw/E111-srvprof.txt` |

**A correction about quality.** Several Moon-Beam rows above compare PPL with Bonsai's directly. The two models use different tokenizers, so the fair comparison is bits per byte (log2 PPL × tokens / bytes; Moon-Beam 5,993 tokens, Bonsai 5,756 on `eval-en.txt`). Bonsai's 13.25 is 0.974 bits/byte; Moon-Beam breaks even at PPL ≈ 11.96. So top-8 bias 0.2 (11.62) is 0.963, better than Bonsai; top-6 bias 0.25 (12.64) is 0.996 and top-5 bias 0.2 (12.94) 1.005, both slightly worse. I reported those as "better than Bonsai" during the session; they aren't.

**What went wrong, so far:**
- The first helper-thread A/B ran on a stale binary: the build had failed and my command didn't stop, so 12 runs measured noise. Caught from the build output before drawing a conclusion.
- The first routing trace (E84) read a view with the wrong stride, and my Belady simulator evicted the wrong end. The result (0.3 % reuse, below chance) was impossible, so it was rechecked, not reported.
- `tv-llm-mode.sh` force-stopped the very activity it had put in front, the launcher came back, and the guard stopped the run cleanly.
- The memory guard fired on chat starts, as designed; there was no thrash this round.
- A second run on a stale binary: the attention sweep ran after another failed build. `tools/build-check.sh` now fails loudly, and it was stopped and rerun.
- `start.sh` left `boxstats.sh` running, which silently made every later run 10 % slower and sent me after swap and ubatch first (both innocent: 14 page-ins per request, and -ub 32 / 128 / 512 all gave 4.64-4.72).

### 19.2 Where Bonsai-8B stands

| Bonsai-8B on this TV | Resident | Bits/byte | llama-bench | Chat | Prompt |
|---|---|---|---|---|---|
| §14 (`patched16`), 839 MB + streaming | 839 MB | 0.974 | 2.2 | 1.21 | 4.9 |
| Headless, all resident, bit-plane kernel (`patched29`) | 962 MB | 0.974 | 3.77 | 3.63 | 6.2 |
| **+ LUT kernel, no screensaver, fork-free guard (`patched34`)** | **962 MB** | **0.974** | **5.08-5.36** | **5.2** (4.04 at ~900 tokens of context) | **5.8-6.1** |

**To use it:** `MODEL=b8-q1-hml.gguf tv-llm-ui/start.sh`. It runs headless mode and starts under the guard. `sh /data/local/tmp/llm/tv-llm-mode.sh off` on the TV brings the launcher and the screensaver back.

**New environment switches:** `GGML_Q1_LUT=0` for the bit-plane kernel (needs its own repack cache, same size: use another `GGML_REPACK_CACHE` prefix or you get garbage). `GGML_MOE_TRACE=<file>` logs MoE routing; `GGML_MOE_POOL=n`, `GGML_MOE_BIAS=d` simulate per-layer expert caches and cache-aware routing.

### 19.3 Where Moon-Beam-8B-A1B stands

| Moon-Beam on this TV (`patched41`) | Flash | Bits/byte | Chat | Long answers degenerate |
|---|---|---|---|---|
| Q2_K experts, top-8, no bias, pager pool 43 | 477 MB GGUF (punched) + 2.58 GB cache | 0.948 | 4.40 | 7/7 |
| top-6, bias 0.25, warm start | same | 0.996 | 8.06 (300-token answers) | 12/13 |
| **+ repeat penalty 1.15 / 256** | same | 0.996 | 7.4 | **1/13** |
| **+ decision-aware pre-gated prefetch (default)** | same | 0.996 | **7.20** (400-token profile run), 7.3 in the loop test | 2/8 |

Max RSS 1.14 GB in headless mode, min MemAvailable ~170 MB. Bits/byte on `eval-en.txt` with the 60K head; Bonsai-8B is 0.974, so the default Moon-Beam is slightly worse per byte and ~40 % faster. A token costs ~140 ms: Q4_K non-expert matmuls ~50 (RAM-bound), flash waits ~20, Q2_K experts ~26, DeltaNet + attention ~18, the rest ~18 (E104).

**To use it:** `MODEL=mb-q2-h60.gguf tv-llm-ui/start.sh` (first `tools/rebuild-models.sh mb`). `POOL`, `BIAS`, `EXPERTS` override the pool size, routing bias and expert count.

**MoE switches:** `GGML_MOE_PAGER=1` (expert pager), `GGML_MOE_POOL`, `GGML_MOE_BIAS`, `LLAMA_MOE_USED`, `GGML_MOE_WARM=<list>`, `GGML_MOE_PREFETCH_K` / `GGML_MOE_PREFETCH_AHEAD` / `GGML_MOE_PREFETCH_INFLIGHT`, `GGML_MOE_VERIFY=1`; `GGML_FA_EPS` / `GGML_FA_Q8=0` / `GGML_FA_CHECK=1` for the Q8_0 attention path.


## 20. Bigger than 9B: Ling-mini-2.0, a 16B MoE (2026-11-08)

**Goal (your call):** "try to go a bit higher with bigger than 9B models and apply all what you learned." Same rules as §19: every speed is real chat on the TV through the server, headless mode, under the guard.

**The result: Ling-mini-2.0 (16.3 B parameters, 256 experts) runs at 5.1 tok/s in real chat on the TV** (5.0 over ~600-word answers), coherent and loop-free, with the §19 recipe unchanged: Q2_K experts paged from flash, cache-aware routing, warm start, pre-gated prefetch, repeat penalty. That is twice Moon-Beam's size at 70 % of its speed. **But it isn't better:** 1.054 bits/byte against Moon-Beam's 0.996 and Bonsai-8B's 0.974. Ling-mini-2.0 (Sep 2025) is a generation older than Moon-Beam's Ling-3.0 base, and on this box a bigger model only helps if it is also a better model per byte.

### 20.1 Candidates (Nov 2026)

What matters on this box is the weight touched per token and the non-expert bytes that must stay in RAM, not the total size.

| Model | Total / active | Experts | Verdict |
|---|---|---|---|
| **Ling-mini-2.0** (inclusionAI) | 16.3 B / 1.4 B | 256 × 1.03 MB (Q2_K), top-8 + shared, 19 MoE layers | **the only > 9 B model with Moon-Beam-sized per-token work**: 279 MB resident non-experts, 5.0 GB of experts |
| LFM2.5-24B-A2B (Liquid, Oct 2026) | 24 B / 2.45 B | 64 × 3.1 MB, top-4, 38 MoE layers | ✗ ~560 MB of non-experts (convolution and attention at Q4_K) leave room for ~4 experts per layer, and each miss reads 3 MB: estimated ~1 tok/s |
| Qwen3.6-35B-A3B (Qwen) | 35 B / 3 B | 256 × 1.03 MB, top-8, 40 layers | ✗ ~1.5 B non-expert parameters (DeltaNet and attention projections, 248K head) = ~870 MB at Q4_K, nearly the whole budget before one expert is cached: well under 1 tok/s. The Qwen route stays closed (E102) |
| Qwen3-30B-A3B-derived "A1.5B" variants, gpt-oss-20b, ERNIE-4.5-21B-A3B, SmallThinker-21B-A3B | 20-30 B / 1.5-3.6 B | | ✗ same arithmetic: too many non-expert bytes or too many bytes per expert |

### 20.2 Experiments

| # | What | Result |
|---|---|---|
| E105 | **Build** (`tools/rebuild-models.sh lm`, `tools/lm-prep.sh`). The Mac has no room for the 17 GB Q8_0, so bartowski's imatrix Q4_K_M is the source: experts requantized to Q2_K with his imatrix, non-experts Q4_K (the fast kernels), same tokenizer as Moon-Beam (9 reserved tokens differ), so the 60K head list is reused; `bailingmoe2.cpp` got the pruned-head port. 5.49 GB → repack cache 5.26 GB → GGUF punched to 472 MB. Warm list from a calibration trace (`data/moe-warm-ling.txt`): a layer's top 64 of 256 experts carry 56-82 % of uses | Mac, eval-en, 60K head: **1.0346 bits/byte** (PPL 13.96), Moon-Beam Q2_K top-8 0.948 on the same basis. Pool 43, top-8: bias 0.2 1.0558, **0.3 1.0537**, 0.4 1.0773, 0.5 1.0778 (knee between 0.3 and 0.4). Top-6: bias 0.2 1.0904, 0.25 1.0902, 0.3 1.1151, 0.4 1.1255: fewer experts costs this model more than Moon-Beam. Raw: `bench/raw/E105-*` |
| E106 | **On the TV** (`tools/lm-tv.sh`, real chat, 4 answers × 400 tokens, penalty on). Pager verify run first: clean | Pool 40, top-6, bias 0.25: **5.3 tok/s**, 8.4 % misses, RSS 1.08 GB. Pool 40, top-8, bias 0.2: 4.1. Pool 43 (RSS 1.13 GB), top-6: 5.3, 8.2 %; without prefetch 5.2, 11.6 %. **Pool 43, top-8, bias 0.3: 5.1 tok/s, 6.6 % misses**: the default, since top-6 is 4 % faster for much worse text. Flash waits are ~85 ms of a ~190 ms token. Raw: `bench/raw/E106-ling-tv.txt` |
| E107 | **Long answers** (default config, 6 × 900-token answers, ~600 words each; `tools/degen_score.py`) | **0/6 loops, 5.0 tok/s**, 6.2 % misses, RSS 1.13 GB, min MemAvailable 188 MB; 0/10 degenerate over every default-config run. Unlike Moon-Beam's (12/13 without the penalty, E99), Ling's Q2_K experts hold long-form coherence. Raw: `bench/raw/E106-ling-long.txt`, texts `E106-ling-texts.txt` |

### 20.3 Where it stands

| On this TV | Total params | Bits/byte | Chat |
|---|---|---|---|
| Bonsai-8B (§19) | 8 B dense, 1-bit | 0.974 | 5.2 |
| Moon-Beam-8B-A1B (§19) | 7.9 B MoE | 0.996 | 7.2 |
| **Ling-mini-2.0 (this section)** | **16.3 B MoE** | 1.054 | **5.1** |

**To use it:** `MODEL=lm2-q2-h60.gguf tv-llm-ui/start.sh` (first `tools/rebuild-models.sh lm`).

**What went wrong:**
- **The Mac crashed** during the first calibration trace: 8.6 GB of RAM, a 5.6 GB model, a 5.5 GB stream to the TV and only ~5 GB of free disk for swap at the same time. Since then: one heavy job at a time, at `nice`, with the disk kept above ~10 GB free.
- **A shell chain deleted the intermediate file:** `build | tail && rm src` tests `tail`'s exit status, so a full disk still ran the `rm`. Six minutes lost.
- **`adb exec-in` returns before the TV has the data:** the Mac side hands everything to the adb server and exits, so restarting the server twice truncated the file on the TV (4.7 and 2.3 of 5.5 GB). The model now goes through `adb push`, which checks completion.
- **zsh doesn't word-split `set -- $cfg`** again: three Mac runs did nothing until rerun under bash.

## 21. Device #2: a 2013 iPad Air (2026-11-09)

**Goal (your call):** "run a small llm on it with llamacpp", then a real chat program and a bigger model.

**The result: SmolLM2-360M runs at 10.9-12.5 tok/s in chat on a 13-year-old tablet**, on the CPU, peaking at 316 MB. It answered all three test questions correctly. Files and rebuild steps are in [`ipad-air/`](ipad-air/README.md).

The device is an iPad Air 1 (`iPad4,1`): Apple A7, 2 cores, NEON without dotprod, 1,021 MB of RAM, iOS 12.5.7, which is its last version.

### 21.1 Getting a binary to run

| Blocker | What happened | Fix |
|---|---|---|
| Xcode | Xcode 16 and later only target iOS 15+. The last one for iOS 12 is 15.4, which officially needs macOS 14, and this Mac runs macOS 27. | Only the iPhoneOS SDK is taken from the 15.4 archive. The Command Line Tools `clang` compiles against it. Xcode itself never runs. |
| App signing | A non-jailbroken iPad needs a signed app bundle for any native code. | Jailbreak with Amethyst, which supports A7 on 12.5.7 and re-applies from Safari after a reboot. Binaries then run from a root shell over SSH. |
| llama.cpp on iOS 12 | The build script targets iOS 16.4, and `std::filesystem` needs a newer OS. | Static CPU-only build. `llama-simple` and `smolchat` skip `common/`. `std::filesystem` in the core is used only to find backend plugins, so a 26-line `GGML_NO_FS` switch compiles it out. |
| CMake | Configure failed because iOS executables default to app bundles. | `-DCMAKE_MACOSX_BUNDLE=OFF`. |
| Launch | Exit 137 at launch with ad-hoc signing, then with `ldid -S`. The kernel log showed `Sandbox: hook..execve() killing`. | `ldid -Sent.xml` with `platform-application` and `no-container`, the entitlements the jailbreak's own tools carry. |

### 21.2 Experiments

| # | Experiment | Result |
|---|---|---|
| E108 | First run: `stories15M-q4_0` in `llama-simple`, default thread count | Works, coherent, but **0.54 tok/s** (89 s for 48 tokens). The default of 4 threads on 2 cores fights over the cores. |
| E109 | Thread count on `stories15M` | 1 thread 187 tok/s, **2 threads 290 tok/s**: about 540× faster than E108. |
| E110 | SmolLM2-135M Q4_K_M in `llama-simple`, 2 threads | 31.5 tok/s, 124 MB peak RSS, 1.8 s load. Greedy sampling loops: "The capital of France is Paris" three times, and tides never answered. |
| E111 | `smolchat` on the 135M, three turns plus `/reset` | Recalled the first question, no loops. Reply length now limited only by `-n`. |
| E112 | Apparent slowdown: `smolchat` measured 9-12 tok/s against 31.5 | **False alarm.** Run back to back: `llama-simple` 34, `smolchat` 29.3 on the first turn. The low numbers came right after copying large files to flash. |
| E113 | Does the repeat penalty or context size cost speed? (measured during the slow period of E112) | No: 8.93 tok/s with the 1.15 penalty, 8.54 without; a 256-token context 8.85 vs 8.93 at 1,024. |
| E114 | Gemma 3 270M-it Q4_K_M (241 MiB) | 15.4-19.0 tok/s, 248-294 MB. Weak answers: tides blamed on "the ocean floor", and an empty reply to the recall question on every run. |
| E115 | Qwen3.5-0.8B Q4_K_M (508 MiB), thinking off, under the guard | **Thrashed.** Stopped by hand after 8 m 40 s with the first of three replies not finished: about 0.2 tok/s, estimated from output size. 506 MB RSS, 33 % CPU, 11.3 million page-ins. It didn't trip the memory guard, because evictable file pages count as available, so `bench.sh` now has a time limit too. |
| E116 | SmolLM2-360M-Instruct Q4_K_M (258 MiB) | **10.9-12.5 tok/s**, prompt 18-20 tok/s, 269-316 MB, about 360 MB still available. Correct on all three questions. The model to use. |
| E117 | Web chat: `smolchat --serve`, a one-request-at-a-time HTTP server with a Safari 12 page, instead of `llama-server` (needs `common/`, and its page is too modern) | Over HTTP from the Mac: tokens stream one by one at 12.9 tok/s, the same as the terminal; the second turn recalls the first; dropping the connection stops generation, and the next reply arrived within 2 s. 320 MB RSS. |

### 21.3 Where it stands

- The web chat works in Safari on the iPad itself.
- The practical ceiling with iOS's normal apps running is a model of about 250-350 MB. SmolLM2-360M sits inside it.
- An untried lead is the TV's headless idea: stopping iOS's UI processes to free RAM might make Qwen3.5-0.8B resident. It would blank the screen until a restart, so it hasn't been tried.
- The TV's kernels mostly need dotprod, which the A7 lacks, so they don't carry over directly.
- Speeds here come from a few short runs, and the first timed run after a file copy is unreliable. Treat them as approximate.

### 21.4 A 2B-class model: Bonsai-1.7B (2026-11-09)

**Goal (your call):** "go optimize a 2b model to run there nicely." Qwen3.5-2B took 0.99 GB on the TV even after optimization, and this iPad struggled at 508 MiB, so the same recipe can't fit. Bonsai-1.7B, trained for 1-bit weights, is 237 MiB.

**The result: Bonsai-1.7B went from 2.75 to about 11 tok/s** on a chip without dotprod, and answered the test questions correctly.

| # | Experiment | Result |
|---|---|---|
| E118 | Bonsai-1.7B Q1_0 baseline, stock Q1_0 NEON path, thinking off | 2.63-2.83 tok/s, prompt 3.5-3.8, 387 MB peak RSS. Correct answers. Stock code emulates dotprod with widening multiplies and byte tables. |
| E119 | Multiplication-free Q1_0 kernel: bits become masks and sum(w * y) = 2 * sum(y where bit set) - sum(y), with lane sums matching the stock path | `kbench.c`: 206,006 vectors bit-identical, including extremes. 2.89 -> 3.44 G weights/s (×1.19). **The first reading was 118 G weights/s and was wrong:** the compiler had removed the repeated calls to a side-effect-free function, and a barrier fixed it. In chat, three alternating runs: 2.75 -> **3.45 tok/s**, word-for-word identical output. |
| E120 | The TV's Four-Russians LUT kernel (§19), ported to base NEON. Its dotprod gate was only an assumption. Tensors needing at least 4 groups of 128 columns; the rest use E119 | Same binary, alternating runs: 3.5 -> **10.5-11.8 tok/s**, prompt 4.6 -> 19-33 tok/s. Peak RSS 454-506 MB, lowest available memory 255 MB, no kills. Output is no longer bit-identical (the kernel rounds differently) but stays correct. |
| E121 | Long use through the web server, 1,024-token context | 400 tokens at 10.15 tok/s, then 26 at 8.08, then 297 at 6.21 as the conversation grew. Coherent, correct `is_prime`, most repeated 3-gram ×2 (no loops), 523 MB RSS, no kills. Whether the drop is longer context, heat, or both was not separated. |

Quality here means the three-question test and the code answer. Perplexity was not measured on this build, because `llama-perplexity` needs `common/`. The TV measured this model at 1.158 bits/byte, against 0.909 for Qwen3.5-2B.

### 21.5 Bonsai-1.7B, round 2: 17 tok/s and more (2026-11-09)

**Goal (your call):** "optimize more till at least 17 tok per second."

**The result: about 24-26 tok/s in short chats (was 10-11) and 18-22 tok/s through long ones (was 6-10)**, same model file, in alternating runs against the previous build. The biggest single cause was a matrix that had never reached the fast kernel.

| # | Experiment | Result |
|---|---|---|
| E122 | **Where a token's 90 ms went.** New per-op profiler (`GGML_OPPROF=1`), RAM probe (`tools/membw`), standalone copy of the LUT kernel | MUL_MAT 83 %, attention 11 % (growing with context), sampling 2-3 ms. RAM reads 10.9 GB/s with 2 threads and a token used about 2.5, so it was not memory-bound. The kernel alone did 58.6 G weights/s from DRAM, which predicts 29 ms per token, against about 72 ms measured. |
| E123 | **The tied LM head never used the LUT kernel.** Bonsai has no `output.weight`: the head is `token_embd`, which GET_ROWS must read in the plain layout, and 151,669 rows is not a multiple of 32. Its 311 M weights (18 % of the model) ran on the per-row dot product, about 45 of the 72 ms. Fix: a private LUT copy of the head's whole 32-row groups, built once on first use across both threads; the last 21 rows are computed directly | **11 → 18-22 tok/s.** One-time cost: the first prompt takes about 2-3 s longer. About 42 MB more anonymous memory. Checked on the iPad against an exact float head for 60 tokens: **same top token on 59 of 60**. The two lines inspected showed relative RMSE about 0.8 % and KL about 0.0003 nats. The check code is not in the patch. |
| E124 | **Attention, first version.** A long chat showed attention at 50 % of compute, which also answers E121's open question: the slowdown was mostly context, not heat. All K dots first, one pass per K row for the two q heads sharing it, then the softmax maximum is known and rows below M + ln(eps) skip expf and V (the TV's E86 idea, here on the F16 cache with no quantization). eps = 1e-4 | Long chat, same three questions: **15.4 / 7.6 / 8.2 → 18.9 / 12.4 / 13.0 tok/s**. |
| E125 | **Row-order attention for one-token decodes.** A K dot took 85 cycles per row in a microbenchmark but about 270 in the model. The KV cache stores all 8 heads of a context row side by side, so one head at a time strides 2 KB through memory. Now each context row is read straight through for all heads, split across threads by rows, and four heads share one vector `exp` | **18.9 / 14.8 / 14.9 tok/s** (eps still 1e-4). |
| E126 | **Sampler.** llama.cpp's chain copies and sorts the 151,669-entry vocabulary per token. `smolchat` now does the same steps (penalty, top-40, min-p 0.05, temperature, weighted draw) with one pass and a 40-entry heap | **3 → 0.23 ms per token. 20.3 / 15.3 / 15.1 tok/s.** |
| E127 | **The pruning threshold.** Perplexity on the Mac (`data/eval/eval-en.txt`), native build of this tree, batch path, CPU | Context 128, 40 chunks: stock 21.5931, eps 0 21.5771, 1e-4 21.5823, 1e-3 21.6088, 3e-3 21.6290. **Context 1,024, 5 chunks:** stock 16.6886, 1e-4 16.7065 (+0.11 %), **1e-3 16.7582 (+0.42 %)**, 3e-3 16.7772 (+0.53 %). The repo's noise floor for these numbers is about ±0.3 %. **Chosen default: 1e-3**, the only setting tried on the iPad that kept long chats above 17: 21.7 / 17.9 / 17.8 tok/s, against 15.3 with 1e-4. `GGML_FA_EPS=0.0001` restores the safer setting. |
| E128 | **Final, on the iPad** (three-turn `bench.sh`, 3 alternating rounds; long chat of 400-token answers, 1,024-token context; web server over HTTP) | Three-turn: **23.7-25.8 tok/s vs 8.1-11.6** for the previous build; later prompts 29-33 tok/s; peak RSS 605 vs 567 MB; lowest available memory 348-360 MB; no kills. Long chat: **21.8 / 17.9 / 18.4 tok/s**, correct `is_prime`, no loops. Web server: **26.9 and 24.8 tok/s**, follow-up kept context, POST without the header still 403. |

**Not measured.** Perplexity with the LUT head: the Mac has dotprod and doesn't take that path, so the head was checked on the iPad only (E123). E127's numbers measure the attention threshold alone, and the 1,024-context row has only 5 chunks. Bonsai still gets facts wrong at this size (in the web test it called Mercury the closest planet to Earth); speed work doesn't change that.

**What went wrong:** one long run read 400 ms per token because a script error had left an earlier benchmark running, so two copies shared the CPU. It was discarded and rerun. A later comparison of pruned and stock attention never ran, because of a shell quoting slip; E127 replaced it.

**New environment switches:** `GGML_OPPROF=1` (op table at exit), `SMOLCHAT_PROF=1` (sampling and decode time per token), `GGML_FA_EPS=x` (attention pruning, 0 keeps every row), `GGML_FA_PRUNE=0` (stock attention).

### 21.6 A 4B model on 1 GB: Bonsai-4B (2026-11-09)

**Goal (your call):** "get to the 4B models." The iPad has 1 GB and a 4B model needs about 3.6 B weights in RAM, so only a 1-bit model can fit: Ternary-Bonsai-4B is 1.07 GB as a file. [Bonsai-4B](https://huggingface.co/prism-ml/Bonsai-4B-gguf) (Q1_0, 572 MB, SHA-256 `4524b3f9…2168`, Apache-2.0, 36 layers, hidden size 2,560, 32 q heads over 8 KV heads) is 4.02 B parameters.

**The result: a 4B model runs on a 2013 iPad at about 10 tok/s (9.2 through long chats), with the context cut to 512 tokens.**

| # | Experiment | Result |
|---|---|---|
| E129 | **Does it fit?** Default settings: 1,024-token context | Short chat fits (627 MB, 10 tok/s). A long chat doesn't: 9.06 tok/s for 400 tokens, then Jetsam reports `compressor-thrashing` with `smolchat` at 705 MB, killing `pkd` and dropping the SSH session. 1-bit weights don't compress, so the compressor can only churn. **About 700 MB is the practical ceiling.** The KV cache adds 147 KB per token. `-c 768 -b 64` also lost the session. |
| E130 | **A smaller context and batch** (`smolchat -b`, new: `n_ubatch` 512 → 64, smaller work buffers) | **`-c 512 -b 64`: three long answers, no kills**, peak RSS 615-682 MB across runs, lowest available memory 137-198 MB. Older turns are dropped once the 512 tokens fill. |
| E131 | **Attention for 4 q heads per K head.** E125's path assumed 2. Generalized with a fixed-width template, for one-token decodes; many-token prompts on this shape use the stock path | Temporary node-by-node comparison with stock on the iPad (`-t 1`): **eps 0: mean NMSE 5.9e-7 (4B), 4.5e-7 (1.7B)**, the level of stock's own F16 accumulators. **Default eps 1e-3: mean 1.4e-5, worst node 7.7e-5 (4B); mean 7.5e-6, worst 4.2e-5 (1.7B).** No node above 1e-4. Greedy text matched stock for 338-646 characters before a near-tie diverged. This is also the direct check E125 had lacked. |
| E132 | **Final numbers**, clean build, `-c 512 -b 64` | Three-turn test ×3: **9.4-11.4 tok/s** after a slower first prompt (3.5 tok/s, including the head build). Long chat: **9.21 / 9.19 / 10.30 tok/s**, about 108 ms per token, correct `is_prime`, no loops. |
| E133 | **Quality on the Mac** (same method as E127: `eval-en.txt`, context 128, 40 chunks) | **Bonsai-4B PPL 17.2105 = 1.0726 bits/byte; Bonsai-1.7B PPL 21.5823 = 1.1579.** Batches use the Mac build's attention, which here means stock attention for the 4B and eps 1e-4 for the 1.7B; the 1.7B moved by under 0.1 % between those in E127. The TV measured 1.073 and 1.158 for the same files. **Bonsai-4B is about 7 % better per byte but failed "What was my first question?" in every run**, saying it had no access to earlier messages. The same chat code passes that question on the 1.7B. |

**Web server, same settings:** 10.93 and 11.82 tok/s over HTTP, RSS 708 MB, right at the ceiling. **The short context shows:** after a 400-token reply the chat dropped its two oldest messages, so the follow-up "Which of those is largest?" had nothing to refer to. Safari was not open during this test, and it will add memory pressure.

**Where it stands.** Matmuls are 84 % of a token and run at about 53 G weights/s, close to the 58.6 the kernel reaches on its own, so more speed needs a faster kernel, not more plumbing. RAM isn't the limit: 534 MB per token at 10 tok/s is about half of the 10.9 GB/s measured. **Bonsai-1.7B stays the default**: about 2.5× faster, and it answers the recall test.

**What went wrong:** the first long 4B run lost its SSH session and left the benchmark running with no guard; it was found and killed by hand. `long.sh`'s guard then got an RSS limit, which wasn't enough at `-c 768`, because Jetsam acts on system-wide pressure.

**To use it:** `./smolchat -m Bonsai-4B-Q1_0.gguf --no-think -c 512 -b 64 --serve 8080`.

### 21.7 8-9B models on 1 GB (2026-11-10)

**Goal (your call):** "go to try 8-9B models." Two candidates had run on the TV: Bonsai-8B (dense, 1-bit) and Moon-Beam-8B-A1B (a mixture of experts that touches about 0.94 B weights per token).

**The result: neither is usable yet.** Dense Bonsai-8B runs at 0.08 tok/s. Moon-Beam can't even be mapped into memory, and the obvious workaround (keeping fewer experts) ruins it. A working Moon-Beam needs an expert pager written for iOS's address-space limit.

| # | Experiment | Result |
|---|---|---|
| E134 | **Flash read speed** (`probes/fbench.c`, uncached, `F_NOCACHE`) | 1 thread: 82 MB/s with 64 KB blocks, 215 with 1 MB, 250 with 4 MB. 2 threads: 150 / 249 / 262 / 258 MB/s. **About 260 MB/s at best**, a bit under the TV's 278. |
| E135 | **Bonsai-8B, dense** ([prism-ml/Bonsai-8B-gguf](https://huggingface.co/prism-ml/Bonsai-8B-gguf), 1.16 GB, SHA-256 `284a335a…`). Its LUT layout needs about 956 MB, well over E129's 700 MB ceiling, so it runs on the plain mapped file (`GGML_Q1_LUT=0`), `-c 256 -b 32` | **0.08 tok/s** (12.4 s per token), prompt 0.63 tok/s, coherent text, no kills, peak RSS 629 MB. Every token re-reads most of the file from flash, and LRU caching of a sequential scan keeps nothing. Streaming the LUT layout could reach roughly 0.4 tok/s (about 500 MB per token at 260 MB/s); not built. |
| E136 | **Moon-Beam (`mb-q2-h60.gguf`, 2.75 GB, the TV's published file).** Ported the TV's pruned-head loader for `bailingmoe3` (output.weight holds 60,000 vocabulary rows, `output_ids` maps them) | **`mmap failed: Cannot allocate memory`.** `probes/vmprobe.c` finds a per-process limit on address space, not RAM: **largest single mapping 1,728 MB as root, 1,344 MB as `mobile`** (the web server's user). It's also the total: holding a 1,200 MB file mapping leaves 512 MB more. A 2.6 GB file can't be mapped at all. |
| E137 | **Fewer experts so the file fits** (`tools/moe_slice.py`: keep the PER most-used experts of each of the 8 router groups, chosen from the E89 calibration trace; experts, router rows and router biases sliced, group order kept) | Routing coverage on a different text (the E84 trace): **40 experts keep 58.6 % of uses, 32 keep 50.8 %.** Mac perplexity on `eval-en.txt`, first 6 chunks (every run turns infinite from chunk 7, where a token outside the 60K head appears; the TV used a patched perplexity tool for this): **all 128 experts 7.98, 40 experts 17.94 (+125 %), 32 experts 26.58 (+233 %).** On the iPad the 40-expert file runs at **2.70 tok/s, but the text is broken** ("Tides is also an old name for moon…"). Both files deleted. |

**And the 4B as a server.** Twice the Bonsai-4B web server died while idle, once after about 30 minutes and once 2 minutes after loading. The second time iOS wrote a `SystemMemoryReset` report (02:00:07): *"User reclaimable memory dropped below the limit. User reclaimable current: 27 %. User reclaimable minimum: 55 %"*, with `smolchat` the largest process at 576 MB. This is a separate watchdog from Jetsam's per-process kills. **So the 4B works for a session but not as an always-on server here.** The 1.7B server (about 450-500 MB once warm) ran for hours without it, and is back as the server.

**What would work:** an expert pager that never maps the expert tensors. Load the non-expert tensors (about 470 MB) normally, and before each `MUL_MAT_ID` read the experts the router picked into a fixed pool of slots with `pread`, then run the stock kernel on the pool with remapped ids. On the TV, E89-E103 did the same with mapped overlays, which this address-space limit rules out. An estimate from E84's routing data: about 45 % of a token's experts were used by the previous token, so a pool of about 250 MB would still miss about 100 of 184 experts per token. At 0.77 MB each that's about 78 MB, or 0.3 s from flash, plus about 0.37 s of compute measured on the 40-expert file: **about 1.5 tok/s**, or roughly 2-2.5 with the TV's cache-aware routing bias (E90: misses down to about 10 %, perplexity +4 %). Built in §21.8.

### 21.8 The expert pager: Moon-Beam-8B-A1B on the iPad (2026-11-10)

**Goal (your call):** "build the expert pager and try to go 8-9B again."

**The result: Moon-Beam, a 7.9 B mixture of experts, runs on this 1 GB iPad at 2.4-2.8 tok/s on longer answers, using about 600 MB.** It answers all three test questions correctly and writes coherent stories. That's 30× dense Bonsai-8B (0.08 tok/s, E135), and the first 8B-class model that's usable here.

**Built:**
- **Experts last.** `tools/gguf_experts_last.py` rewrites the GGUF so all 69 `*_exps` tensors come after everything else. The bytes are unchanged; tensors and their infos are reordered. For `mb-q2-h60` the non-expert part is the first 474 MB of 2,753 MB, and each expert (gate, up and down together) is 0.774 MB.
- **Partial mapping.** With `GGML_MOE_PAGER=1`, llama.cpp maps only that first part (`llama_mmap_limit_next`, set in the loader). Expert tensors past the mapping go to the pager.
- **Slot pools.** The pager (`ggml-cpu/moe-pager-ios.inc`) gives each expert tensor a pool of `GGML_MOE_POOL` slots per layer, shared by gate, up and down, and points the tensor at it.
- **Paged `MUL_MAT_ID`.** It asks for the experts its rows use, in groups no larger than the pool. Missing experts go into least-recently-used slots by `pread` from 4 reader threads, with `F_NOCACHE`. The stock kernel then reads each expert from its slot. When generating, the gate op also queues the up and down reads at once.
- **Cache-aware routing.** `GGML_MOE_BIAS` ports the TV's E87 to the argsort that picks experts.
- **Pre-gating prefetch** (the TV's E103). `GGML_MOE_PREFETCH=1` and `GGML_MOE_PREFETCH_K` apply the next layer's router, with the model's full decision (sigmoid, selection bias, group limit, top 8), to the current router input.

| # | Experiment | Result |
|---|---|---|
| E138 | **Exactness** (Mac, `eval-en.txt`, c = 128, 6 chunks) | Plain file and pager give the same perplexity: **7.9836 batched**, at pools 12 and 8 (pool 8 with 64-token batches forces several groups per op), and `GGML_MOE_VERIFY` is clean. One token at a time it's 7.9548 at every pool size with no bias, with or without prefetch. |
| E139 | **Pool size and routing bias** (Mac, one token at a time, the same 768 tokens) | **Misses with no bias:** pool 8 53.4 %, 10 49.0 %, 12 45.1 %, 14 41.5 %, 16 38.2 %. **Bias costs** (PPL, misses): 0.03 at pool 12: 8.06, 37.8 %; **0.03 at pool 14: 7.98 (+0.3 %, inside the ±0.3 % noise floor of E86), 34.1 %**; 0.05: 8.23-8.29 (+4 %); 0.1: 8.28-8.45 (+4-6 %); 0.2: 8.54-9.64 (+7-21 %). The bias costs more here than on the TV. |
| E140 | **iPad, pool 8** (142 MB of slots), `-c 512 -b 64`, repeat penalty 1.15 over 256 | **Runs, and all three answers are right:** "The capital of France is Paris." / a correct two-sentence answer on tides / "The first question was 'What is the capital of France?'". **1.59-1.88 tok/s**, prompt 1.9-3.1, **peak RSS 495 MB**, lowest available 325 MB. 73.7 % misses including prompts, 7.2 GB read in 48 s. Reading through the page cache instead (`GGML_MOE_NOCACHE=0`): **0.69-0.94 tok/s**, half the speed, probably because cached expert pages push out the mapped non-expert weights. |
| E141 | **iPad, pool 16** (285 MB of slots) | **Kernel panic** about 35 s in: `vm_map_delete(...): attempt to remove permanent VM map entry` in `kernel_task` (`panic-full-2026-11-10-103908`). The iPad rebooted, which removes the jailbreak, so it had to be run again. Expected peak was about 640 MB, under E129's 700 MB ceiling, so this isn't a plain out-of-memory kill. Cause unknown; pools 12 and 14 ran many times without it. |
| E142 | **Pool 12 and 14** (short answers / a 220-token story) | Pool 12: 1.71-2.11 tok/s short, **2.37** long, peak 561-575 MB. Pool 14: **2.42-2.47** long (five runs), peak 591-596 MB, lowest available 293 MB. Bias 0.1 at pool 12: 1.9-2.6 short, but the recall answer degrades ("The capital of France."). |
| E143 | **Where a token goes** (`GGML_OPPROF`, pool 14, 220 tokens) | About 0.46 s per token: **waiting on flash ~0.22 s**, Q2_K expert matmuls ~0.10, Q4_K non-expert matmuls ~0.08, the rest ~0.05. Each token reads about 70 MB. **Random-read speed** (`probes/rrbench.c`, uncached, expert region): 258 KB reads 151 / 255 / **272** / 265 MB/s at 1 / 2 / 4 / 8 threads; 774 KB reads 192 / 275 / 279 / 281. With 4 readers, reading each expert in one piece would gain almost nothing. |
| E144 | **Pre-gating prefetch** | Mac, pool 14: top-2 / 4 / 6 / 8 predictions **97 / 94 / 89 / 80 % used**, extra reads +0.4 / +3 / +10 / +26 %; misses a token waits on 41.5 → 33.7 / 25.8 / 18.3 / 13.7 %. **iPad: no faster.** Pool 12 top-8: 2.24 vs 2.37 (two runs each). Pool 14 top-4 vs off, alternated three times: **2.40 / 2.42 / 2.43 vs 2.43 / 2.47 / 2.44**. The flash is busy most of the token anyway, so moving reads earlier doesn't shorten it. Off by default. Single runs vary by up to ±10 % (one plain run gave 2.21), so only alternated repeats count. |
| E145 | **Bias 0.03 at pool 14** (the free point from E139) | 220-token prompt: the story ends on its own at 141 tokens, coherent, at **2.76 tok/s** (misses 38.2 %, from 45.4 %). Three-question test with `--no-think`: all right, 1.75 (cold) / 2.42 / 2.23 tok/s, prompt 3.2-3.6, peak RSS 594 MB. Without `--no-think` the model writes a visible reasoning block first: the template thinks unless the prompt ends in `<think></think>`. |

**Where it stands:**

| Moon-Beam-8B-A1B on the iPad | Long answers | Short answers | Peak RSS | Quality |
|---|---|---|---|---|
| Plain mapping | can't load (E136) | | | |
| 40 of 128 experts (E137) | 2.70 tok/s | | | broken |
| **Pager, pool 14, bias 0.03 (recommended)** | **2.4-2.8 tok/s** | 1.8-2.4 | **~600 MB** | all three right; PPL within noise of the full model |

**To use it:** `python3 tools/gguf_experts_last.py mb-q2-h60.gguf mb-q2-h60-xl.gguf`, copy it over, then:

```sh
GGML_MOE_PAGER=1 GGML_MOE_POOL=14 GGML_MOE_BIAS=0.03 ./smolchat -m mb-q2-h60-xl.gguf -c 512 -b 64 --no-think --repeat-penalty 1.15 --repeat-last 256
```

Like the 4B, it's for a chat session, not the always-on server: at about 600 MB it's above the 576 MB at which iOS's `SystemMemoryReset` watchdog reset the 4B server (§21.7). The 1.7B is still the server.

**What would make it faster:** fewer bytes per token, since the flash is the limit. The options are a bigger pool (pool 16 is where the panic happened), more bias (quality drops fast past 0.03), or a smaller expert format that this CPU runs fast. Overlapping reads with compute (E144) doesn't help once the flash is saturated.

**What went wrong:** my first `gguf_experts_last.py` reordered only the data, not the tensor infos, and llama.cpp rejected the file (`tensor ... has offset ..., expected ...`). The first pool/bias sweep ran under zsh, where `set -- $cfg` doesn't split words, so the bias runs were really bias 0; they were rerun under bash. The pool-16 panic cost a re-jailbreak.

### 21.9 Moon-Beam round 2: 3.4 tok/s (2026-11-10)

**Goal (your call):** "make it faster, push moon-beam past 5 tok/s you need to innovate."

**The result: 3.37 tok/s on the 220-token story prompt, up from 2.76, at +0.7 % perplexity on the full eval text against the unpaged model. That's short of 5 tok/s.** Per token it still reads about 53 MB from flash at about 266 MB/s, so reads alone cost about 0.2 s, which by itself caps it near 5 tok/s. Getting past 5 means either cutting reads, which costs quality, or overlapping reads with compute almost perfectly. The second iPad kernel panic came from trying to cut compute by wiring weights in RAM (E150).

**Built:**
- **Priority read queue.** Reads are ordered by (layer, the tensor's position in the layer: gate, up, down), and prefetches for layers already passed are dropped as stale. E144's queue was first come, first served, so prefetches for layer L+1 sat in front of layer L's up and down reads. The prefetch is split into `mpi_pregate_one`, with `GGML_MOE_PREFETCH_AHEAD` (layers ahead, default 1) and `GGML_MOE_PREFETCH_INFLIGHT` (cap, 48).
- **A lookup-table kernel for Q2_K experts.** `tools/gguf_q2lut.py` repacks each expert of the experts-last file, in place and at the same size, into groups of 16 rows × 256 columns: the fp16 scales, the 4-bit sub-block scales and mins, then the 2-bit codes regrouped so that one byte holds two 4-bit pair codes of one row. It also adds the key `ipad.expert_layout = "q2lut"`. For each pair of activations the kernel builds a 16-entry int8 table of q0·a0 + q1·a1, with one scale per 256 columns, so one `vqtbl1q` gives 16 rows' partial sums of two weights each, with no multiplies. The Q2_K min term uses int16 sub-block sums of the activations. Only the pager reads this layout; to stock llama.cpp the experts are scrambled Q2_K. `GGML_MOE_Q2LUT_REF=1` falls back to the stock kernel.
- **Probes** (`ipad-air/probes/`): `rrspin.c` (flash reads while both cores compute), `membw.c` (read bandwidth, mapped vs anonymous memory), `q4kbench.c` (raw Q4_K × Q8_K dot speed on the model's own weights).
- **`perplexity` on the pruned head.** Tokens outside the 60K-token output head score -inf; they are now skipped and counted (`skipped N tokens`), so the full `eval-en.txt` gives a finite perplexity (42 tokens skipped at `-c 512`).

| # | Experiment | Result |
|---|---|---|
| E146 | **Priority queue, prefetch top-4** (`mb-q2-h60-xl.gguf`, pool 14, bias 0.03, story prompt, alternated twice) | Prefetch off **2.77 / 2.79** tok/s (misses 36.8 %, 41 s waiting); top-4, 1 layer ahead **2.94 / 2.94** (23.6 %, 31 s); 2 layers ahead 2.90 / 2.92 (22.4 %). The first prefetch gain on the iPad (E144's first-come queue gained nothing). Mac, 768 tokens: perplexity 7.98 off, 7.91 / 7.96 with prefetch (the prefetch changes which experts are cached, and so the bias's choices). **Readers aren't CPU-starved:** `rrspin` reads 265 MB/s with 4 uncached readers alone and 266 with both cores spinning NEON FMA (they lose 16 %, 22.2 → 18.6 Gflop/s); raising reader priority changes nothing. |
| E147 | **Q2_K lookup-table kernel** (`mb-q2-h60-lut.gguf`) | Mac, 768 tokens, one at a time, bias 0: **8.0134** vs 7.9548 for the stock kernel (+0.7 %); in 64-token batches 7.9926 vs 7.9836. On the full text (E151): **+0.1 %**. iPad, pool 14, bias 0.03, top-4: **3.04 / 3.08** tok/s (from 2.94), peak RSS 593-601 MB. **Prefetch sweep:** top-4 1 ahead 3.06 (misses 24.4 %), top-6 3.24 (17.8 %), **top-8 3.37** (12.9 %, 9.96 GB read, 19.8 s waiting), top-6 2 ahead 3.31, top-8 2 ahead 3.23 (more reads wasted). Repeats of top-8: 3.35 / 3.36 / 3.38 / 3.39. **More bias at top-8:** 0.05 **3.60**, 0.1 **3.84** tok/s. |
| E148 | **Q4_K dot without dotprod** (`ggml_vec_dot_q4_K_q8_K`: 4 products per int16 lane, scales applied with `vmlal_n_s16`, one reduction per block, 8 independent chains) | Bit-identical output (SmolLM2-360M reply identical, 14.0 tok/s either way). Inside Moon-Beam it's **slower: 518 µs per call vs 421-454** for the stock path. Off by default now (`-DGGML_IPAD_Q4K_ALT`). |
| E149 | **Where a 3.35 tok/s token goes** (`GGML_OPPROF`, 149 tokens) | About 0.30 s per token: `MUL_MAT_ID` 190 ms, of which about 130 ms is waiting on flash; **Q4_K non-expert matmuls 109 ms (469 µs per call)**; the f32 router matmul 14 ms (620 µs per call, where its size suggests about 50); gated delta-net 11 ms. **The Q4_K matmuls run at half the CPU's speed:** `q4kbench` runs the same kernel on the model's own weights at **9.0 G weights/s** warm but **3.05 G/s on the first pass**, when it page-faults; in the model it gets about 4.5. Memory bandwidth isn't it: `membw` reads a file mapping at **10.75 GB/s** and anonymous memory at 10.85. **Hypothesis (unconfirmed):** under memory pressure iOS keeps clearing the page-table entries of the mapped non-expert weights, and every token soft-faults its working set back in (about 75K faults of about 1.5 µs). Actual page-ins are only about 35 MB per run (`vm_stat`), so the pages stay in RAM; only their mappings go. |
| E150 | **Wiring the hot weights** (`LLAMA_MLOCK_HOT=1`: `mlock` of the mapped weights every token reads, about 340 MB, next to the pool-14 slots) | **Kernel panic**, the same signature as E141: `vm_map_delete(...): attempt to remove permanent VM map entry` in `kernel_task` (`panic-full-2026-11-10-140154`). The iPad rebooted and lost the jailbreak again. Two panics, one cause: this kernel panics once too much memory can't be reclaimed (wired, or a pool larger than 14). The code stays, behind the flag, with a warning; **don't use it, and don't use pool 16.** |
| E151 | **Quality on the full text** (Mac, `eval-en.txt`, `-c 512`, one token at a time, pool 14, 42 tokens skipped) | See the table below. Mac, reads per token against pool size (bias 0.03, top-8 prefetch): pool 14 57 MB, 20 43, 24 36, 28 31, 32 26, 40 19. Halving the reads would take a pool of about 28 (about 500 MB of slots), which this 1 GB device can't spare. |

**Quality against reads (E151):**

| Moon-Beam setting | Perplexity | vs the stock kernel | Flash read |
|---|---|---|---|
| xl file, top-8, no bias | 12.79 | | 100 % |
| LUT file, top-8, no bias | 12.80 | +0.1 % | 100 % |
| **LUT, top-8, bias 0.03 (recommended)** | **12.88** | **+0.7 %** | **71 %** |
| LUT, top-8, bias 0.05 | 12.94 | +1.2 % | 60 % |
| LUT, top-7, bias 0.03 | 13.02 | +1.8 % | 62 % |
| LUT, top-6, no bias | 13.33 | +4.2 % | 70 % |
| LUT, top-6, bias 0.03 | 13.48 | +5.4 % | 53 % |

Bias is the cheaper way to cut reads: bias 0.05 at top-8 reads less than top-6 without bias, at under a third of the quality cost. The short 768-token eval (E139) overstated the bias's cost. Top-k is set with `--override-kv bailingmoe3.expert_used_count=int:6` plus `GGML_MOE_USED=6` for the prefetch; `smolchat` has no flag for it yet.

**Where it stands:**

| Moon-Beam-8B-A1B on the iPad | Story prompt (tok/s) | Peak RSS | Perplexity vs unpaged |
|---|---|---|---|
| §21.8: xl file, pool 14, bias 0.03 | 2.76 | ~600 MB | +0.7 % |
| + priority-queue prefetch, top-4 | 2.94 | ~600 MB | |
| + LUT kernel | 3.04-3.08 | 593-601 MB | +0.7 % |
| **+ top-8 prefetch (recommended)** | **3.35-3.39** | **~600-615 MB** | **+0.7 %** |
| same, bias 0.05 | 3.60 | | +1.2 % |
| same, bias 0.1 | 3.84 | | (+4-6 % on the short eval) |

**To use it:** `python3 tools/gguf_q2lut.py mb-q2-h60-xl.gguf mb-q2-h60-lut.gguf`, copy it over, then:

```sh
GGML_MOE_PAGER=1 GGML_MOE_POOL=14 GGML_MOE_BIAS=0.03 GGML_MOE_PREFETCH=1 ./smolchat -m mb-q2-h60-lut.gguf -c 512 -b 64 --no-think --repeat-penalty 1.15 --repeat-last 256
```

**What's left for 5 tok/s:** the reads (about 0.2 s per token) and the compute (about 0.17 s outside the waits) each need to shrink, and overlap better. Options: (1) trade quality: bias 0.05 (+1.2 %) or top-7 (+1.8 %), measured on the Mac; (2) if E149's hypothesis holds, stop the soft faults without wiring memory, for example by copying the hot non-expert weights into anonymous memory or making them smaller; that would recover about 60 ms of Q4_K time and most of the router's; (3) more precise LUT tables (int8 tables cost +0.1 %, so there's room to try coarser ones too).

**What went wrong:** E144 concluded that prefetch can't help because the flash is saturated. The real problem was the queue order: prefetches blocked urgent reads. The Q4_K kernel was benchmarked in isolation first and looked promising; in the model it lost. `LLAMA_MLOCK_HOT` cost a second panic and re-jailbreak; after E141 the hazard of non-reclaimable memory should have ruled it out.

### 21.10 Moon-Beam round 3: the cache policy, and what 5 tok/s costs (2026-11-10)

**Goal (your call):** "keep going" on getting Moon-Beam past 5 tok/s.

**The result: 3.5 tok/s with no quality cost (a frequency-based cache policy), and 4.9-5.1 tok/s with 4 experts per token instead of 8, at +27 % perplexity.** Nothing lossless gets close to 5: the compute left is about 190 ms per token, the flash about 200 ms, and both kernels already run at 75-90 % of what this CPU does on the bare kernel. Choosing a variable number of experts per token (adaptive routing) is no better than a fixed count at the same average.

**Built:**
- **Fault counter in the op profiler.** `GGML_OPPROF=2` adds page faults per call for each op, from `task_info(TASK_EVENTS_INFO)`.
- **Expert trace.** `GGML_MOE_TRACE=file` writes each layer's experts per call, for cache-policy simulation offline.
- **Decayed-frequency eviction.** `GGML_MOE_LFU=a`: each use adds 1 to an expert's score, which decays by `a` per layer call; the slot with the lowest score is evicted (empty slots first). The default stays least recently used.
- **`smolchat --experts N`** overrides `<arch>.expert_used_count` and sets `GGML_MOE_USED` for the prefetch.
- **Adaptive expert count.** `GGML_MOE_TOPP=p` (with `GGML_MOE_KMIN`, default 4), in the argsort hook: keep the selected experts, best first, until they hold a share p of the top 8's routing weight (unbiased sigmoid probabilities: the selection bias and the cache bonus are subtracted), and give the rest id -1. `GET_ROWS` turns -1 into weight 0, so the weight normalization rescales over the kept experts, and `MUL_MAT_ID` already skips -1.

| # | Experiment | Result |
|---|---|---|
| E152 | **Testing E149's soft-fault hypothesis** (`GGML_OPPROF=2`, pool 14, LUT file, top-8 prefetch, 185 tokens) | **Wrong.** Q4_K matmuls take **2.2 faults per call** (about 3 µs of 439), `MUL_MAT_ID` 6.6, the router 1.5. E149's "half speed" came from a wrong weight count: about 695 M non-expert Q4_K weights per token in 99 ms is **about 7 G weights/s**, against `q4kbench`'s 9 warm. The two runs straight after a deploy read 2.50 and 2.36 tok/s, with Q4_K at 698 µs per call and `q4kbench` at 7.4 G/s; 20 minutes later they were back to 3.25-3.35 and 9.1. |
| E153 | **Cache policy** (Mac trace of 3,072 tokens, one at a time, no bias; then the Mac and the iPad) | Simulated miss rates at pool 14: **least recently used 42.0 %** (measured: 41.9 %), decayed frequency 0.5 / 0.8 / 0.9 / **0.95** / 0.98: 41.3 / 39.0 / 38.1 / **37.7** / 37.6 %, the clairvoyant optimum 30.0 %. Mac with bias 0.03 and prefetch: **8.7 % fewer expert loads**, perplexity 18.52 vs 18.69 (noise); with no bias 18.63 vs 18.62, and 37.6 % misses as simulated. **iPad, seed 7, alternated three times: 3.54 / 3.52 / 3.33 tok/s vs 3.31 / 3.34 / 2.90**, about 56-61 MB read per token vs 66. With this policy the text varies from run to run even at a fixed seed: the slot it evicts depends on which reads are still in flight, and the bias then sees a different cache. |
| E154 | **Fewer experts per token** (`--experts N`, decayed frequency 0.95, prefetch, seed 7; perplexity on the Mac, 3,072 tokens, against 18.62 for top-8 with no bias) | See the table below. **The three questions:** right at top-6, top-5 and top-4. Top-4's story is fluent but drifts (it changes a character's gender mid-scene). |
| E155 | **Adaptive expert count** (`GGML_MOE_TOPP`, same settings) | Perplexity at the average expert count, against fixed top-k interpolated to the same count: P 0.9: 6.49 experts, **19.13** (fixed ~19.23); P 0.85: 5.80, **19.40** (~19.70); P 0.8: 5.22, **20.21** (~20.31; KMIN 3: 20.31); P 0.7: 4.40, **22.40** (~22.38). At most 1.5 % better, inside the noise. iPad: P 0.8 4.12-4.35 tok/s, P 0.7 4.49 (4.61 experts): speed follows the average count too. Moon-Beam's routing weight is spread fairly evenly over its top experts, so dropping the small ones costs as much as dropping fixed ranks. |
| E156 | **An SSH drop mid-sweep** | No panic and no Jetsam report, but the sweep's SSH sessions were closed, and `longbench.sh` had started smolchat in the background, so **two Moon-Beam runs ended up running at once** (about 600 MB each in a 1 GB device). They were killed by hand within a minute. The bench now kills any leftover Moon-Beam run before starting and stops its own run if its session dies. |

**Speed against quality (E154):**

| Experts per token | Bias | iPad tok/s (story prompt) | Mac perplexity | vs top-8 |
|---|---|---|---|---|
| **8 (recommended)** | 0.03 | **3.47-3.60** | 18.52 | ±0 |
| 8 | 0.05 | 3.66-3.76 | (12.94 vs 12.79 on the full text, E151) | +1.2 % |
| 7 | 0.03 / 0.05 | 3.73-3.86 / 4.02-4.04 | 18.97 / 18.70 | +0.5-1.9 % |
| 6 | 0.03 / 0.05 | 4.23-4.26 / 4.32 | 19.49 / 19.33 | +4-5 % |
| 5 | 0.03 / 0.05 | 4.54 / 4.52-4.62 | 20.54 | +10 % |
| **4** | 0.03 | **4.92-5.12** | 23.60 (bias 0: 23.41) | **+27 %** |

**To use it:**

```sh
# best quality, about 3.5 tok/s
GGML_MOE_PAGER=1 GGML_MOE_POOL=14 GGML_MOE_BIAS=0.03 GGML_MOE_PREFETCH=1 GGML_MOE_LFU=0.95 ./smolchat -m mb-q2-h60-lut.gguf -c 512 -b 64 --no-think --repeat-penalty 1.15 --repeat-last 256
# faster: add --experts 7 (about 3.9 tok/s, +1-2 %) or --experts 6 (about 4.3, +4-5 %); --experts 4 reaches 5 tok/s at +27 %
```

**What's left:** the lossless options are small: the router (14 ms per token where its size suggests a few), a better Q4_K kernel (at most about 20 ms), and a cache policy closer to the optimum (30 % misses vs 37.7 %). Even all of them together would leave this model short of 5 tok/s at full quality on this iPad. A bigger expert pool is the only large lever, and memory rules it out (pool 16 panicked, E141).

**What went wrong:** E149's soft-fault hypothesis rested on a miscounted weight total; measuring faults directly took one run. Two orphaned runs at once on a 1 GB device was a real panic risk; the bench should have guarded against a dropped session from the start. A sweep loop again lost its arguments, this time because an `ssh` inside a here-document read the rest of the script as its input.

### 21.11 A 16B model on the iPad: Ling-mini-2.0 at 2.6-4.5 tok/s (2026-11-11)

**Goal (your call):** "let's try to push for the 16B models at 2-5 tok/sec".

**The result: Ling-mini-2.0 (16.3 B parameters, 256 experts per layer, 1.4 B active) runs on the 1 GB iPad at 2.56 tok/s with unchanged routing, 3.3 with a small routing bias (+2.6 % perplexity), 4.4-4.6 with a strong one and 5.2 with 6 experts per token**, in 556-576 MB. That's twice Moon-Beam's size, and as fast or faster, because its experts are small (1.03 MB) and its non-expert part is (285 MB resident against Moon-Beam's ~470 MB). It isn't a better model: perplexity on the same text is 13.30 against Moon-Beam's 12.89 (+3 %), as the TV found (§20).

**Built:**
- **`bailingmoe2` pruned head**, ported from `bailingmoe3` (§21.8): with `output_ids` present, `token_embd` is a lazy mapping and the 60K-row head's logits are scattered into the full vocabulary.
- **A model build that never stores the model on the Mac** (it had 2.7 GB free; the TV recipe started from the 9.9 GB Q4_K_M). `tools/gguf_remote.py` reads a GGUF header and tensors over HTTP range requests. `tools/gguf_fetch_subset.py` takes only the non-expert tensors of bartowski's Q4_K_M (0.7 GB), which are requantized to Q4_K with his imatrix and get the 60K head (466 MB). `tools/gguf_moe_assemble.py` writes the final file strictly in order (header, non-experts, then the experts, fetched 10 ranges at a time from mradermacher's imatrix Q2_K_S and repacked for the q2lut kernel on the fly) to stdout, which goes over SSH into the iPad. All of it: `tools/lm-ipad.sh`. The experts are now quantized from BF16 (the TV's were requantized from Q4_K); one layer's `ffn_down_exps` is Q4_K in that file and runs on the stock kernel.
- **`tinyppl`** (`ipad-air/tinyppl.cpp`): perplexity on the device. llama.cpp's `common/` needs `std::filesystem`, which iOS 12 doesn't have, so this is 90 lines on `llama.h` that score like `llama-perplexity` (second half of each chunk) but decode one token at a time, so cache-aware routing sees the same pool states as in chat. Targets outside a pruned head are counted and skipped. Checked on the Mac: Moon-Beam, 6 chunks, no bias, **18.6295, identical to `llama-perplexity`** (E153).

| # | Experiment | Result |
|---|---|---|
| E157 | **Build and copy** (`tools/lm-ipad.sh`) | 279 tensors, 57 expert tensors last, 56 repacked; **5,549,932,192 bytes, md5 identical on the device**. Non-experts 2.5 min (one stream, ~4.6 MB/s), experts 8.5 min (10 streams, 9.9 MB/s), Mac disk peak about 1.4 GB. To make room on the 16 GB iPad (1.1 GB free), the Moon-Beam files, Bonsai-8B and Qwen3.5-0.8B were deleted from it (your call; all have identical copies on the Mac): Moon-Beam and Ling don't fit on it together. |
| E158 | **First run** (pool 14, LFU 0.95, prefetch, bias 0.3 as on the TV, top-8, story prompt, seed 7) | **4.55 tok/s, prompt 2.78 tok/s, 12.0 % misses, peak RSS 557 MB.** The story is coherent (one typo, "adrif"). |
| E159 | **Speed sweep** (same, story prompt) | Bias 0.3: **4.56 / 4.43** tok/s (11.8 / 13.6 % misses); 0.2: 3.96; 0.1: 3.33; **0: 2.56** (21.8 %, 59 s of 96 waiting on flash); bias 0.3 with `--experts 6`: **5.21**. Peak RSS 556-576 MB in every run. |
| E160 | **Quality on the iPad** (`tinyppl`, eval-en, 2 chunks of 512 = 506 scored tokens, 0.78 % of targets outside the head; same pager settings) | See the table below. Only 506 tokens, so differences under ~10 % are noise: bias 0.2 scored worse than 0.3. Decoding the eval text ran at 2.56 (bias 0) to 5.68 tok/s (0.3, top-6). |

**Speed against quality (E159, E160):**

| Bias | Experts | Story tok/s | Perplexity (iPad, 506 tokens) | vs unbiased top-8 |
|---|---|---|---|---|
| **0** | 8 | **2.56** | 13.30 | ±0 |
| **0.1** | 8 | **3.33** | 13.65 | +2.6 % |
| 0.2 | 8 | 3.96 | 15.68 | +18 % (noise; see 0.3) |
| **0.3** | 8 | **4.43-4.56** | 14.44 | +8.5 % |
| 0 | 6 | (3.43 on the eval text) | 14.38 | +8.1 % |
| 0.3 | 6 | 5.21 | 15.55 | +17 % |

For comparison, Moon-Beam on the same 2 chunks (Mac, same settings, no bias): 12.89, at 3.5 tok/s on the iPad (§21.10).

**To use it:**

```sh
# best quality, about 2.6 tok/s
GGML_MOE_PAGER=1 GGML_MOE_POOL=14 GGML_MOE_PREFETCH=1 GGML_MOE_LFU=0.95 ./smolchat -m lm2-q2-h60-lut.gguf -c 512 -b 64 --repeat-penalty 1.15 --repeat-last 256
# GGML_MOE_BIAS=0.1: about 3.3 tok/s at +2.6 %; GGML_MOE_BIAS=0.3: about 4.5 tok/s at roughly +9-18 %; add --experts 6 for 5.2
```

No `--no-think`: Ling-mini-2.0 isn't a reasoning model, and its template (`bailing2`) already sets "detailed thinking off".

**What's left:** a bigger pool would help this model most (256 experts per layer, 14 in RAM), but pool 16 panicked the iPad with Moon-Beam (E141), and Ling's RSS at pool 14 is only ~25 MB lower. Perplexity over more of the text would pin down the bias cost between 0.1 and 0.3. As on the TV, the bigger model is not the better one per byte here; Moon-Beam stays the default.

**What went wrong:** `python3 -I` drops the script's directory from the import path, so the new tools first failed to import their helper. A header parser kept memory-mapped slices where it needed bytes. Deleting the old model files on the iPad was refused by the session's safety check until you approved it. And llama.cpp's own perplexity tool can't be built for iOS 12 at all.