# RV32 inside Qwen3.5

`chipformer/rv/` compiles a RISC-V core into stock Qwen3.5 Gated DeltaNet + SwiGLU layers with
the same q35 compiler as the CHIP-8 machine (`chipformer/q35/ir.py`, `prims.py`, `emit.py`).
P3 (branch `rv-p3b`) is first, then P2 (branch `rv-p2`), P1 (branch `rv-p1`) and the P0
feasibility study (branch `rv-p0`).

## P3: the RV32 computer inside Qwen3.5-4B (a `calc` tool)

### What it is
- **rv-q4p2:** a stock text-only `Qwen3_5ForCausalLM`.
  - Layers: the 11 chip layers of the P2 core plus the bridge first (`core.build(text=...)`),
    then Qwen3.5-4B's 32 layers.
  - Hidden 3968; 478,656 tokens (Qwen's plus 230,324 chip tokens appended as special tokens);
    18.7 GB in bf16.
  - Built by `python -m chipformer.rv.merge BASE --out DIR` (needs about 31 GB of RAM).
- **The bridge** (in the circuit):
  - Qwen's tool-call tokens after `<tool_call>` are captured into a text buffer that the core
    reads as memory at 0x8000: 16 bytes per token, up to 80 tokens per call.
  - `</tool_call>` starts a run (GO). Each byte the program prints is echoed as the matching Qwen
    token, so Qwen reads the response as ordinary text; HALT is echoed as a newline.
  - The circuit owns exactly those positions (owner, tau and mu). Everywhere else Qwen predicts.
- **The firmware:** `env/calc/calc.c`, built with `-DBRIDGE` as `tests/data/rv/calc-bridge.elf`.
  - It parks between calls. It takes the call's first parameter, evaluates it on P2's digit file
    and prints `\n<tool_response>\nVALUE\n</tool_response>\n<think>\n\n</think>\n`.
  - Values have up to 62 digits.
  - Integers: `+ - * // % ** gcd()` and parentheses.
  - Decimals: `+ - *`, printed with a point and no trailing zeros.
  - calc v2 (`2db7827`): `/` is exact rational arithmetic throughout (reduced fractions `p/q`).
  - Error messages:
    - `error (too complex: split it)` when the 7 bignum registers run out;
    - `error (too long: split it)` for a call longer than the 80-token capture window.
- **Decoding** (`rv/chat.py` `RvChat`):
  - Chip tokens run only the chip layers and are invisible to Qwen's layers.
  - The snapshot (firmware, RAM, registers) goes into the chip layers' states after the prompt.
  - Every circuit token is checked in lockstep against the reference machine (`bridge.Session`).

### The chip layers on the Triton executor (`908fdd4`)
- `RvChat(executor=True)` runs the chip layers as the circuit itself (`tritonexec.Executor2`: the
  IR's arithmetic in fp32, one CUDA graph per token) instead of their HF modules.
- After the chip layers, a text token's residual is Qwen's embedding row plus the 23 kept dims
  (output code, tau, mu); the exit clears every other chip dim and the ballast. So `hc` is
  rebuilt from those and handed to Qwen's layers unchanged.
- **Checked** (`rv/execcheck.py`, 6 records including two mul_div ones):
  - "verify" mode runs both paths. The HF chip view's residual equals the executor's bit for
    bit at every position.
  - Executor-only decoding gives the records' tokens exactly.
- **Speed (L4):** about 250 chip tokens/s on the HF path, 3,200 on the executor (13x).
  - A calc call (1.3k to 3k chip tokens) now takes about 0.5 to 1 s.
  - Text tokens (about 68 ms each at batch 1) are most of the time.
  - mul_div problems: median 585 s per problem on the HF path, 242 s on the executor.

### Batching (`rvbatch.py`, `evalarith --batch N`)
- Each conversation keeps its own executor (batch 1, its own CUDA graph) for its chip bursts.
- The text tokens of all live rows go through Qwen's layers in one batched step, on q35's
  per-row-position cache (`batch._Rows`). Finished rows are refilled from the queue.
- **Invariance** (`rv/batchcheck.py --alone`, A10G, B = 8):
  - Each of 6 records (two of them mul_div) decoded alone in the 8-row decoder gives the same
    tokens as all 6 decoded together, so the results do not depend on the batch's other rows.
  - Not bit-identical to batch-1 `generate`: Qwen's GEMMs round differently at M = 1 and
    M = B, and the static KV cache with a mask differs from the dynamic one. Five of 6
    records still matched the L4 batch-1 records exactly; the sixth differs from token 8,390.
  - The circuit is exact either way: 0 lock errors, 0 head mismatches.
- **Throughput:**
  - On the same 6 mixed problems: 128 problems/h at batch 1 vs 199/h at batch 8. Six
    problems are limited by the longest.
  - Per text step: 62 ms for 1 row, about 90 ms for 8.
  - Full sets at batch 8 are in the calc v2 section below.
  - The rows' chip bursts run one after another, about a third of a batched run's wall time; a
    batched executor (Executor2 takes B rows) is the next speedup.
- **Base Qwen** (pytool, base_nudge): `--batch` uses q35's `BatchDecoder` (compiled steps). It
  is not token-identical to the batch-1 HF `generate` path: 1 of 6 pytool texts matched, with
  the same accuracy, 4/6 both ways.

### Checks
- **Text fidelity** (`rv/textcheck.py`, 16 prompts):
  - Mean KL of the merge vs base: 6.1e-4. The noise floor between base's own sdpa and eager
    kernels: 5.1e-4.
  - Greedy text identical on 10/16 prompts; the tokenizer is identical (4/4).
- **IR replay** (`rv/replay.py`): every circuit-owned position of an eval record re-run in the
  IR evaluator. All clean (0 wrong predictions, 0 owner mismatches):
  - arith: 40/40 records (89,453 owned positions);
  - DM: 6 records from each of 4 sets;
  - mul_div: 5 records, 4 of them from the executor path (up to 24 calls, 78k owned positions);
  - calc v2: 6 batched records.
- **Lockstep:** 0 lock errors and 0 head mismatches in every rv1 run (head: the stock lm_head's
  argmax equals the circuit's token at every owned text position).

### Gate results (n = 40, Qwen budget 4096 tokens; exact rational match)
calc v1 (fractions only as the final value), L4, batch 1. rv1 = the merge with the machine
inside; pytool = base Qwen3.5-4B with the same tool, answered by the host (`calc.expected`);
base_nudge = base Qwen with no tool.

| Set | rv1 | pytool | base_nudge |
|---|---|---|---|
| arith (evals' seeded set) | 39/40 | 39/40 | 30/40 |
| dmx_mul_big | 39/40 | 40/40 | 22/40 |
| dmx_add_or_sub_big | 39/40 | 39/40 | 35/40 |
| dmx_div_big | 38/40 | 39/40 | 29/40 |
| dmx_add_sub_multiple_longer | 40/40 | 40/40 | 39/40 |
| dmx_mul_div_multiple_longer | 20/40 | 19/40 | 36/40 |

- The DM base_nudge runs used q35's `BatchDecoder` (A10G, batch 16). The arith base_nudge run
  used batch-1 HF `generate`.
- **rv1's misses are Qwen's, not the machine's.**
  - Every result the machine returned was exact (lockstep and replay clean).
  - Misses: one re-subtraction after a correct result (arith); a dropped sign (add_or_sub,
    which pytool misses too); one 44-call loop (mul_big); two in div_big. In both div_big misses
    the machine had returned the right fraction; Qwen then answered a different number once and
    left no final answer the other time.
- **mul_div: the tool was the problem.**
  - 610 of the 1,040 calc calls returned `error`, because v1 allowed a fraction only as the final
    value (`1/2*1/2`, `(-27/4)/(-27)`). Qwen then retried in loops: 15 of the 20 misses made
    20+ calls.
  - pytool used the same semantics and loops the same way.
  - calc v2 fixes this. Re-run on v2, Qwen's same 1,083 calls (mul_div and div_big) give 1,020
    values, 0 of them wrong; 41 errors (decimals mixed with `/`); 22 too complex.

### calc v2 (rationals) re-run (A10G, rv1 at batch 8, pytool at batch 16)
- pytool's host here is the firmware itself on a reference bridge Session (`--host bridge`).
  Its responses are byte for byte what rv1's machine answers: the same capture window, error
  messages and echo tokens.

| Set | rv1 | pytool |
|---|---|---|
| dmx_mul_div_multiple_longer | **40/40** (204 calls) | 39/40 (209 calls) |
| dmx_div_big | 39/40 | 38/40 |

- rv1's one div_big miss is a 44-call loop on a result the machine returned correctly.
- IR replay of 6 v2 batched records: clean.
- Throughput (A10G, batch 8): 109 problems/h on mul_div and 270/h on div_big. The L4 at batch
  1 managed 14.5/h on mul_div (v1, executor) and 80/h on div_big (v1, HF chip path).
- Base Qwen's batched runs (`BatchDecoder`, fixed batch 16) don't depend on the batch's other
  rows either: 12/12 prompts decoded alone matched the full batch.

### Not done in P3
- The 80-token capture window was not raised. Its thermometer weights 8·2^m overflow bf16
  past 80 after the emitter's 2^36 fold, so a wider window needs a two-level counter.
- Text decoding is still the HF modules at batch 1 or B, about 68 ms per token on an L4. Two
  fallbacks are installed instead of the optimized kernels: torch's linear attention, as
  `flash-linear-attention` is missing, and the `causal_conv1d` reference.
- 0.8B was skipped (user decision). It has 1 value head per key head, and the core needs 2.
- MicroPython has run on the standalone core and executor (P2), not yet inside the merged
  model; it needs a bridge build (P4).

## P2: RV32IM + Zba, a software stack (MicroPython, calc)

### What changed (ISA-visible parts first, commit `77fa2ae`)
- **Memory map v2:**
  - RAM: 32 KB at 0. It holds .data, TLS, .bss, the heap, and the stack below 0x8000.
  - DROM: 128 KB of read-only data at 0x20000 (.rodata; loads only).
  - ROM: 256 KB of code at 0x40000.
  - The linker scripts are `env/rom.ld` and `env/qemu.ld`.
- **ROM word:** one pre-decoded 4-byte word per instruction (`ref.rom_word`):
  - Layout: uop | S0, RD slot | S1-3, RS1 slot | S4-6, RS2 slot | S7-9.
  - Every format fits, with static facts folded in: branch targets are word offsets with a
    bad-target bit, JAL targets are absolute, AUIPC's upper bits are precomputed, and the CSR
    stubs are uops.
  - It decodes in one MLP stage. The two value heads of a ROM key head now hold different
    words, so 256 KB fits in the same 2 sublayers that held 128 KB.
  - Stores take their data register through a third regfile read port (the RD slot), so I-type
    and S-type share one immediate layout.
- **DIV / DIVU / REM / REMU** in hardware.
  - The record is `QS(k0)`, then `QD(k << 4 | q_k)` for k = k0..7 (the hex digits of
    |A| / |B|, most significant first), then the result's value record.
  - k0 = 7 - dA + dB (from the operands' nibble counts) skips digits that must be 0.
  - Random operands average 4.1 tokens per DIV.
  - DIV by 0 and the overflow case follow RISC-V.
  - C builds use `-march=rv32im_zba` with picolibc's rv32im libc.
- **Zba:** sh1add / sh2add / sh3add.
  - MicroPython measured -4% with Zba+Zbb, and Zba was almost all of it.
  - Zbb is not added.
- **The digit file:** 8 registers of 128 decimal digits, outside the address space and part of
  the snapshot. The custom-0 / custom-1 ops (`ref.py` has the exact semantics):
  - `BMAC`: `D[i+off] = (ovw ? 0 : D[i+off]) + S[i]*m + c`. This one op covers add, mul-row,
    copy, scale and zero.
  - `BMSB`: `D[i+off] -= S[i]*m + b`. This covers sub and the div-digit step.
  - `BLD`: loads decimal digits from a RAM byte buffer.
  - `BSCAN`: the length of the ASCII digit run at an address.
  - `BGET`: reads one digit into a register.
  - `BPRD`: prints one digit as a BYTE token. Printing uses a sled of BPRD instructions, one
    token per digit.
  - All operands come in one register: `d | s<<3 | ovw<<6 | n<<7 | off<<13 | m<<20`. The
    common words fit a one-token value.
  - A row op is a `BD(digit | carry << 4)` token per digit, then rd's value (the final carry).
- **Tokens:**
  - New families: QS (16), QD (128) and BD (256), plus the control SC.
  - The vocabulary is now 230,068 rows plus PAD.

### The circuit (commit `3c33d0f`)
- **Size:**
  - **11 layers**, the same as P1.
  - 2,679 dims and 99 heads (at most 16 per sublayer).
  - 42,012 MLP units, at most 8,149 per sublayer out of 9,216.
  - **1,040,595 nonzero weights**; the emitted standalone hidden size is 2816.
  - With unlimited capacity it is still 11 layers. The depth is the dependency chain (DIV's
    compare signs at L10, the exit at L11), so list scheduling (`ir.Graph.reschedule`) cannot
    remove a layer here.

| Layer | Attention (key heads) | MLP (units) |
|---|---|---|
| L1 | pc, count, j, v (4) | thermometers (306) |
| L2 | — | PC word, link, digits-so-far nd (1,376) |
| L3-L4 | ROM 16 + 16 (256 KB) | decode (4,138) |
| L5 | registers x3, quotient heads x4 (7) | adder s1, MUL / REM products, DIV signs, digit-file rows (7,876) |
| L6 | digit file: 2 read ports x 4 (8) | adder s2, shifter, logic, \|A\|>>4s, Q x \|B\| and m x \|B\| products (6,621) |
| L7-L8 | RAM 16 + 16 (32 KB) | multiplier s3-s4, DIV product splits and compare columns, loads, Zba, branch target (3,751 / 7,923) |
| L9 | DROM 16 (128 KB) | multiplier s5, DROM bytes to bits, DIV lookahead, digit / carry (8,149) |
| L10 | — | the 15 DIV compare signs, rd value select (1,074) |
| L11 | — | the output code (798) |

- **DIV**, at the position computing digit k (s = 7 - k):
  - q_k = #{m in 1..15 : D >= m|B|}, where D = (|A| >> 4s) - 16 Q |B| and Q is the digits
    so far.
  - Each of the 15 tests is the sign (bit 39) of D - m|B| mod 2^40. It is built from
    un-normalized nibble-product columns in 3 MLP stages: column split, lookahead, sign.
  - The column offsets are solved so that every column stays in [0, 255].
  - |B| = (B xor sb) + sb, with the +1 folded into nibble 0 (range 0..16).
  - |A| >> 4s is a mux with the ~A + 1 correction.
  - The quotient heads (L5: 4 key heads x 2 value heads, one per digit, one row each) store
    each digit at its QD token in one copy per alignment. One-hot reads gated by nd return
    the digits so far, right-aligned.
  - A first version read 8 rows of one head at once. rv-tiny's fla chunk kernel then left
    fp32 residues (2e-8) on c-crc32p: the multi-hot query's 1/sqrt(8) is not exact, and the
    rows are rewritten often. The IR's read emulation reads rows directly, so it cannot
    see this. Every read is one-hot or 2-hot (the ROM's) again.
  - REM is A -+ |Q| B mod 2^32 through the multiplier: Q replaces A as the product input, and
    the column forms change.
- **DROM:** byte tables read at L9, like the code ROM. Bits come out of the L9 MLP; the lane and
  sign select happen in the value select.
- **Digit file:** port 1 reads row j (the record index); port 2 reads row j + off or k. Writes go
  to both ports' copies at port 2's row from t-1.

### Gate results
- **Compliance:** riscv-tests **rv32ui (38), rv32um (all 8, including div / divu / rem / remu)
  and rv32uzba (3) are green.**
  - Teacher-forced in the IR, with the HF-kernel read emulation on traces under 1,500 tokens.
  - Also free-running on the GPU.
  - DIV fuzz (edge values, all operand sizes, signs, rd = x0) is exact in the IR. The
    reference's DIV decomposition is checked against the ISA on 80k random cases.
  - The local suite (test_rv_core, test_rv_ref, test_rv_fuzz) passes: 228 tests, 14 skipped.
- **Programs:** all 61 ELFs give the same exit code and stdout under the reference model as
  under qemu-riscv32. Free-running on the L4 (the executor generates every token, which must
  equal the reference trace):

| Program | Instructions | Tokens | tok/instr | P1 instructions |
|---|---|---|---|---|
| MicroPython s5 (18-line snippet) | 492,526 | 610,544 | 1.24 | — |
| MicroPython s1 (`print(2**100)`) | 39,888 | 50,150 | 1.26 | — |
| fib(25) | 2,114,629 | 2,174,558 | 1.03 | 2,117,133 |
| qsort 1000 | 199,157 | 240,176 | 1.21 | 256,975 |
| sieve below 10^4 | 123,535 | 176,225 | 1.43 | 126,897 |
| mini-gmp 20x20 | 14,767 | 22,671 | 1.54 | 15,967 |
| crc32 printf | 4,109 | 6,283 | 1.53 | 9,998 |
| crc32, sort, strops, sieve(2000) + 7 riscv-tests (div, rem, Zba) | 44,722 | | | |

  - The two free-running batches total **3.03M instructions**.
  - Fuzz (now with DIV, Zba and DROM loads): 64 programs, 152,848 instructions.
  - A bigger fuzz run: 128 programs at 300 loop iterations, **2,986,619 instructions,
    exact**.
  - **Altogether P2 free-ran 15.3M instructions exactly**: the programs above, the fuzz
    runs, the 10k calc cases and the v2 runs.
- **MicroPython** (P2.4a, `env/build_mpy.sh`, `env/mpy/`) is rv_tool's best build under 256 KB,
  for this machine:
  - The 3 source patches, 5 hot files at -O3, fastlib, computed goto, the map cache, no GC
    clear, parse chunk 512.
  - rv32im_zba, with the script compiled into the DROM and output through putchar ecalls.
  - Sizes: text 205 KB (ROM), rodata 26 KB (DROM), bss 2 KB, and a GC heap of about 23 KB.
  - s1 and s5 are free-running exact. rv_tool measured 38,964 / 490,092 instructions with
    Zba+Zbb; this build takes 39,888 / 492,526.
- **calc** (P2.4b, `env/calc/`, `chipformer/rv/calc.py`): C firmware on the digit file.
  - It handles + - * / // % ** ^ gcd() and parentheses with Python's semantics, on values up to
    62 digits.
  - It has a fast path for "A op B" and a shunting-yard loop otherwise.
  - Long division estimates each quotient digit from the window's top 3 digits and the divisor's
    top 2 (at most 2 too big), then runs BMSB and an add-back if needed.
  - **10,000 random cases equal Python on the reference machine.**
  - **All 10,000 are also free-running exact on the circuit** (the executor on the L4):
    8,951,798 instructions and 12,298,128 tokens in 19 min.
  - Six expressions are teacher-forced in the IR with the kernel check
    (test_rv_core.test_calc).
  - Tokens per case (from the text in RAM to the end of the print), against CHIP-8 v15 (the
    median of 20; CHIP-8 counts from after its CALL record, so its CALL record, about
    2 tokens per digit plus 10, is added):

| Call | RV32 calc tokens (instructions) | of which digit-file / print tokens | CHIP-8 v15 after CALL | CHIP-8 incl. CALL |
|---|---|---|---|---|
| add 4+3 | 254 (202) | 25 | 30 | ~54 |
| add 12+12 | 303 (210) | 76 | 55 | ~113 |
| sub 8-5 | 289 (218) | 44 | 42 | ~78 |
| mul 2x2 | 340 (283) | 20 | 44 | ~62 |
| mul 4x3 | 377 (296) | 38 | 68 | ~92 |
| mul 8x8 | 520 (352) | 113 | 206 | ~248 |
| mul 12x12 | 684 (402) | 217 | 394 | ~452 |
| div 4/4 | 435 (352) | 23 | 526 | ~552 |
| div 6/2 | 660 (510) | 52 | 570 | ~596 |
| div 12/4 | 882 (618) | 114 | 751 | ~793 |

  - The digit work itself (BD / SC / BYTE / QS / QD tokens) costs about 1 token per digit per
    row, about the same as the CHIP-8 macros.
  - What costs more is the C around it: about 200-250 tokens per call for the dispatch,
    register bookkeeping and the call / return of the bignum routines, against CHIP-8's 22-byte
    BIOS.
  - On random 1-24-digit operands the means are: add 358, sub 432, mul 680, div 1,117,
    mod 1,056, gcd 3,394, pow 1,731, and mixed 2-4-operand expressions 991.
  - Closing the gap would mean writing the dispatcher in assembly, or giving more of the
    control to the digit-file ops.
- **Executor v2** (P2.5; NVIDIA L4, crc32 printf at 1.53 tokens per instruction):

| Path | Batch | Per step | tok/s | instr/s |
|---|---|---|---|---|
| P1 executor (dense weight blocks), P2 circuit | 1 | 1.066 ms | 938 | 614 |
| sparse weights (gather, product, scatter-add), torch.compile + CUDA graph | 1 | 0.261 ms | 3,836 | 2,509 |
| sparse | 8 | 0.427 ms | 18,738 | 12,254 |
| sparse | 32 | 1.132 ms | 28,261 | 18,483 |
| **executor v2: Triton, one fused kernel per sublayer, CUDA graph** | **1** | **0.177 ms** | **5,657** | **3,699** |
| executor v2 | 8 | 0.441 ms | 18,150 | 11,870 |
| executor v2 | 32 | 1.502 ms | 21,309 | 13,936 |

  - **Batch 1 is now 3.7k instructions/s, 1.8x P1's 2.0k**, on a circuit twice the size
    (and 6x the P1 executor on the P2 circuit).
  - At batch 8 the two paths are level; at batch 32 the sparse torch path is ahead (27k
    tok/s).
  - v2's value-head programs (one per vhead and row) are latency-bound and scale badly
    with batch. The next levers are several rows per program, and one persistent kernel
    for the whole step.
  - v2 is checked free-running: the fuzz set (64 programs, 152,848 instructions) and
    MicroPython s1, gmpmul, crc32p, rv32um-div and rv32um-rem (58,880 instructions),
    all exact.

  - The P2 circuit's dense weight blocks are 165 MB per step, against 0.78M nonzeros, so
    batch 1 is bandwidth-bound. Sparse rows cut that about 25x.
  - `tritonexec.Executor2` fuses each sublayer into one Triton kernel under one CUDA graph:
    - MLP: a block of units per program.
    - Heads: one program per value head: projections over the lagged residual, decay, the
      one-row write, the k-hot read with the ballast norm, and the output columns.
    - Plus an apply kernel.
  - It equals the IR on every dim.
- **rv-tiny on the GPU:** a stock `Qwen3_5ForCausalLM`, 2,112M parameters, hidden 2816,
  12 layers, saved on q35-eval in `~/wrv2/dist/rv-tiny-p2`.
  - bf16 with the fla kernels, for rv32um-div, rv32um-rem, rv32uzba-sh2add and c-crc32p: the
    residual before the exit equals the IR on every dim, and free-running `generate`
    reproduces the reference trace. crc32p's 6,282 generated tokens print `crc32=414fa339`.
  - tinycheck now warms up fla's autotuner before each check. On a new shape the autotuner
    reruns kernels that update the recurrent state in place, which corrupted the first
    result.
  - Longer traces (gmpmul, 22k tokens) run out of memory in tinycheck's one-shot prefill:
    the logits alone are 22k x 230k. This is a limit of the check, not of the model.
- **Re-checked after the one-hot quotient heads** (free-running on the final circuit):
  - MicroPython s5 and s1, qsort, crc32p, gmpmul and rv32um div / divu / rem / remu:
    750,680 instructions with executor v2.
  - 2,000 calc cases: 1,773,438 instructions.
  - The fuzz set: 64 programs, 152,848 instructions.
  - All exact.

### Not in P2
- **Zbb:** it measured under 0.5% for MicroPython.
- **RAM:** it stays at 32 KB. That is in the asked 32-64 KB range, and MicroPython's rodata now
  sits in the DROM.
- **The text bridge to Qwen:** that is P3. The calc input sits in RAM (`calc_in`, 0x100) for
  now; P3's capture will write it.
- **calc's C overhead** (above).
- **A persistent single-kernel step:** v2 uses one kernel per sublayer instead.

## P1: the RV32IM core (minus DIV), standalone

### The core
- **ISA:** all of RV32I, plus MUL / MULH / MULHSU / MULHU through a 5-stage multiplier. DIV and REM
  are an illegal-instruction fault; C code uses gcc `-march=rv32i_zmmul`, which sends DIV/REM to
  libgcc. FENCE commits as NT.
- **System:**
  - `ecall` a7 = 1 is putchar: the record is one `BYTE(a0 & 255)` token.
  - `ecall` a7 = 93 is exit (`HALT`).
  - The CSR stubs are pure reads, folded into the ROM as constants: cycle / time / instret (+h)
    read 0, mhartid 0, misa 0x40001100. Any CSR write is illegal.
- **Faults halt and print.** The record is `FLT(cause)`, the PC's 8 hex digits (`HEX`), then
  `HALT`. The causes are illegal, ebreak, misaligned load / store, load / store out of range, bad
  jump (misaligned or outside ROM; static targets are flagged in the ROM image) and unknown ecall.
- **Sizes:**
  - ROM 128 KB in 2 sublayers (32 key heads).
  - RAM 32 KB in 2 sublayers (32 key heads × 2 value heads × 128 rows, words).
  - The address map: RAM at 0 (data, rodata, TLS, bss and heap from 0x100, the stack below
    0x8000), ROM at 0x10000.
- **Tokens:**
  - V / F0 / F1 have 15-bit payloads; J / R have 16 bits; BYTE 8; FLT 3; HEX 4; there are 7
    controls.
  - The vocabulary is **229,667 rows**.
  - The uop byte keeps 64..127 free for the custom-0/1 bignum ops of P2.
- **Circuit:**
  - **11 layers** (the exit is folded into the output sublayer), 1,253 IR dims, 69 heads,
    19,576 MLP units (at most 6,198 per sublayer), **501,396 nonzero weights**.
  - A j counter head (2^-j, tokens of the current record) replaces P0's previous-token head.

| Layer | Attention | MLP |
|---|---|---|
| L1 | pc, count, j | thermometers |
| L2 | — | PC word, link |
| L3-L4 | ROM | field decode, immediate, link |
| L5 | registers | adder s1, logic, shift s1, multiplier s1 (products), store data |
| L6 | — | adder s2, shift s2, multiplier s2 (product nibbles) |
| L7-L8 | RAM | multiplier s3-s4, loads, merge, compares, faults |
| L9 | — | multiplier s5 |
| L10 | — | rd value and size |
| L11 | — | output code + exit |

### Gate results
- **Compliance in the IR:** riscv-tests **rv32ui (38 tests) and rv32um mul / mulh / mulhsu / mulhu,
  all green**, teacher-forced, with HF-kernel read emulation on traces under 1,500 tokens.
  fence_i and ma_data are excluded, and so is rv32um div / rem (no DIV).
- **Other IR tests:**
  - 10 fault programs, one per cause plus variants.
  - The CSR stubs and putchar.
  - ALU / branch / MUL fuzz.
  - The n = 63 refresh.
  - Snapshots at n > 0.
  - C programs: crc32 printing through picolibc printf (13.4k tokens), strops, and the
    **mini-gmp 20×20-digit multiply (16k instructions)**. The last two take 12-44k tokens, so
    they run under `RV_SLOW=1`.
- **Reference vs qemu-riscv32:**
  - 54 ELFs (gcc 13.2; C with picolibc for rv32i, libgcc): exit codes and stdout equal qemu's.
  - The traced machine ends in the ISA emulator's exact register state.
- **C programs verified free-running** (the executor generates every token itself; each must
  equal the reference trace up to HALT):

  | Program | Instructions | Output |
  |---|---|---|
  | fib(25), recursive | 2,117,133 | `fib(25)=75025` |
  | qsort 1000 ints (picolibc) | 256,975 | sorted, hash 72e4eb11 |
  | sieve below 10^4 | 126,897 | `primes below 10000: 1229` |
  | mini-gmp 20×20 multiply | 15,967 | `mul=1219326311370217952249809480012498094790` |
  | crc32, sort, strops, sieve(2000) | 51,459 | |
  | **total** | **2,568,431** | all equal |

  sudoku-easy (8.4M instructions) passes the qemu / reference check but was not free-run.
- **Fuzz** (`chipformer/rv/fuzz.py`): random fault-free RV32IM programs covering every ALU,
  shift and MUL op, all load / store widths through masked addresses, branches, calls, putchar,
  LUI / AUIPC. **64 programs, 1,481,677 instructions free-running, all equal.**
- **In total, 4.05M instructions** are verified free-running.
- **`rv-tiny`** (`python -m chipformer.rv.tinycheck --save dist/rv-tiny`; saved on q35-eval in
  `~/wrv/dist/rv-tiny`):
  - **The model:** a stock `Qwen3_5ForCausalLM`, 960M parameters, hidden 1280, 12 layers (11
    chip + 1 zero full-attention), Qwen3.5-4B's DeltaNet head shapes, vocabulary 229,667 + PAD,
    with tied embeddings. The code dims of each token row score bit agreement.
  - **`firmware.py`:** ELF → `ref.Machine` → `core.snapshot` → the HF cache, after a zero PAD
    prefix, then plain greedy `generate`.
  - **Checks:** the bf16 residual before the exit layer equals the IR on every circuit dim at
    every position, and **free-running `generate` reproduces the reference trace exactly**.
    - On GPU with the fla kernels: 7 programs, including crc32, whose 13.4k generated tokens
      print `crc32=414fa339`.
    - On CPU with the torch kernels: 3 programs.
- **Speed** (NVIDIA L4, crc32 trace at 1.34 tokens per instruction):

  | Path | Per step | tok/s | instr/s |
  |---|---|---|---|
  | executor, batch 1 (torch.compile + CUDA graph) | 0.37 ms | 2,706 | 2,017 |
  | executor, batch 8 | 0.60 ms | 13,367 | 9,963 |
  | executor, batch 32 | 0.85 ms | 37,743 | 28,131 |
  | rv-tiny HF modules, StaticCache + torch.compile (reduce-overhead) | 8.9 ms | 112 | 83 |
  | the same after `fast.compact` (nonzero blocks only) | 1.64 ms | 609 | 454 |

  - **The executor** (`fastexec.py`) now has batch rows and free-running generation (decode plus
    embedding on the GPU, one graph per step).
  - **The stock HF path is not within 2× of the executor.** Compacted and compiled it is 4.4×
    slower at batch 1, and 24× without compaction. The dense layers read 0.96B parameters per
    token, including the tied 229k-row output head. After compaction it is launch-bound: each
    layer runs conv, l2norm, the fla GDN kernel, gated RMSNorm and the MLP.
  - Both rows of the HF path generate the exact reference trace.
- **Tokens per instruction:** 1.0-1.55 on the C programs (1.40 averaged over crc32, sieve, sort
  and strops); 1.19 on the P0.6 small-C mix.

### Not in P1
- DIV/REM in hardware (the P2 micro-code).
- The text I/O bridge to Qwen (P3). The BYTE family stands in for Qwen's byte tokens.
- causal-conv1d (it doesn't build on q35-eval).

# P0: feasibility (branch `rv-p0`)

Everything below is measured on `rv-p0` unless marked *estimate*.

## Verdict

| Criterion (plan P0.8) | Result | |
|---|---|---|
| IR exact | 38 riscv-tests rv32ui + 4 gcc -O2 C programs (up to 31.7k instructions / 44k tokens) + 480 fuzzed ALU ops + branch fuzz, teacher-forced: 0 wrong tokens. Every head read also replayed as the HF torch kernels compute it (2^-3.5 q scale, bf16 cast of o, gated RMSNorm): exact | GO |
| Real HF modules | stock `Qwen3_5ForCausalLM` (bf16, torch reference kernels, CPU), snapshot injected into the cache: residual == IR on every IR dim at every position, chunked prefill and token-by-token decode | GO |
| Depth ≤ 12 (P1), ≤ 32 (P2) | the P0 skeleton is **8 layers**; P1 extrapolates to ~11-13, P2 to ~16-20 (below) | GO |
| Width / text-visible dims ≤ 3072 | 884 IR dims, all chip-only (overlayable); text-visible = output code (19) + mu/tau/ballast until the P3 bridge adds capture state | GO |
| Byte columns exact on fla kernels | **bf16: bit-exact** on an NVIDIA L4 with fla 0.5.2: `chunk_gated_delta_rule` (prefill) and `fused_recurrent_gated_delta_rule` (decode) on 9 programs, every IR dim at every position, including crc32's 4,075 positions (`gpucheck --decode`). **fp32: 0 wrong tokens** on the same 18 runs, but the residual is not bit-exact; see "fp32" below | GO (bf16) |
| Compact executor ≥ 3k instr/s | **3.6k instr/s on an NVIDIA L4** (5,562 tok/s, 0.18 ms per token at crc32's 1.55 tokens per instruction; about 4.7k at the small-C mix's 1.19): the circuit's own executor (`fastexec.py`), torch.compile'd and captured as one CUDA graph, its replays checked exact against the IR | GO |

## Design as built

### Trace format (`spec.py`)
One record per instruction; state changes only at a record's last token, and the PC is implied:
PC = ROM_BASE + 4·(anchor + n), with the anchor the payload of the latest J/R token and n the commits
since (an R refresh is emitted before the instruction when n reaches 63).

| Record | Tokens |
|---|---|
| rd-writing (ALU, LUI, AUIPC, loads) | `F0` · `V F1` · `V V F2` by the value's sign-extended size (15-bit chunks) |
| rd = x0, not-taken branch, FENCE | `NT` |
| taken branch, JAL, JALR | `J(word)`; the link rd = PC+4 is computed in-circuit |
| SW | `ST0`/`ST1` (the RAM value head = address bit 6); the data is rs2 from t-1, in-circuit |
| SB, SH | `V V V SS0/SS1`: the merged word (old bytes merged at the top of the circuit) |
| ECALL, EBREAK, illegal | `HALT` |

Vocabulary: V, F0, F1, J, R × 32,768 + F2 × 4 + 7 controls = **163,851 rows**.

### The commit rule
Every position computes the instruction its *next* token belongs to: at a record's last token
the circuit already works on the next instruction; inside a record it recomputes the current one.
A write takes its **key from t-1** (a conv tap), its **write gate beta from the token at t** (beta =
sigmoid(256 g − 128) with g ∈ {≤0, 1}, exactly 0 or 1), and its **value from token payloads at lags
0..3 or from dims computed at t-1 below the writing layer**. The payload-to-bit wiring needs no
constants (which would break at the sequence start): for "V bit at t-1 when t is F1" the channel is
P = −Vb + isF1 + isV with taps (1, −1), which equals the bit exactly when t is F1 and is ≤ 0
otherwise. Information only flows down through tokens, so nothing writes a value computed above
its own layer at the same position; that is why the PC is implied rather than written.

### Memory
| Table | Layout | Read | Write |
|---|---|---|---|
| ROM (code) | 16 key heads × 2 value heads; row pairs (p, p+64); 8 bytes per instruction **pre-decoded** at load: (uop, rdw, rs1, rs2) in value head 0, imm in value head 1; byte-valued columns | 2-hot query {p, p+64} (effective q scale 2^-4, exact for bytes); the word is selected by z per column block, so no select MLP | snapshot only |
| registers | 2 key heads (rs1, rs2 ports), 32 rows, 109 columns per row (payload sources per bit + the link) | one-hot query; a bit = sum of its columns | key rdw(t-1), beta = F0+F1+F2+J |
| RAM | 16 key heads × 2 value heads × 128 rows × 1 word = 16 KB per sublayer; key head = EA[5:2], value head = EA[6], row = EA[13:7] | one-hot query, z picks the value head | key = address(t-1) AND store(t-1); beta = ST_x/SS_x |
| pc, count, prev | 1 head each | | J/R payload; 2^-n decay counter; token kinds at lags 1-3 |

The ROM is a compiled image, so static facts are folded in (`ref.rom_fields`): rdw = rd only if
the instruction writes rd, AUIPC/LUI are constants, branch and JAL targets are absolute words.
The uop byte has room for the custom-0/custom-1 bignum ops planned for P2.

Address map: **RAM at 0** (.data from 0x100, the stack below 0x4000), **ROM at 0x10000**. RAM at
the bottom makes data and stack pointers 1-token values: 1.42 → 1.19 tokens per instruction on
the small-C mix.

### Datapath (8 layers)
| Layer | Attention | MLP (units) |
|---|---|---|
| L1 | pc, count, prev (3 heads) | n thermometer, record position j (129) |
| L2 | — | PC word w = anchor + n, link w+1, thresholds + carry into bit 6 (1,072) |
| L3 | ROM (16) | field one-hots, immediate bits, gated link (4,660) |
| L4 | registers (2) | adder stage 1 for 4 operand modes (generate / propagate / digit bits per nibble), logic ops, shifter stage 1, store data (5,216) |
| L5 | — | adder stage 2 (carries and result bits in one stage), shifter stage 2 (1,008) |
| L6 | RAM (16) | load lanes / sign extension, SB/SH merge, SLT/SLTU/zero (492) |
| L7 | — | rd value select, its size class, branch decision (190) |
| L8 | — | output code: family id + 15-bit payload (175) |

The 32-bit add is 2 MLP sublayers: nibble sums are free linear forms, then stage 1 thresholds them
per operand mode (with the mode as an AND literal, so no operand mux layer), then stage 2 applies
the carry sources as AND literals on the digit bits.

## Measurements (`python -m chipformer.rv.measure --nonzeros`)
- **Size:** 884 IR dims, 37 heads, 12,942 MLP units; **298,676 nonzero weights** in the 8 chip layers.
  At 4B width the dense chip layers hold 1.08B parameters, almost all zeros. The ROM, RAM and
  registers live in the recurrent state, not in the weights.
- **Capacity per sublayer:** ROM 64 KB (pre-decoded, 8 bytes per instruction), RAM 16 KB (words).
- **Tokens per instruction:**
  - Measured on gcc -O2 C programs (crc32, sieve, sort, strops): **1.40**. Byte stores are 4 tokens
    and sieve is store-heavy.
  - The small-C mix from P0.6 gives **1.19**, against 3-5 for CHIP-8.
- **Payload width:**

| Chunk bits | Value records (1/2/3 tokens) | Tokens per instruction, mix | Vocabulary rows | Embedding at 3072 wide |
|---|---|---|---|---|
| 12 | 20.4k / 4.2k / 2.6k | 1.44 (2-token jumps) | 20.7k | 0.13 GB |
| **15** (built) | 23.6k / 2.2k / 1.4k | **1.19** | 164k | 1.01 GB |
| 16 | 23.7k / 3.5k / — | 1.11 (SB/SH as V+SS) | 459k | 2.82 GB |

  Recommendation: keep 15-bit V/F and make **J/R 16-bit** (+65k rows), since 256 KB ROMs need
  16-bit word indices (MicroPython with float is 229 KB).
- **HF cache snapshot (R11):** injecting the initial states into `DynamicCache` recurrent states
  after a zero (text-token) prefix works for prefill and decode (`hf.py`). Static cache not tried.
- **Speed** (`python -m chipformer.rv.timing --hidden 3072`, batch 1, crc32 trace at 1.55 tokens
  per instruction):

  | Path (NVIDIA L4) | ms/token | tok/s | instr/s |
  |---|---|---|---|
  | HF modules, dense, eager, DynamicCache (4B width: hidden 3072, 16/32 heads, 9216 units) | 11.0 | 91 | 59 |
  | the same after `fast.compact` (nonzero blocks only) | 12.5 | 80 | 52 |
  | circuit executor (`fastexec.py`), eager | 5.1 | 194 | 126 |
  | executor, one CUDA graph per step | 0.53 | 1,898 | 1,226 |
  | **executor, torch.compile + CUDA graph** | **0.18** | **5,562** | **3,594** |

  - **What the executor is.** It runs the IR's arithmetic in fp32 over the circuit's live
    structure:
    - each sublayer's projections as one matmul over the input dims they read;
    - one-hot key writes as row scatters, ≤2-hot reads as top-2 row gathers;
    - decay only on the counter.
  - **How it is checked.** It equals `ir.evaluate` on every dim at every position, eager on CPU and
    on GPU, and its CUDA-graph replays are checked against the IR on 300 positions.
  - **It replaces the HF modules for chip-only steps.** Same arithmetic, not the same kernels.
    The HF rows stay eager and launch-bound here; the speed fork's static-cache CUDA-graph decode
    path (`decode_rows`, about 1.3 ms for a 16-layer chip step on an L40S) was not applied to
    this model.
  - **L40S *estimate*.** By bandwidth alone (864 vs 300 GB/s) the executor row would be about
    16k tok/s on an L40S. These steps are launch- and latency-bound, so the real gain should be
    smaller; it is an upper estimate.
  - **Next.** A persistent single-kernel step (Triton/CUDA) over the 0.3M nonzeros is the next
    step if more is needed.

## GPU numerics (P0.2, NVIDIA L4, fla 0.5.2, `gpucheck --decode`)

Programs: signext, call, memcpy, rv32ui-lb / -sh / -jalr / -sra, c-crc32 (4,075 positions) and
c-strops. Each runs as one chunked prefill (`chunk_gated_delta_rule`) and token by token
(`fused_recurrent_gated_delta_rule`), with the snapshot injected into the cache.

- **bf16** (the merged model's dtype): **bit-exact.** Every IR dim matches at every position in
  all 18 runs. That covers the byte-column 2-hot ROM reads, the sibling value heads' write gates,
  the lag-1 key commits through prefill-then-decode, and the counter's decays through the chunk
  kernel's exp(cumsum).
- **fp32: 0 wrong tokens** in all 18 runs, but 70-400 dims per run are not bit-exact (1-ulp
  noise).
  - **Why.** Without the final bf16 cast, the kernel's q scale 1/sqrt(128) and the chunk kernel's
    exp(cumsum) decays leave about 2^-23 relative error, e.g. a counter read of 0.2500611 for 0.25.
  - **Why the tokens are still right.** The circuit's AND/threshold margins absorb it.
  - **Status.** fp32 decoding is correct, not bit-exact. Making it bit-exact would need power-of-2
    q scales the kernels don't have; bf16 is the target dtype.

## Options measured or costed
- **8-byte granule RAM rows** (the spike fork's alternative): 2× RAM per sublayer, but SW becomes
  4 tokens (+0.27 tokens per instruction on the mix). RAM needs are tiny (P0.6: at most 16 KB
  touched), so word rows stay.
- **ROM density:** pre-decoded 64 KB per sublayer. 256 KB = 4 sublayers, which adds about 3 layers
  of fetch depth. The alternative is raw 4-byte words at 128 KB per sublayer plus one decode
  MLP stage. Pick for P1 against the depth budget.
- **SB/SH cost** is 4 tokens with 15-bit payloads; 2 with 16-bit (V + SS16).

## P1 / P2 extrapolation (*estimates*)
- **P1** (all RV32I, MUL via 5 parallel MLP stages, putchar/exit, 128 KB ROM, 32-64 KB RAM): MUL
  runs beside the adder/RAM path (L4-L8), so it adds no depth. The ROM grows to 2 sublayers (+1),
  RAM to 2-4 sublayers (+1..3) and output select (+1), giving **11-13 layers**.
- **P2** (256 KB ROM, micro-coded DIV, CARRY/instret, custom bignum ops): **16-20 layers**.

## Gaps
- **causal-conv1d is untested.** It doesn't build on q35-eval: its CUDA toolkit is 12.8, while
  torch is built for cu13.0. The conv runs through the torch fallback, as in every run above.
- **Not in the P0 circuit:** MUL*, DIV*/REM*, putchar, faults (misaligned, out of range), text I/O.
- `fence_i` (self-modifying code) and `ma_data` (misaligned access) are skipped, as planned.

## Files
- `chipformer/rv/`:
  - `spec.py`: tokens and records.
  - `ref.py`: the ISA emulator, `rom_fields`, the traced `Machine`.
  - `asm.py`: assembler.
  - `elf.py`: ELF loader.
  - `programs.py`: assembly test programs.
  - `core.py`: the circuit and snapshot.
  - `hf.py`: HF modules plus cache injection.
  - `measure.py`, `gpucheck.py` (GPU numerics), `timing.py`.
  - `fastexec.py`: the compact circuit executor (the P0.5 prototype).
  - `env/`: riscv_test.h, link scripts, crt0, C programs, `build_tests.sh` (WSL).
- `tests/`:
  - `test_rv_ref.py`: ref == qemu-riscv32 on 44 ELFs; Machine == ISA.
  - `test_rv_core.py`: the circuit, teacher-forced.
  - `test_rv_hf.py`.
  - `data/rv/`: the built ELFs and qemu results.
- **IR additions** (`q35/ir.py`):
  - from store-v1: sibling value heads, per-value-head beta, `into`.
  - new here: head `init` (snapshot) states, byte-valued and decay levels (snapshots at n > 0, tested mid-run), `kernel_check` (HF kernel
    emulation), beta g ≤ 0.
- `q35/prims.py`: `thead`.