# Results

All numbers below are under one consistent three-tier accounting (head excluded
from core — see `src/budget.py`). Earlier mixed-accounting runs are archived in
`runs/_archive_old_accounting/` and should not be cited.

**Status (2026-07-21): validated end to end.** PLE beats baseline (2 seeds), the
gain survives 4-bit PTQ (2 seeds), on-chip bandwidth confirms the flash table is
nearly free, and the complete 28.9M-parameter model now generates coherent text
on an ESP32-S3 N16R8. The C runtime generates text at **~9.5 tok/s end to end**
(what a viewer sees; includes serial output), equivalent to 102.9ms/model step or
9.72 tok/s of pure compute, using an int8-staged output head with int8 activations
(host-validated, val perplexity delta ~0). Lead public numbers with the end-to-end
figure.

## Headline (the deployable config)

Vocab 32768, core-matched across arms at ~559K params (273KB at 4-bit, small
enough for the ESP32-S3's 512KB internal SRAM; the design constraint that sized
it. The polished firmware in fact leaves the core flash-mapped XIP, which
measured fast enough — see the on-chip section), 25M-param PLE table (12MB
flash), d_model=96, 6 layers, ple_dim=128. Two seeds.

| arm | core | total | ppl | vs baseline |
|---|---:|---:|---:|---:|
| `baseline` | 559K | 3.7M | 12.58 | — |
| **`ple`** | 558K | **28.9M** | **11.41** | **+0.098 nats / 9.3% ppl** |
| `fatembed` | 559K | 28.9M | 11.94 | +0.052 nats |

- **PLE beats a same-core, SRAM-fitting baseline by 0.098 nats (2 seeds, ±0.006).
  ~16x the seed noise.** ppl 12.58 -> 11.41.
- **Per-layer injection beats bottom injection by 0.046 nats** (`ple` vs
  `fatembed`, both 2 seeds). At realistic vocab the *where* of injection is worth
  roughly 2x the params-at-the-bottom approach.
- As comparison context (prior independent work, no relation to this codebase):
  DaveBben's esp32-llm ran 260K params on an ESP32-S3. This is ~110x the stored
  parameter count, on a tighter fast-memory budget, with the extra params living
  in flash as a per-token-sparse table.

**What "28.9M params" does and does not mean.** It is 28.9M *stored* parameters:
a 559K dense core (SRAM), a 3.1M output head (streamed sequentially), and a 25M
lookup table (flash, one row per token). It is still a TinyStories-domain model —
better coherence and consistency, not new capability. Quote it as "parameters
resident via a memory-hierarchy split," never as a capability multiple.

## Why vocab matters (and why the small-vocab number is modest)

The same architecture at vocab 4096, core-matched at ~1.5M, 2 seeds:

| arm | ppl | vs baseline |
|---|---:|---:|
| `baseline` | 8.21 | — |
| `ple_notable` | 8.35 | **-0.017 (worse)** |
| `fatembed` | 8.26 | -0.006 (~nothing) |
| `ple` | 8.00 | +0.025 |
| `bigcore` (2x core) | 6.93 | +0.170 |

At vocab 4096 PLE's edge is only +0.025 nats. At vocab 32768 it is +0.098 — **4x
larger.** This is the memory-tiering thesis working: a large vocabulary makes the
table both huge and cheap (more rows, each a sparse per-token lookup), and that
is precisely the regime PLE was designed for. The deployable config lives in the
favourable regime; the small-vocab ablation is a control, not the product.

## What the controls establish

- **The table does the work, not the plumbing.** `ple_notable` (all of PLE's
  per-layer adapters and projection, no lookup table) is *worse* than baseline at
  vocab 4096 (-0.017): the plumbing spends core params on machinery that returns
  nothing unless a table feeds it. The isolated table contribution
  (`ple` - `ple_notable`) is +0.043. So the flash-resident table is the entire
  source of the gain, which is exactly what the ESP32 premise needs.
- **Row-width saturates; vocab-rows do not.** The corrected sweep (`runs/*fix-d*`,
  fixed FFN, vocab 4096) shows the isolated table benefit peaking around a 6M
  table when scaling `ple_dim` (row width): +0.045 (d64) -> +0.094 (d256) ->
  +0.087 (d512). But the deploy config scales the table a different way — more
  *rows* via a 32k vocab — and a 25M table there still clearly pays off (+0.098).
  Widening rows plateaus; adding rows does not, over the range tested.
- **PLE is not free capacity.** `bigcore` (2x the dense core) gets +0.170 at vocab
  4096. PLE recovers ~15% of that at vocab 4096, more at vocab 32768. On a desktop
  you would just buy the core; on an ESP32 the core is fixed silicon and flash is
  abundant, which is the whole point.

## Hardware: bandwidth measured on the N16R8 (2026-07-21)

Measured on the real ESP32-S3 (`firmware/bandwidth_bench`, cycle-accurate timing
via the Xtensa cycle counter). This is what turns the estimated tok/s into a real
one, and it is the number the whole approach rested on.

| measurement | value |
|---|---|
| PSRAM sequential read | 60.7 MB/s |
| internal SRAM sequential read | 240 MB/s |
| flash random-read, 512B row | 20.3 us |
| **per-token TABLE cost** (6 random rows) | **~0.12 ms** |
| **per-token HEAD cost** (1.5MB PSRAM scan) | **~17.3 ms** |
| **bandwidth-only tok/s ceiling** | **~58 tok/s** |

**The core bet is confirmed on silicon.** In the isolated bandwidth benchmark,
the 25M-param flash table costs ~0.7% of the per-token memory time — nearly free,
exactly as designed. The output HEAD (which the baseline pays too) dominates the
memory traffic. This is not yet a measured end-to-end PLE-vs-baseline speed
comparison, so quote the 0.7% specifically as the table's synthetic bandwidth
share rather than total inference overhead.

Caveats: this is a bandwidth-only ceiling, not observed inference throughput.
The first complete scalar port showed that unpacking int4 and doing millions of
scalar float operations dominate well before raw bandwidth does. The 20us
random-read latency is real, not zero — it stays negligible at this ple_dim/layer
count but would grow with a wider table.

## On-chip generation: complete model running (2026-07-21)

The exported model is 14,912,332 bytes and fits in a custom 15,597,568-byte
flash partition with 685,236 bytes spare. The 619KB application occupies a
separate 1MB partition. The portable C runtime matches the exported PyTorch
golden across all 32,768 logits (`max abs diff = 0.00001`) before the same code
is compiled for the device.

On the ESP32-S3, the 1.64MB tied embedding/output head is copied to PSRAM at
boot, the 25M-parameter PLE table stays memory-mapped in flash, and scratch plus
KV cache live in PSRAM. After all allocations, 5,228KB of PSRAM remains free.
Greedy generation from `Once upon a time` produces coherent TinyStories text.
One captured on-device continuation begins:

> Once upon a time, there was a little girl named Lily. She loved to play
> outside in the sunshine. One day, she saw a big tree with a hole in it. She
> was curious and wanted to see what was inside.

| implementation | 200-token result | model-step time |
|---|---:|---:|
| first correct portable port | 0.57 tok/s end to end | 1,757.2 ms |
| PSRAM head + scalar cleanup | 4.61-4.77 tok/s end to end | 193.9 ms |
| exact dot/RoPE/attention cleanup | — | 172.9 ms |
| dual-core exact head | 5.67-6.22 tok/s end to end | 139.4 ms |
| **int8-staged head + int8 activations** | **~9.5 tok/s end to end** | **102.9 ms** |

The int8 head is the current runtime: **9.72 tok/s compute-only** (102.9 ms/step),
a further 1.35x over the exact fp32 dual-core head. The output head is staged as
int8 in PSRAM once at boot (int4 nibbles unpacked once), and activations are
quantized to int8 per token, so each output row is a plain int8xint8 -> int32 dot
with no per-token unpacking. **The int8-activation change was validated on host
val perplexity (delta ~0, see firmware/host_verify/ppl.c) before shipping**, and
on-chip text stays coherent. The scalar fp32 head (139.4ms) remains the exact
baseline; the fp32 host golden still matches PyTorch to 1e-5.

Profile after the int8 head (ms/token, dual-core wall): head 57.6 | attn 25.6 |
ple 8.5 | ffn 6.9 | input 4.4. The head is now **PSRAM-bandwidth-bound, not
compute-bound**: it reads 2.43MB of int8 weights per token, which at 60.7 MB/s is
a ~40ms floor, so only ~17ms is compute. Literal S3 vector-SIMD would cut that
17ms but not the 40ms bandwidth floor (bounded ~15% further gain). The bigger
levers from here are reducing bytes-read (int4 head + SIMD unpack) or a
smaller/factorised output head (a model change) -- not vectorising harder.

The earlier fp32 exact optimizations (139.4ms baseline) were: staging the head
in PSRAM, converting each fp16 group scale once, unpacking both int4 values per
byte, applying a group scale after its dot product, compiling at `-O3`, skipping
7,415 unreachable padded vocabulary rows, computing RoPE values once per token,
caching attention scores, and splitting independent output rows across both
LX7 cores.

Historical profile of that earlier exact fp32-head path (139.4ms/step, superseded
by the int8 head; the current runtime's profile is the 57.6 | 25.6 | 8.5 | 6.9 |
4.4 breakdown above), averaged over 200 generated tokens (wall-time share; the
head runs on both cores, so its *compute* share is higher — ~80% — than its
wall share):

| stage | ms/token (wall) | wall share |
|---|---:|---:|
| output head (dual-core) | 93.2 | 66.9% |
| attention | 26.4 | 18.9% |
| PLE input + per-layer path | 12.9 | 9.3% |
| FFN | 6.9 | 4.9% |

Explicitly staging the remaining 0.29MB quantized core in PSRAM and norms in
internal RAM saved only 2.0ms/token (1.4%) while adding allocation complexity,
so that experiment was removed from the polished runtime.

There is still bounded exact work—parallel attention, precomputed RoPE
frequencies, and a one-group-specialized head loop—but the profile caps the
entire attention opportunity at 26.4ms and the other items at low single-digit
milliseconds. They are intentionally deferred rather than presented as another
large scalar speedup.

The measured throughput is much lower than the 58 tok/s bandwidth ceiling and
disproves the earlier 20-40 tok/s compute estimate for the naive scalar kernel.
It is nevertheless in the same practical speed range as the prior independent
260K-parameter ESP32 project (comparison context only) while holding roughly
110x as many stored parameters. With int8 activations now shipped and the head
PSRAM-bandwidth-bound, the next large speed step is reducing bytes read (an
int4-in-PSRAM head with SIMD unpack) or a factorized/smaller output head — not
more scalar cleanup and not a question about PLE table bandwidth.

## 4-bit quantization: the gain survives (2026-07-21)

Group-wise symmetric int4 PTQ (group 64), the GGUF-Q4-style format you would flash
(`src/quantize.py`). Every large weight quantized, including the 25M table. Two
seeds, vocab 32768 deploy models.

| arm | fp32 -> 4-bit degradation | 
|---|---|
| `baseline` | +0.079 / +0.088 nats |
| `ple` | +0.055 / +0.061 nats |
| `fatembed` | +0.046 / +0.050 nats |

PLE vs baseline edge: fp32 +0.101/+0.095 -> 4-bit +0.125/+0.121, i.e. **fully
retained (124-128%)**. Read carefully: all arms degrade under 4-bit (~ppl +1);
PLE degrades *less*, because a large redundant lookup table with per-group scales
is inherently more quantization-robust than a small dense model where every weight
is critical. So the part we bet the flash budget on is also the most 4-bit-robust.
**No QAT needed for the headline.**

The final flash artifact uses a tighter group-128 format with ragged rows and
fp16 scales to fit 16MB flash. The exact storage scheme was also validated on
both seeds:

| arm | fp32 -> shipping-format degradation |
|---|---|
| `baseline` | +0.089 / +0.109 nats |
| `ple` | +0.063 / +0.089 nats |
| `fatembed` | +0.056 / +0.061 nats |

PLE's edge is +0.101/+0.095 nats in fp32 and +0.127/+0.115 in the shipping
format: **126%/121% retained**. Thus the exact bytes flashed to the board have
the same two-seed conclusion as the group-64 headline PTQ check.

## Remaining limitations

- **The ESP32-S3's SIMD instructions remain unused.** The shipping runtime
  (~9.5 tok/s end to end) quantizes activations to int8, host-validated, but its
  dot products are still scalar. The head is PSRAM-bandwidth-bound (a ~40ms
  read floor inside its 57.6ms), so SIMD alone buys a bounded ~15%; the real
  levers are reducing bytes read (int4 head + SIMD unpack) or a smaller head.
  The 58 tok/s number remains only a bandwidth ceiling.
- **Domain is TinyStories.** World knowledge, arithmetic, and multi-step reasoning
  remain absent; that ceiling is set by the dense core, not moved by the table.
- **Provenance.** This is independent work. Its actual dependencies are the
  TinyStories dataset (Eldan & Li, Microsoft Research, arXiv:2305.07759) and
  Google's published Gemma Per-Layer Embeddings design (reproduced from the
  `transformers` Gemma implementation and Google's documentation). No code,
  model, checkpoint, or method derives from llama2.c (Karpathy) or from
  DaveBben's esp32-llm; both are prior independent work in the tiny-LM /
  microcontroller space and appear in this document only as comparison context.
  The novel claim here is narrowly: applying Gemma-style Per-Layer Embeddings
  to a microcontroller SRAM/flash hierarchy so a larger stored model fits than
  fast memory allows.

## Next

1. Add interactive serial prompting/tokenization, then dialogue fine-tuning for
   the simple-conversation milestone.
2. Record the on-chip text-generation demo and publish the measured result.
3. Treat an ESP32-S3 SIMD/int8 head as a separate experiment and compare its
   quality and speed against the exact 139.4ms/token baseline.