# NanoGPT Speedrun (Track 1)

NanoGPT-speedrun Track-1 solutions discovered by Recursive's automated AI-research system:
train GPT-2-small tier model to ≤ 3.28 FineWeb validation loss on 8×H100, as fast as possible
(benchmark: [KellerJordan/modded-nanogpt](https://github.com/KellerJordan/modded-nanogpt)).

This bundle contains two solutions.

| Path | Contents |
|---|---|
| [`from_best/`](from_best/) | Best solution: branched from leaderboard record **#83** ("Sign Trick on Bigram Embed", 79.7 s) and reached **77.3 s**. |
| [`from_unoptimized/`](from_unoptimized/) | Solution discovered from a weak ~15-minute baseline, sharing no lineage with recent records, reaching **≈ 185 s**. |

## From Best

### Result

| Metric | Value |
|---|---|
| Mean val loss | **3.2789** (13 runs, min 3.2766 / max 3.2819) |
| Validity | one-sided t-test, **p(mean < 3.28) = 0.0078** (< 0.01) |
| Mean training time | **77.34 s** (8×H100) |
| Steps | 1398 |

### Baseline (#83, same hardware)

Rule 4 requires beating the prior record on the **same hardware**, so we re-ran #83 ("Sign Trick on
Bigram Embed") on the same 8×H100 Modal setup as from-best. Both pass mean val < 3.28, and from-best
is **3.3 s faster on identical hardware** (80.61 s → 77.34 s):

| | val loss | p(mean<3.28) | training time | n |
|---|---|---|---|---|
| #83 — official record | — | — | 79.7 s | — |
| #83 — our same-hardware baseline | 3.2785 | 0.0047 | 80.61 s | 10 |
| **from-best** (ours) | 3.2789 | 0.0078 | **77.34 s** | 13 |

Logs in `from_best/baseline_logs/`.

### Changes vs Record #83

1. **FP8 attention projections** — QKV + output projections in `float8_e4m3` on the forward pass
   (bf16 backward, bf16 at eval); #83 used FP8 only in the LM head.
2. **Annealed SGLD/Langevin exploration noise** in the NorMuon optimizer update (per-row-RMS-scaled,
   warmed up then annealed to 0 by the cooldown).
3. **Cautious (sign-agreement) Adam** on the bigram / value-embedding banks.
4. **Leaner fused ReLU²-MLP Triton kernel** — the forward stores only `post = relu(pre)²`; the
   backward reconstructs `relu(pre) = √post` in-kernel, dropping a full (M, N) HBM round-trip.
5. Schedule / architecture retunes: fewer paired-head layers, embedding tied throughout, and
   step-count changes.

### Files

- `from_best/train_gpt.py`, `from_best/triton_kernels.py` — the solution.
- `from_best/reproducibility_logs/` — full reproducible run logs plus `statistics.md`
  (per-run table, mean/std/min/max, one-sided t-test p-value).
- `from_best/baseline_logs/` — the #83 record re-run on the same hardware (run logs + `statistics.md`),
  for the Rule-4 comparison.
- `from_best/analyze_sweep.py` — self-contained, scipy-free verifier (regularized incomplete beta
  for the t-distribution).

## From Unoptimized

### Result

| Metric | Value |
|---|---|
| Mean val loss | **3.2778** (10 runs, min 3.2757 / max 3.2792) |
| Validity | one-sided t-test, **p(mean < 3.28) = 3.6e-05** (≪ 0.01) |
| Mean training time | **185.4 s** (8×H100) |
| Steps | 682 |

### Approach

The [`from_unoptimized/`](from_unoptimized/) solution uses a different mechanism mix.

1. **Stitched-stream attention** — consecutive micro-batch rows are stitched into one sequence and
   run under a single document-causal + sliding-window FlexAttention, recovering cross-chunk
   left-context fully differentiably at near block-sparse cost.
2. **Per-layer sliding-window pyramid** — most blocks use a small local window; a few interleaved
   blocks get a wide "regional" window for long-range signal.
3. **Bigram-hash input embeddings** with per-order learnable gates, trained with periodic local-SGD
   parameter averaging.
4. **Narrow 4-head attention** (head_dim 128, attn_dim 512 < 768) — fewer attention FLOPs per step.
5. **FP8 MLP GEMMs** (e4m3, training-only forward; bf16 backward, bf16 at eval) plus a **fused
   softcapped cross-entropy** Triton kernel that avoids materializing the [M, V] logits in fp32.
6. **Cross-layer-diff blend**, a hotter softmax scale, and **Muon gradient-accumulation
   sub-stepping** (each micro-batch drives its own full Muon update).

Notably, it reaches ~185 s without FP8 attention, merged-QKV, or dropping layers. Attention is native
PyTorch **FlexAttention** (no external FlashAttention-3 kernel).

### Files

- `from_unoptimized/train.py`, `from_unoptimized/triton_kernels.py` — the solution.
- `from_unoptimized/reproducibility_logs/` — full reproducible run logs plus `statistics.md`
  (per-run table, mean/std/min/max, one-sided t-test p-value).
- `from_unoptimized/analyze_sweep.py` — self-contained, scipy-free verifier.

## Reproducing the Statistics

From this directory:

```bash
python3 from_best/analyze_sweep.py
# runs   n=13  val_loss=3.2789±0.0014  t=-2.815 p(mean<3.28)=0.007803  time=77.34±0.20s
python3 from_unoptimized/analyze_sweep.py
# runs   n=10  val_loss=3.2778±0.0010  t=-6.880 p(mean<3.28)=3.612e-05  time=185.42±0.27s
```

Use `--logs PATH` to analyze another directory of run logs.

## Environment and Kernel

The `from_best` runs were logged on **Modal**, 8× NVIDIA H100 80GB (PyTorch 2.10.0+cu128, Triton
3.6.0). Training times are measured on this hardware, official leaderboard validation box numbers
pending access to PrimeIntellect compute.

`from_best` uses the `varunneal/flash-attention-3` kernel rather than the default
`kernels-community/flash-attn3`, for compatibility with this build.

## License

This subdirectory is covered by the repository's top-level Apache-2.0
[`LICENSE`](../LICENSE).

The training code in `from_best/` and `from_unoptimized/` is derived from
[KellerJordan/modded-nanogpt](https://github.com/KellerJordan/modded-nanogpt),
which is MIT-licensed. The upstream copyright and permission notice is
preserved verbatim at [`LICENSE-modded-nanogpt`](LICENSE-modded-nanogpt), per
MIT's requirements.