# NanoChat Autoresearch

Training scripts and evaluation results for the NanoChat solutions compared in the
blog post. Every solution is a single self-contained training script trained for
**5 minutes on a single Modal B200 GPU** and scored by **validation
bits-per-byte (BPB), lower is better**. Each was evaluated over **10 random
seeds**.

## `solutions/`

Model training scripts. The Val BPB column is the mean validation BPB across
the 10 seeds (also noted in each script's docstring):

| File | Description | Val BPB |
|------|-------------|--------:|
| `vanilla_transformer.py`     | A plain transformer baseline.                                | 1.0587 |
| `optimized_from_vanilla.py`  | Our autoresearch run optimizing from the vanilla transformer. | 0.9344 |
| `optimized_from_karpathy.py` | Our best solution, optimizing from Karpathy's baseline.       | 0.9109 |

`lib.py` is the shared harness (tokenizer, dataloader, BPB evaluation, time
budget) that every solution imports. It is required to run any of the scripts.

## Reference solutions

We also benchmark against three external solutions. We do not include their
code; see the original sources:

- Karpathy seed: https://github.com/karpathy/autoresearch/blob/master/train.py
- Karpathy's optimized solution: https://github.com/karpathy/autoresearch/blob/438a26e/train.py
- Community best: https://www.ensue-network.ai/lab/autoresearch?run=results%2Fforge--final-lr-frac-0-05-0-02-with-wsd-sqrt--3f0b52

## `results/`

Evaluation results aggregated from all 6 solutions × 10 seeds (the three
reference solutions above were re-run on our harness to produce comparable
numbers):

- `val_bpb.csv` — final validation BPB per solution per seed
  (`label, seed, val_bpb`).

`label` values: `vanilla_transformer`, `optimized_from_vanilla`,
`optimized_from_karpathy` (our solutions, with code in `solutions/`), plus
`karpathy_seed`, `karpathys_best`, `community_best` (the reference solutions
linked above).

## License

This subdirectory is covered by the repository's top-level Apache-2.0
[`LICENSE`](../LICENSE).

The training scripts build on Andrej Karpathy's
[nanochat](https://github.com/karpathy/nanochat), which is MIT-licensed. The
upstream copyright and permission notice is preserved verbatim at
[`LICENSE-nanochat`](LICENSE-nanochat), per MIT's requirements.