# Selected Experiment Reports

This is a semi-automatically generated list of experiment issues and reports. It is periodically updated by a script,
and then curated by hand.

This page includes only experiments that have at least one run or report.

## Curated experiment series

- [Agent MoE experiment digest](./agent-moe-experiments.md) — outcomes from small-scale modeling, optimizer, routing, systems, and data tests.

## Published analysis sites

One-off analysis pages published to durable public hosting (see
[Publishing an Analysis Site](../tutorials/publish-analysis-site.md)). The machine-readable list is
[`gs://marin-public/index.json`](https://storage.googleapis.com/marin-public/index.json).

- [DataKit side-by-side](https://storage.googleapis.com/marin-public/held/datakit-sidebyside/2026.07.01/index.html) — cluster examples, side by side (held)
- [Delphi mid-training dossier](https://storage.googleapis.com/marin-public/ahmad/delphi-midtraining/2026.07.01/index.html) — mid-training analysis dossier (ahmad)

## Marin 8B Base

- Tootsie 8B (Main Issue) [![#600](https://img.shields.io/github/issues/detail/state/marin-community/marin/600)](https://github.com/marin-community/marin/issues/600)
    - [GitHub Issue #600](https://github.com/marin-community/marin/issues/600)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/Tootsie-8B---VmlldzoxMTY3MzU3OA)

### Cooldowns

- Try deepening the cooldown of "monumental-jellyfish" (tootsie 8b cooldown 1) to see if it improves SFT [![#898](https://img.shields.io/github/issues/detail/state/marin-community/marin/898)](https://github.com/marin-community/marin/issues/898)
    - [GitHub Issue #898](https://github.com/marin-community/marin/issues/898)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/898-Tootsie-Soft-Raccoon--VmlldzoxMTk3NjUwNg?accessToken=06f87pmmvhdulczenkg3349jxk7e1pwbd4pdci2i8wvyxg9289122gfnckr9ymwc)
- not-quite-so-deep cooldown (Spoonbill) [![#916](https://img.shields.io/github/issues/detail/state/marin-community/marin/916)](https://github.com/marin-community/marin/issues/916)
    - [GitHub Issue #916](https://github.com/marin-community/marin/issues/916)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/916-Tootsie-Hypnotic-Spoonbill--VmlldzoxMjA1NjU2Nw)
    - Conclusion: Exploding logits in deep parts of cooldown can be mitigated by Z-Loss.
- Tootsie Phoenix Cooldown (sensible-starling) [![#977](https://img.shields.io/github/issues/detail/state/marin-community/marin/977)](https://github.com/marin-community/marin/issues/977)
    - [GitHub Issue #977](https://github.com/marin-community/marin/issues/977)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/Tootsie-8B-phoenix-cooldown-starling---VmlldzoxMjQ2MjM5Ng)
    - [WandB Run: tootsie-8b-sensible-starling](https://wandb.ai/marin-community/marin/runs/tootsie-8b-sensible-starling?nw=nwuserdlwh)

## Marin 32B

- Marin 32B [![#1295](https://img.shields.io/github/issues/detail/state/marin-community/marin/1295)](https://github.com/marin-community/marin/issues/1295)
    - [GitHub Issue #1295](https://github.com/marin-community/marin/issues/1295)
    - [WandB Run: Marin 32B run1](https://wandb.ai/marin-community/marin/runs/llama-32b-tootsie-2?nw=nwuserdlwh)
    - [Marin 32B Retrospective](./marin-32b-retro.md)

## Big Runs

- \[EPIC\] Big Runs [![#859](https://img.shields.io/github/issues/detail/state/marin-community/marin/859)](https://github.com/marin-community/marin/issues/859)
    - [GitHub Issue #859](https://github.com/marin-community/marin/issues/859)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/Big-Tootsies--VmlldzoxMTEyOTQ0MA?accessToken=st1rajwy32etqi5rrlm3kuhgqa4ods6fnwsbyk8azjc8ar3eikf4dnz1p2ldz8yx)
- Marin 13B [![#860](https://img.shields.io/github/issues/detail/state/marin-community/marin/860)](https://github.com/marin-community/marin/issues/860)
    - [GitHub Issue #860](https://github.com/marin-community/marin/issues/860)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/Big-Tootsies--VmlldzoxMTEyOTQ0MA?accessToken=st1rajwy32etqi5rrlm3kuhgqa4ods6fnwsbyk8azjc8ar3eikf4dnz1p2ldz8yx)
- Marin 24B [![#861](https://img.shields.io/github/issues/detail/state/marin-community/marin/861)](https://github.com/marin-community/marin/issues/861)
    - [GitHub Issue #861](https://github.com/marin-community/marin/issues/861)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/Big-Tootsies--VmlldzoxMTEyOTQ0MA)
- Marin 70b [![#750](https://img.shields.io/github/issues/detail/state/marin-community/marin/750)](https://github.com/marin-community/marin/issues/750)
    - [GitHub Issue #750](https://github.com/marin-community/marin/issues/750)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/Big-Tootsies--VmlldzoxMTEyOTQ0MA)

## Modeling

- Pick Tokenizer type [![#524](https://img.shields.io/github/issues/detail/state/marin-community/marin/524)](https://github.com/marin-community/marin/issues/524)
    - [GitHub Issue #524](https://github.com/marin-community/marin/issues/524)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/Tokenizer-Comparison--VmlldzoxMDI0Njg3Nw)
    - Conclusion: Llama3 tokenizer is the best.
- Default z-loss? [![#935](https://img.shields.io/github/issues/detail/state/marin-community/marin/935)](https://github.com/marin-community/marin/issues/935)
    - [GitHub Issue #935](https://github.com/marin-community/marin/issues/935)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/ZLoss-vs-Not-1-4B--VmlldzoxMjEzMzA1NA)
    - Conclusion: z-loss seems not harmful. We'll use it.
- Figuring out learning rate schedule! [![#764](https://img.shields.io/github/issues/detail/state/marin-community/marin/764)](https://github.com/marin-community/marin/issues/764)
    - [GitHub Issue #764](https://github.com/marin-community/marin/issues/764)
    - [WandB Report](https://wandb.ai/marin-community/marin-optimizer/reports/Deciding-the-optimal-lr-schedule-which-is-cosine---VmlldzoxMTIxNDk5NA)
    - Conclusion: Cosine is best. High LR is important. WSD isn't terrible.
- Mixture of Experts [![#929](https://img.shields.io/github/issues/detail/state/marin-community/marin/929)](https://github.com/marin-community/marin/issues/929)
    - [GitHub Issue #929](https://github.com/marin-community/marin/issues/929)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/929-MoE--VmlldzoxMjIxMjI0MQ?accessToken=9lec16kiwqm4kg70o6e9zhzlrv7wgiecsd0fhwb6r76mr8vgsy44tr0tejrity6o)
- Hybrid Norm and Input Embedding Norm [![#961](https://img.shields.io/github/issues/detail/state/marin-community/marin/961)](https://github.com/marin-community/marin/issues/961)
    - [GitHub Issue #961](https://github.com/marin-community/marin/issues/961)
    - [WandB Report](https://wandb.ai/marin-community/hybrid-norm/reports/Hybrid-Norm--VmlldzoxMjY2MDgxMA)
- OLMoE replication - MoE vs dense [![#1183](https://img.shields.io/github/issues/detail/state/marin-community/marin/1183)](https://github.com/marin-community/marin/issues/1183)
    - [GitHub Issue #1183](https://github.com/marin-community/marin/issues/1183)
    - [WandB Report](https://api.wandb.ai/links/marin-community/qi3u8nx7)
    - Conclusion: Despite having a lower MFU, MoE outperforms similar sized dense model in both training and evaluation.

## Training and Performance

- INT8 training in Levanter [![#620](https://img.shields.io/github/issues/detail/state/marin-community/marin/620)](https://github.com/marin-community/marin/issues/620)
    - [GitHub Issue #620](https://github.com/marin-community/marin/issues/620)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/620-Int8-Training--VmlldzoxMTQ4NTY4Mg?accessToken=opus2o11hqwefpfjv1ii6sfjduq0jrdfjho5raj6i0lvi41lkv0u1rj5ij4elyz7)
    - Conclusion: Int8 training is much faster on the right hardware, but might lead to worse performance in terms of time-to-loss except in the early stages.
- MuP for scaling laws [![#621](https://img.shields.io/github/issues/detail/state/marin-community/marin/621)](https://github.com/marin-community/marin/issues/621)
    - [GitHub Issue #621](https://github.com/marin-community/marin/issues/621)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/621-MuP--VmlldzoxMTIxMTUzNQ?accessToken=h5qjejzau65v7bab5bau94hu8cltm1q9a0v6tdabocd398wagpnr6rjk4u8yc41a)
    - Conclusion: not worth it compared to our heuristic version.
- Figuring out learning rate schedule! [![#764](https://img.shields.io/github/issues/detail/state/marin-community/marin/764)](https://github.com/marin-community/marin/issues/764)
    - [GitHub Issue #764](https://github.com/marin-community/marin/issues/764)
    - [WandB Report](https://wandb.ai/marin-community/marin-optimizer/reports/Deciding-the-optimal-lr-schedule-which-is-cosine---VmlldzoxMTIxNDk5NA)
- Try out different remat strategies to get the 70b working on fewer slices [![#906](https://img.shields.io/github/issues/detail/state/marin-community/marin/906)](https://github.com/marin-community/marin/issues/906)
    - [GitHub Issue #906](https://github.com/marin-community/marin/issues/906)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/Remat-Strategies--VmlldzoxMTkxNzk3Ng)
    - Conclusion: Substantial performance hit but helpful. Still need to iterate.
- Fantastic Pretraining Optimizers And Where to Find them [![#1290](https://img.shields.io/github/issues/detail/state/marin-community/marin/1290)](https://github.com/marin-community/marin/issues/1290)
    - [GitHub Issue #1290](https://github.com/marin-community/marin/issues/1290)
    - [WandB Report](https://wandb.ai/stanford-mercury/optimizer-scaling/reports/Fantastic-Optimizers-and-Where-to-Find-Them--VmlldzoxMjgzMDU5NQ?accessToken=2ib82ugpvrywe89kndzbdffpjc7rezwlo7jdvz5gc9huc9thwyjs9itpu2nvs8zz)
- Qwen (QK Norm) Speedruns [![#1572](https://img.shields.io/github/issues/detail/state/marin-community/marin/1572)](https://github.com/marin-community/marin/issues/1572)
    - [GitHub Issue #1572](https://github.com/marin-community/marin/issues/1572)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/Qwen-3-speedruns-QK-Norm-Muon---VmlldzoxNDM1NjY5MA)

## Data Experiments

### High Quality Data Ablations

- Ablations on Cooldown for Markdownified Wikipedia [![#845](https://img.shields.io/github/issues/detail/state/marin-community/marin/845)](https://github.com/marin-community/marin/issues/845)
    - [GitHub Issue #845](https://github.com/marin-community/marin/issues/845)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/845-6-Wiki-and-Arxiv-Quality-Ablations--VmlldzoxMTg4MzY2OA)
    - Conclusion: No major improvement compared to control.
- Ablations on Cooldown for Markdownified Arxiv [![#846](https://img.shields.io/github/issues/detail/state/marin-community/marin/846)](https://github.com/marin-community/marin/issues/846)
    - [GitHub Issue #846](https://github.com/marin-community/marin/issues/846)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/845-6-Wiki-and-Arxiv-Quality-Ablations--VmlldzoxMTg4MzY2OA)
    - Conclusion: No major improvement compared to control.
- Ablations on Cooldown for Markdownified StackExchange [![#847](https://img.shields.io/github/issues/detail/state/marin-community/marin/847)](https://github.com/marin-community/marin/issues/847)
    - [GitHub Issue #847](https://github.com/marin-community/marin/issues/847)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/845-6-Wiki-and-Arxiv-Quality-Ablations--VmlldzoxMTg4MzY2OA)
    - Conclusion: No major improvement compared to control.
- Mixture of Formats Training on Wikipedia and Arxiv [![#818](https://img.shields.io/github/issues/detail/state/marin-community/marin/818)](https://github.com/marin-community/marin/issues/818)
    - [GitHub Issue #818](https://github.com/marin-community/marin/issues/818)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/818-Mixture-of-Formats--VmlldzoxMTg4MzU0NA)
    - Conclusion: No major difference observed, switch to @Helw150's annealing setup for evaluations.
- High Quality Many Epochs vs. Low Quality Few Epochs [![#636](https://img.shields.io/github/issues/detail/state/marin-community/marin/636)](https://github.com/marin-community/marin/issues/636)
    - [GitHub Issue #636](https://github.com/marin-community/marin/issues/636)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/High-Quality-Many-Epochs-vs-Lower-quality-fewer-epoch--VmlldzoxMDU2MTI1Mg)
    - Conclusion: There's no data like more data.
- Add MegaMath data and run ablation [![#942](https://img.shields.io/github/issues/detail/state/marin-community/marin/942)](https://github.com/marin-community/marin/issues/942)
    - [GitHub Issue #942](https://github.com/marin-community/marin/issues/942)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/Exp942-MegaMath-Annealing--VmlldzoxMjk0NzcxMg)
- OpenWebMath Crawl Annealing Experiments [![#1167](https://img.shields.io/github/issues/detail/state/marin-community/marin/1167)](https://github.com/marin-community/marin/issues/1167)
    - [GitHub Issue #1167](https://github.com/marin-community/marin/issues/1167)
    - [WandB Report](https://wandb.ai/stanford-mercury/marin/reports/OpenWebMath-Crawl-Ablations--VmlldzoxMzA2MzQ5NQ?accessToken=rsopvcnt8fswcg75c180c1i1101vqd5fdoo0m94rwd4qfgc8hbwdpyn8bguaafk3)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/OpenWebMath-Annealing-Runs--VmlldzoxNDExNDIyMg)
- Fineweb-Edu Crawl Annealing Experiments [![#1168](https://img.shields.io/github/issues/detail/state/marin-community/marin/1168)](https://github.com/marin-community/marin/issues/1168)
    - [GitHub Issue #1168](https://github.com/marin-community/marin/issues/1168)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/FineMath-Annealing-Runs--VmlldzoxMjA4NjI4Nw)

### Data Filtering

- Stack Exchange Quality Classifier [![#596](https://img.shields.io/github/issues/detail/state/marin-community/marin/596)](https://github.com/marin-community/marin/issues/596)
    - [GitHub Issue #596](https://github.com/marin-community/marin/issues/596)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/Quality-Classifier-Comparison--VmlldzoxMDI2MzI1MQ)
    - Conclusion: Seems to lead to better loss than using Reddit ELI5 or OpenHermes.
    - NOTE: this seems like a loose end, we should pursue this further.
- Cascading Quality Filters [![#963](https://img.shields.io/github/issues/detail/state/marin-community/marin/963)](https://github.com/marin-community/marin/issues/963)
    - [GitHub Issue #963](https://github.com/marin-community/marin/issues/963)

### Text Extraction and Formatting

- Compare HTML -> text methods [![#246](https://img.shields.io/github/issues/detail/state/marin-community/marin/246)](https://github.com/marin-community/marin/issues/246)
    - [GitHub Issue #246](https://github.com/marin-community/marin/issues/246)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/246-Web-Extraction-Method-Comparison--Vmlldzo5OTg4MTAw?accessToken=8t7elz382va7ftc1uttuo43ccpej2yg220y33y6i9ek7q81q5xfmp9vzcw7av6ih)
    - Conclusion: some amount of format preservation is helpful for loss on Paloma.
- Wikipedia Training Runs with DOLMA source substitution [![#647](https://img.shields.io/github/issues/detail/state/marin-community/marin/647)](https://github.com/marin-community/marin/issues/647)
    - [GitHub Issue #647](https://github.com/marin-community/marin/issues/647)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/647-Wikipedia-Training-Runs-with-DOLMA-source-substitution--VmlldzoxMDkyNjIxNw)
- Ar5iv Training Runs with DOLMA source substitution [![#648](https://img.shields.io/github/issues/detail/state/marin-community/marin/648)](https://github.com/marin-community/marin/issues/648)
    - [GitHub Issue #648](https://github.com/marin-community/marin/issues/648)
- [Markdownification Processing Report](./markdownified-datasets.md)

## Supervised Fine Tuning

- Reproduce olmov2 SFT [![#606](https://img.shields.io/github/issues/detail/state/marin-community/marin/606)](https://github.com/marin-community/marin/issues/606)
    - [GitHub Issue #606](https://github.com/marin-community/marin/issues/606)
    - [WandB Run: marin_olmo_tulu_sft_v3-acca67](https://wandb.ai/marin-community/marin/runs/marin_olmo_tulu_sft_v3-acca67)
- Create Mixture from All SFT Datasets [![#804](https://img.shields.io/github/issues/detail/state/marin-community/marin/804)](https://github.com/marin-community/marin/issues/804)
    - [GitHub Issue #804](https://github.com/marin-community/marin/issues/804)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/Llama-3-1-vs-Tootsie-SFT-on-Total-Mixture--VmlldzoxMTY3MzcxNQ?accessToken=4cmsxu0stayrjyce8gwqzkephf5har86wflq86oldcw3yze4g6s78y4in5u3hc58)
- Add Llama-Nemotron & Openthoughts3 into post-training dataset [![#905](https://img.shields.io/github/issues/detail/state/marin-community/marin/905)](https://github.com/marin-community/marin/issues/905)
    - [GitHub Issue #905](https://github.com/marin-community/marin/issues/905)
    - [WandB Run: deeper_starling_sft_nemotron_and_openthoughts3](https://wandb.ai/marin-community/marin/runs/deeper_starling_sft_nemotron_and_openthoughts3)
- SFT on further cool-downed tootsie checkpoints [![#897](https://img.shields.io/github/issues/detail/state/marin-community/marin/897)](https://github.com/marin-community/marin/issues/897)
    - [GitHub Issue #897](https://github.com/marin-community/marin/issues/897)
- SFT Deeper Starling [![#1237](https://img.shields.io/github/issues/detail/state/marin-community/marin/1237)](https://github.com/marin-community/marin/issues/1237)
    - [GitHub Issue #1237](https://github.com/marin-community/marin/issues/1237)
    - [WandB Run: deeper_mixture_sft_starling_1e-4-longer-2](https://wandb.ai/marin-community/marin/runs/deeper_mixture_sft_starling_1e-4-longer-2?nw=nwuserheld)

## Scaling Laws

- Scaling laws to predict tootsie performance [![#654](https://img.shields.io/github/issues/detail/state/marin-community/marin/654)](https://github.com/marin-community/marin/issues/654)
    - [GitHub Issue #654](https://github.com/marin-community/marin/issues/654)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/654-Scaling-Law--VmlldzoxMDU2MjYwNQ?accessToken=oziscr4jytpwbat2z8erg4z7xapmi7hf82quru1b5qwu5ekklaes79gfr1b8eukk)
    - [WandB Report on Soft Metrics](https://wandb.ai/marin-community/marin/reports/654-Scaling-Laws-with-soft-metrics--VmlldzoxMDk2ODkxNw?accessToken=5mk1trabr6p2vpn2qxub79wqcsp1q0zk9yrzvvf9e2t8zqid7s9868v104m4amhx)
- Optimizer Scaling Law Part 1: AdamW [![#725](https://img.shields.io/github/issues/detail/state/marin-community/marin/725)](https://github.com/marin-community/marin/issues/725)
    - [GitHub Issue #725](https://github.com/marin-community/marin/issues/725)
    - [WandB Report](https://wandb.ai/marin-community/marin-optimizer/reports/AdamW-Sweeping--VmlldzoxMTE3Nzc5OA)
    - Conclusion: After sweeping, we discovered that the (near) optimal set of hyperparameters for AdamW remains surprisingly stable across three settings.
- Verify scaling batch size widens the gap between Muon & AdamW [![#1565](https://img.shields.io/github/issues/detail/state/marin-community/marin/1565)](https://github.com/marin-community/marin/issues/1565)
    - [GitHub Issue #1565](https://github.com/marin-community/marin/issues/1565)
    - [WandB Report](https://wandb.ai/marin-community/optimizer-scaling/reports/Verify-scaling-batch-size-widens-the-gap-between-Muon-AdamW--VmlldzoxNDI5MjAzMw)

## Baselines and Reproductions

- Train a simple Dolma/Olmo baseline to flex the pipeline [![#442](https://img.shields.io/github/issues/detail/state/marin-community/marin/442)](https://github.com/marin-community/marin/issues/442)
    - [GitHub Issue #442](https://github.com/marin-community/marin/issues/442)
    - [WandB Report](https://api.wandb.ai/links/stanford-mercury/e20j5423)
- Build DCLM 7b baseline [![#143](https://img.shields.io/github/issues/detail/state/marin-community/marin/143)](https://github.com/marin-community/marin/issues/143)
    - [GitHub Issue #143](https://github.com/marin-community/marin/issues/143)
    - [WandB Report](https://wandb.ai/marin-community/marin/reports/DCLM-7B-Replication--Vmlldzo5MTA3NjU5/edit)

## Other Projects

### Compel
- Compression-Ratio Quality Filter 1.4B Models [![#633](https://img.shields.io/github/issues/detail/state/marin-community/marin/633)](https://github.com/marin-community/marin/issues/633)
    - [GitHub Issue #633](https://github.com/marin-community/marin/issues/633)
    - [WandB Run: compel-fineweb-edu-baseline](https://wandb.ai/marin-community/marin/runs/compression-fineweb-edu-0.6-0.8-da53bf?nw=nwusereobbad)
    - [WandB Run: compel-fineweb-edu-0.65-0.8](https://wandb.ai/marin-community/marin/runs/compression-ratio-filter-fineweb-edu-786908?nw=nwusereobbad)
    - [WandB Run: compel-fineweb-baseline](https://wandb.ai/marin-community/marin/runs/compression-train-full-dataset-llama1.4b-20fa75?nw=nwusereobbad)
    - [WandB Run: compel-fineweb-0.65-0.8-e5cdae](https://wandb.ai/marin-community/marin/runs/compression-ratio-filter-llama1.4b-0.6-0.8-e5cdae?nw=nwusereobbad)


## Uncategorized

(This is for experiments that have been added via the script but have not yet been curated.)