# Automated Research

## Program and algorithm discovery

- **AlphaTensor** - Reinforcement learning searches tensor decompositions for matrix multiplication. Finds a 4-by-4 algorithm using 47 rather than 49 multiplications over the two-element field; this result does not apply to arbitrary real-valued matrices. [Fawzi et al. 2022](<https://doi.org/10.1038/s41586-022-05172-4> "Discovering faster matrix multiplication algorithms with reinforcement learning; Nature").
- **AlphaDev** - Reinforcement learning builds assembly programs using correctness and latency rewards. Discovered small sorting routines were translated into C++ and adopted in LLVM libc++; the search relies on supplied tests, instructions and hardware objectives. [Mankowitz et al. 2023](<https://doi.org/10.1038/s41586-023-06004-9> "Faster sorting algorithms discovered using deep reinforcement learning; Nature").
- **FunSearch** - Uses a frozen LLM to evolve functions inside a supplied program skeleton using executable scores and separate populations. In direct cap-set search, 4 of 140 runs found a size-512 set in eight dimensions; this establishes a lower bound, not optimality. [Romera-Paredes et al. 2024](<https://doi.org/10.1038/s41586-023-06924-6> "Mathematical discoveries from program search with large language models; Nature").
- **AlphaEvolve** - Extends FunSearch to multi-function program evolution with supplied evaluators. Reusing earlier programs beat restarting from the initial program on matrix-decomposition and kissing-number tasks, averaged over three runs. [Novikov et al. 2025](<https://arxiv.org/abs/2506.13131v1> "AlphaEvolve: A coding agent for scientific and algorithmic discovery; arXiv").
- **Learning-algorithm search** - [AutoML-Zero](https://mlpapers.org/automl/#model-structure-and-algorithm-discovery) evolves learning programs from basic operations. It supplies a non-LLM connection to algorithm discovery; pipeline and hyperparameter search remain on AutoML.

## Experiment automation

- **Adam robot scientist** - Combines logical hypotheses about yeast enzymes with robotic growth experiments. Humans supplied models and lab infrastructure; subsequent protein assays confirmed three predicted gene-enzyme links. Some rediscoveries came from an incomplete database. [King et al. 2009](<https://doi.org/10.1126/science.1165620> "The Automation of Science; Science").
- **The AI Scientist** - Proposes ideas, edits code, runs experiments and writes papers from an experiment template. Authors report incorrect implementations and changed metrics compared against unchanged baselines, despite automated review. [Lu et al. 2024](<https://arxiv.org/abs/2408.06292v3> "The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery; arXiv"); [code](<https://github.com/SakanaAI/AI-Scientist>).
- **Recursive automated research** - Industrial case study of parallel experiment threads for model training and GPU kernels. Released scripts and selected outputs allow inspection, but the complete research controller is not released. [Recursive 2026](<https://recursive.com/updates/first-steps-toward-automated-ai-research> "First Steps Toward Automated AI Research; blog"); [code](<https://github.com/recursive-org/first-steps-toward-automated-ai-research/tree/a962ec43e2e3d7c018e59a2ece623fe6e232fdfb>).
- **Pipeline automation** - [MLZero](https://mlpapers.org/automl/#pretrained-models-and-agent-driven-automation) uses data inspection, library knowledge and execution history to revise ML pipelines. Predictive pipeline performance and production of new research knowledge are different evaluation targets.
- **AIRA2** - Combines asynchronous experiment workers with fixed hidden search data and a separate final-selection split. Its ablation removes late-search degradation caused by inconsistent evaluation. Eight-GPU results are not compute-matched to single-GPU baselines. [Hambardzumyan et al. 2026](<https://arxiv.org/abs/2603.26499v2> "AIRA_2: Overcoming Bottlenecks in AI Research Agents; NeurIPS").
- **Heuresis** - Among six search strategies, Greedy wins on nanoGPT and unlearning; MAP-Elites wins on Breakout. Islands transfers best to held-out Asterix; Greedy's five elites all fail or time out. Search-task scores do not identify the best transfer strategy. [Antoniades et al. 2026](<https://arxiv.org/abs/2606.25198v2> "Heuresis: Search Strategies for Autonomous AI Research Agents Across Quality, Diversity and Novelty; NeurIPS").

## Improving the researcher

- **Fisher-R1** - On 222 hard P-Bench tasks, SFT plus RL raises Qwen2.5-Coder-14B strict pass@1 from 23.7% to 33.0%, versus GPT-5.4's 30.5%. Strict scores require a correct decision and near-reference p-value; three evaluation runs. [Miao et al. 2026](<https://arxiv.org/abs/2608.07437v1> "Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing; NeurIPS").

## Research benchmarks and evidence

- **Human evaluation of research ideas** - In a blinded study with over 100 NLP researchers contributing ideas or reviews, LLM ideas received higher novelty ratings but slightly lower feasibility ratings. These were judgments of proposals before execution. [Si et al. 2024](<https://arxiv.org/abs/2409.04109v1> "Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers; arXiv").
- **MLAgentBench** - Provides 13 ML experimentation tasks with file editing, execution and output inspection. Evaluates baseline improvement and time/token costs. Its ReAct-based agent tracks research plans, status and recent actions. [Huang et al. 2024](<https://proceedings.mlr.press/v235/huang24y.html> "MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation; ICML"); [code](<https://github.com/snap-stanford/MLAgentBench>).
- **Ideation-execution gap** - 43 experts executed NLP research ideas. AI ideas lost more review score after execution; with projects as independent units, post-execution human-AI differences were not significant. Proposal novelty did not establish better research outcomes. [Si et al. 2025](<https://arxiv.org/abs/2506.20803v1> "The Ideation–Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas; arXiv").
- **RE-Bench** - Seven ML research-engineering tasks compare agents with 61 experts. Best agents lead at two total hours; humans lead at eight and 32. Larger-budget estimates use best-of-k attempts, including four eight-hour human runs at 32 hours. [Wijk et al. 2025](<https://proceedings.mlr.press/v267/wijk25a.html> "RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts; ICML").
- **CORE-Bench** - 270 tasks from 90 papers separate reading supplied outputs, executing code in a supplied environment, and setting up that environment. Tests reproducibility using authors' code and data, unlike PaperBench's implementation-from-paper task. [Siegel et al. 2025](<https://arxiv.org/abs/2409.11363v2> "CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark; TMLR").
- **PaperBench** - Tests replication of 20 ICML papers using author-developed rubrics, fresh code execution and an LLM judge. Scores measure weighted rubric completion. For o1, code-only scores correlate weakly with full replication scores. [Starace et al. 2025](<https://proceedings.mlr.press/v267/starace25a.html> "PaperBench: Evaluating AI’s Ability to Replicate AI Research; ICML").
- **ML engineering benchmarks** - [MLE-bench](https://mlpapers.org/automl/#evaluation) tests agents on ML competitions. Competition scores measure task performance; PaperBench instead checks replication of specified research results. Neither alone establishes novelty.
- **MLS-Bench** - Tests 140 tasks across 12 domains with fixed editable components, capacity limits and multiple evaluation settings. Agents usually trail reproduced human methods even with their code in context; removing capacity limits lets some scores rise through larger models. [Lyu et al. 2026](<https://arxiv.org/abs/2605.08678v3> "MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI; NeurIPS").
- **Unsatz** - Agents buy experiments and infer hidden variables and equations in 80 simulated systems. A supplied fitter estimates coefficients. Nine LLMs solve 21-60% on unseen interventions, longer horizons and conditions outside the explored range. [Ji et al. 2026](<https://arxiv.org/abs/2611.03713v1> "Unsatz: Evaluating Scientific Discovery Without a Given Ansatz; arXiv").

## Software

- **autoresearch** - Agents edit train.py and retain changes by validation bits per byte. prepare.py fixes data/evaluation; humans edit program.md. Five minutes is a per-run training budget, excluding startup and compilation; total search costs accumulate across runs. [code](<https://github.com/karpathy/autoresearch>).
- **FunSearch implementation** - Releases discovered programs, evaluation notebooks and a single-threaded search pipeline. Users must supply the language model and execution sandbox; the original distributed search infrastructure is not included. [code](<https://github.com/google-deepmind/funsearch>).

## Related topics

- [AutoML](https://mlpapers.org/automl/)
- [Optimization](https://mlpapers.org/optimization/)
- [Evaluation](https://mlpapers.org/evaluation/)
- [Natural Language Processing](https://mlpapers.org/nlp/)
- [Symbolic Regression](https://mlpapers.org/symbolic-regression/)
- [Scientific Machine Learning](https://mlpapers.org/scientific-machine-learning/)