# Automatic Machine Learning (AutoML) ## Surveys - **AutoML principles and practices** - Organizes automated learning by search space, search algorithm, and evaluation strategy, connecting pipeline configuration with architecture search and the underlying bilevel objective. [Shen et al. 2018]( "Automated Machine Learning: From Principles to Practices; arXiv"). - **Hyperparameter optimization survey** - Compares optimization approaches for common learning algorithms, summarizes software, and discusses strengths, limitations, and unresolved research questions. [Yang and Shami 2020]( "On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice; Neurocomputing"). - **AutoML survey** - Reviews automation across data preparation, feature engineering, hyperparameter optimization, and neural architecture search; distinguishes one-shot search and joint architecture/hyperparameter optimization. [He et al. 2021]( "AutoML: A Survey of the State-of-the-Art; Knowledge-Based Systems"). ## Hyperparameter search - **Sequential model-based optimization** - Introduces sequential expected-improvement methods for conditional hyperparameter spaces, including tree-structured Parzen estimation; evaluates them against random search. [Bergstra et al. 2011]( "Algorithms for Hyper-Parameter Optimization; NeurIPS"). - **Random search** - Samples configurations instead of enumerating a grid. Its analysis explains why grids waste trials when only a few hyperparameters strongly affect validation performance. [Bergstra and Bengio 2012]( "Random Search for Hyper-Parameter Optimization; JMLR"). - **Practical Bayesian optimization** - Models validation performance with a Gaussian process and selects subsequent trials from its posterior; also considers variable evaluation costs and parallel experiments. [Snoek et al. 2012]( "Practical Bayesian Optimization of Machine Learning Algorithms; NeurIPS"). - **Neural surrogates for Bayesian optimization** - Uses a neural network to learn basis functions for Bayesian regression, replacing a Gaussian-process surrogate when many evaluations make its fitting cost restrictive. [Snoek et al. 2015]( "Scalable Bayesian Optimization Using Deep Neural Networks; ICML"). - **Hyperband** - Allocates increasing resources to randomly sampled configurations and stops weak candidates early; searches across different tradeoffs between breadth and evaluation budget. [Li et al. 2018]( "Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization; JMLR"). - **BOHB** - Combines model-guided configuration selection with Hyperband's budget allocation, joining informed search with evaluations at multiple resource levels. [Falkner et al. 2018]( "BOHB: Robust and Efficient Hyperparameter Optimization at Scale; ICML"). - **ASHA** - Uses asynchronous resource allocation and aggressive early stopping for parallel hyperparameter tuning, allowing search to exploit many workers without synchronized evaluation rounds. [Li et al. 2020]( "A System for Massively Parallel Hyperparameter Tuning; MLSys"). ## Pipeline search and ensembles - **Auto-sklearn** - Combines Bayesian pipeline search with learning from previous datasets and ensembles of evaluated models; jointly selects preprocessing, classifiers, and hyperparameters. [Feurer et al. 2015]( "Efficient and Robust Automated Machine Learning; NeurIPS"). - **Auto-sklearn 2.0** - Adds portfolio-based meta-learning and bandit budget allocation, then selects among AutoML strategies to reduce the configuration burden on users. [Feurer et al. 2022]( "Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning; JMLR"). - **AlphaD3M** - Uses a pipeline grammar and model-based reinforcement learning with pretrained guidance to synthesize classification and regression pipelines; the paper reports public code, models, and data. [Drori et al. 2019]( "Automatic Machine Learning by Pipeline Synthesis using Model-Based Reinforcement Learning and a Grammar; ICML AutoML Workshop"). - **AutoGluon-Tabular** - Builds multilayer stacked ensembles for tabular prediction, emphasizing combinations of models rather than spending the entire budget on selecting one configuration. [Erickson et al. 2020]( "AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data; arXiv"). - **AutoML-DSGE** - Evolves scikit-learn classification pipelines with Dynamic Structured Grammatical Evolution, constraining candidate pipelines through a grammar. [Assunção et al. 2020]( "Evolution of Scikit-Learn Pipelines with Dynamic Structured Grammatical Evolution; Lecture Notes in Computer Science Applications of Evolutionary Computation"). - **Lale** - Unifies manual and automated pipeline construction through high-level Python interfaces, including topology search, higher-order operators, and consistency checks. [Baudart et al. 2020]( "Lale: Consistent Automated Machine Learning; AutoML@KDD"). - **Auto-PyTorch** - Jointly optimizes network architecture and training hyperparameters using multiple evaluation budgets, portfolio warm starts, and ensembles with tabular baselines; introduces LCBench learning curves. [Zimmer et al. 2021]( "Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL; IEEE Transactions on Pattern Analysis and Machine Intelligence"). - **FLAML** - Adapts learner and hyperparameter search to trial cost and model error, using resource-aware search strategies for tabular AutoML under limited computation budgets. [Wang et al. 2021]( "FLAML: A Fast and Lightweight AutoML Library; MLSys"). - **GAMA** - Provides modular pipeline-search and post-processing components with logging and visualization, supporting controlled comparisons between AutoML strategies. [Gijsbers and Vanschoren 2021]( "GAMA: a General Automated Machine learning Assistant; LNCS"). - **VolcanoML** - Decomposes large pipeline-search spaces into smaller searches and composes them into execution plans, separating search-space structure from optimization execution. [Li et al. 2021]( "VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition; Proc. VLDB Endow."). ## Pretrained models and agent-driven automation - **TabPFN** - Pretrains on synthetic prediction tasks, then conditions predictions on a labeled table. Amortizes learning across datasets; the paper evaluates small tables up to 10,000 rows, not unrestricted tabular prediction. [Hollmann et al. 2025]( "Accurate predictions on small data with a tabular foundation model; Nature"). - **MLZero** - Uses data-inspection agents, library knowledge and execution history to generate and revise ML pipelines. Evaluated on MLE-bench Lite and an author-created multimodal benchmark. [Fang et al. 2025]( "MLZero: A Multi-Agent System for End-to-end Machine Learning Automation; NeurIPS"); [code](). ## Neural architecture search - **Reinforcement-learning NAS** - Trains a recurrent controller to generate network descriptions, using candidate validation accuracy as a reinforcement-learning reward. [Zoph and Le 2017]( "Neural Architecture Search with Reinforcement Learning; ICLR"). - **ENAS** - Searches for subgraphs of a shared computation graph. Parameter sharing amortizes candidate training, while a policy-gradient controller selects architectures using validation rewards. [Pham et al. 2018]( "Efficient Neural Architecture Search via Parameter Sharing; ICML"). - **Regularized evolution** - Uses tournament-based architecture evolution with an age preference for younger candidates; studies search efficiency through controlled comparisons with reinforcement learning. [Real et al. 2019]( "Regularized Evolution for Image Classifier Architecture Search; AAAI"). - **DARTS** - Continuously relaxes discrete architecture choices so gradients can optimize the search representation, replacing evolutionary or reinforcement-learning search over discrete candidates. [Liu et al. 2019]( "DARTS: Differentiable Architecture Search; ICLR"). - **Petridish** - Grows networks by adding shortcut connections, motivated by forward stage-wise regression and gradient boosting; jointly trains candidate connections before selecting a subset. [Hu et al. 2019]( "Efficient Forward Architecture Search; NeurIPS"). - **MnasNet** - Searches mobile architectures using both accuracy and measured device latency in its objective; hardware measurements replace operation counts as the deployment-cost signal. [Tan et al. 2019]( "MnasNet: Platform-Aware Neural Architecture Search for Mobile; CVPR"). - **AgEBO-Tabular** - Combines aging evolution for architectures with asynchronous Bayesian tuning of data-parallel training settings, jointly adapting network structure and training configuration. [Egele et al. 2021]( "AgEBO-Tabular: Joint Neural Architecture and Hyperparameter Search with Autotuned Data-Parallel Training for Tabular Data; Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis"). - **Zero-cost NAS proxies** - Ranks candidate architectures using inexpensive initialization-time scores, including pruning-inspired proxies computed from one minibatch; proxy rankings still require validation against trained models. [Abdelfattah et al. 2021]( "Zero-Cost Proxies for Lightweight NAS; ICLR"). - **AIRA-Compose** - Agents assemble fixed computational primitives, then scale selected architectures. At 1B parameters and 37.5B tokens, AIRAformer-D improves over Composer; attention-heavy winners differ from the best equal-FLOP designs. Three training seeds. [Pepe et al. 2026]( "Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design; NeurIPS"). ## Model structure and algorithm discovery - **Compositional kernel search** - Searches Gaussian-process kernels built from sums and products of elementary kernels, yielding decompositions of regression functions into interpretable components. [Duvenaud et al. 2013]( "Structure Discovery in Nonparametric Regression through Compositional Kernel Search; ICML"). - **Automatic statistician** - Searches compositional Gaussian-process regression models and generates figures and natural-language descriptions of trends, periodicity, and changepoints. [Lloyd et al. 2014]( "Automatic Construction and Natural-Language Description of Nonparametric Regression Models; AAAI"). - **AutoML-Zero** - Evolves learning programs from basic mathematical operations, searching model structure and learning rules together; demonstrations use constrained tasks rather than unrestricted autonomous research. [Real et al. 2020]( "AutoML-Zero: Evolving Machine Learning Algorithms From Scratch; ICML"). ## Evaluation - **NAS-Bench-101** - Provides repeated evaluations of a fixed convolutional architecture space on CIFAR-10, allowing reproducible search comparisons through table queries rather than retraining each candidate. [Ying et al. 2019]( "NAS-Bench-101: Towards Reproducible Neural Architecture Search; ICML"). - **Random-search NAS baselines** - Li and Talwalkar find early-stopped random search competitive with ENAS on PTB and CIFAR-10. Weight sharing improves their baseline further; comparisons need repeated runs, matched training recipes and explicit search cost. [Li and Talwalkar 2019]( "Random Search and Reproducibility for Neural Architecture Search; UAI"). - **NAS-Bench-201** - Records training results for all 15,625 cells in a fixed search space on three datasets. Enables controlled comparisons of search algorithms; ranking performance within this space does not establish superiority across architectures or training recipes. [Dong and Yang 2020]( "NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search; ICLR"). - **AMLB** - Benchmarks AutoML systems across classification and regression tasks, measuring predictive quality, inference time, and failures; task subsets can produce different framework rankings. [Gijsbers et al. 2024]( "AMLB: an AutoML Benchmark; JMLR"). - **Budget-sensitive AutoML evaluation** - Across 104 tasks, framework rankings remain fairly stable down to five-minute budgets. Early stopping has less consistent effects; implementations stop individual models, search generations or ensembles. [Jurado et al. 2025]( "AutoML Benchmark with shorter time constraints and early stopping; ICLR Data Practices Workshop"). - **MLE-bench** - Benchmarks agents on ML competitions requiring data preparation, model training and experimental iteration; examines resource limits, pretraining contamination and grading against competition metrics. [Chan et al. 2025]( "MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering; ICLR"). - **TabArena** - Curates IID tabular tasks and compares defaults, tuning and ensembles. The best deep models match or beat boosted trees with tuning and ensembling at larger budgets; foundation models cover subsets in this paper snapshot. [Erickson et al. 2025]( "TabArena: A Living Benchmark for Machine Learning on Tabular Data; NeurIPS"). - **CARPS** - Connects optimizers to black-box, multi-fidelity and multi-objective benchmarks through one interface. Selects representative task subsets to reduce evaluation cost; records trajectories for analysis. [Benjamins et al. 2025]( "carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks; arXiv"); [code](). - **BeyondArena** - Across 142 datasets, default tabular foundation models lead on small IID tasks; tuned, ensembled conventional models lead on grouped/temporal tasks. Above 100k training rows, missing TabPFN/TabDPT scores use a random-forest fallback. [Purucker et al. 2026]( "Beyond IID: How General Are Tabular Foundation Models, Really?; NeurIPS"). - **Test-time compute for tabular models** - On TabArena, greedy selection improves TabPFN-3 while uniform averaging of the same 96 configurations worsens error. Selection gains shrink when fewer out-of-fold labels guide the ensemble. [Ning et al. 2026]( "Test-Time Compute for Tabular Foundation Models: Mechanisms, Gains, and Limits; arXiv"). ## Software ### Optimization and experiment software - **Hyperopt** - Describes search-space specification, sequential model-based optimization, and parallel execution in Python, with analysis of completed tuning trials. [Bergstra et al. 2013]( "Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms; SciPy 2013"). - **Optuna** - Defines conditional search spaces in Python; samplers propose parameters and pruners stop unpromising trials. Supports parallel studies and inspection of trial histories. [Akiba et al. 2019]( "Optuna: A Next-generation Hyperparameter Optimization Framework; KDD"); [docs](). - **SMAC3** - Bayesian optimization for conditional configuration spaces, multiple fidelities, objectives and problem instances. Provides ask-and-tell and warm-start interfaces. [Lindauer et al. 2022]( "SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization; JMLR"); [docs](). - **Ray Tune** - Separates training scripts from model-selection algorithms through a common interface, supporting distributed search and scheduling across compute clusters. [Liaw et al. 2018]( "Tune: A Research Platform for Distributed Model Selection and Training; ICML AutoML Workshop"). - **Katib** - Presents a framework-agnostic tuning service designed around scalability, fault tolerance, extensibility, and multiple users in cloud-native deployments. [George et al. 2020]( "A Scalable and Cloud-Native Hyperparameter Tuning System; arXiv"). - **Ax** - Adaptive experimentation platform for defining, managing, and running optimization studies, including field experiments. [docs](). - **BoTorch** - PyTorch library for Bayesian optimization research; integrates with Ax for experiment management and data handling. [docs](). - **KerasTuner** - Defines hyperparameters during model construction and searches them with Bayesian optimization, Hyperband, or random search. [docs](). ### Pipeline and domain software - **TPOT** - Optimizes machine-learning pipelines through genetic programming; its graph-based rewrite supports flexible search spaces and multiple optimization objectives. [docs](). - **AutoKeras** - Keras-based AutoML interfaces for fitting models from data, with task-specific estimators and architecture search. [docs](). - **H2O AutoML** - Automates supervised model training and tuning in H2O, including ensembles of candidate models. [docs](). - **AutoGBT** - Automatically tunes gradient-boosted trees for streaming classification under concept drift; developed for the lifelong AutoML challenge. [code](). - **Ludwig** - Declarative framework for training and fine-tuning models from configuration files, including language, multimodal, and tabular models. [code](). - **AutoGluon** - Pipeline interfaces for tabular, time-series and multimodal prediction. Tabular presets combine conventional learners and pretrained models; model choices, hardware needs and checkpoint licenses vary by preset. [docs](). - **FLAML** - Searches learners and configurations under time or trial budgets, with optional stacked ensembles. Supports prediction-time and custom metric constraints; search cost and deployment constraints are separate controls. [docs](). ### Archived and legacy implementations - **NNI** - Toolkit for hyperparameter search, neural architecture search and model compression. Microsoft archived the repository in September 2024; preserved documentation describes the archived implementation. [code](). - **AdaNet** - TensorFlow framework combining neural-subnetwork growth with ensemble learning. The repository was archived in April 2026; the method remains a research reference, while this implementation is read-only. [code](). - **HpBandSter** - Distributed Hyperband/BOHB implementation. Maintainers explicitly state that it is no longer maintained and point to SMAC3 and DEHB as successors; retain for historical reproduction. [code](). ## Related topics - [Optimization](https://mlpapers.org/optimization/) - [Neural Networks](https://mlpapers.org/neural-nets/) - [Ensemble Learning](https://mlpapers.org/ensemble-learning/) - [Feature Selection](https://mlpapers.org/feature-selection/)