# Optimization

## Foundations and constrained optimization

- **L-BFGS** - A limited-memory quasi-Newton method for large nonlinear optimization. Approximates curvature without storing a full dense Hessian. [Liu and Nocedal 1989](<https://link.springer.com/article/10.1007/BF01589116> "On the limited memory BFGS method for large scale optimization; Mathematical Programming").
- **Convex optimization** - Convex objectives and feasible sets make local optima global. Covers duality, optimality conditions, and numerical algorithms; convexity must be established for the actual formulation. [Boyd and Vandenberghe 2004](<https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf> "Convex Optimization; book").
- **Linear programming** - Optimizes a linear objective under linear equality and inequality constraints. Simplex and interior-point methods address this structured problem class. [Boyd and Vandenberghe 2004](<https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf> "Convex Optimization; book").
- **KKT conditions** - Combines feasibility, stationarity, and complementary slackness. These conditions are sufficient for convex problems; necessity requires appropriate regularity or strong duality. [Boyd and Vandenberghe 2004](<https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf> "Convex Optimization; book").
- **Newton and interior-point methods** - Newton steps use local quadratic models. Interior-point methods solve constrained convex problems through a sequence of smooth subproblems. [Boyd and Vandenberghe 2004](<https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf> "Convex Optimization; book").
- **ADMM** - Splits structured optimization into coupled subproblems. The review covers distributed convex learning, convergence conditions, and implementation; nonconvex extensions need separate analysis. [Boyd et al. 2011](<https://web.stanford.edu/~boyd/papers/pdf/admm_distr_stats.pdf> "Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers; Found. Trends ML").
- **Frank-Wolfe / conditional gradients** - Uses linear minimization over a convex feasible set instead of Euclidean projection. Jaggi gives primal-dual certificates and sparse-update analysis; the advantage depends on having a tractable linear oracle. [Jaggi 2013](<https://proceedings.mlr.press/v28/jaggi13.html> "Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization; ICML").
- **Proximal algorithms** - Handles nonsmooth penalties and constraints through proximal subproblems, generalizing projection. Includes proximal gradient, acceleration, and splitting methods. [Parikh and Boyd 2014](<https://web.stanford.edu/~boyd/papers/pdf/prox_algs.pdf> "Proximal Algorithms; Found. Trends Optim.").
- **Mirror descent** - Uses a distance induced by a convex mirror map to adapt optimization to non-Euclidean geometry. Bubeck's analysis connects the map and norm to convergence; Euclidean gradient descent is one geometry choice. [Bubeck 2015](<https://arxiv.org/abs/1405.4980> "Convex Optimization: Algorithms and Complexity; Foundations and Trends in Machine Learning").
- **Coordinate descent** - Approximately minimizes along individual coordinates or blocks. Reviews convergence, acceleration, and parallel variants, mainly for convex objectives. [Wright 2015](<https://arxiv.org/abs/1502.04759> "Coordinate Descent Algorithms; Mathematical Programming").

## Gradient methods and adaptive updates

- **Nesterov acceleration** - Combines gradient steps with extrapolation. Establishes an O(1/k²) objective-gap rate for smooth convex minimization under the paper's assumptions, not arbitrary neural-network training. [Nesterov 1983](<https://hengshuaiyao.github.io/papers/nesterov83.pdf> "A Method of Solving a Convex Programming Problem with Convergence Rate O(1/k²); Soviet Math. Dokl.").
- **AdaGrad** - Adapts optimization geometry to observed gradients, with online regret guarantees. Useful when informative features occur at different frequencies. [Duchi et al. 2011](<https://jmlr.org/papers/v12/duchi11a.html> "Adaptive Subgradient Methods for Online Learning and Stochastic Optimization; JMLR").
- **AdaDelta** - Adapts learning rates per dimension using first-order information. Proposed to address AdaGrad's diminishing learning rates and sensitivity to a manually chosen global rate. [Zeiler 2012](<https://arxiv.org/abs/1212.5701> "ADADELTA: An Adaptive Learning Rate Method; arXiv").
- **Initialization and momentum** - Studies how initialization and the momentum schedule affect training deep and recurrent networks. Poor initialization can defeat otherwise effective momentum settings. [Sutskever et al. 2013](<https://proceedings.mlr.press/v28/sutskever13.html> "On the importance of initialization and momentum in deep learning; ICML").
- **SAGA** - Caches component gradients and uses their average to correct stochastic updates, reducing gradient variance for finite-sum objectives; supports proximal updates for nonsmooth regularization. [Defazio et al. 2014](<https://arxiv.org/abs/1407.0202> "SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives; NeurIPS").
- **Adam and AdaMax** - Adam combines first- and second-moment gradient estimates with bias correction; AdaMax uses an infinity-norm variant. Later convergence analyses qualify the original guarantees. [Kingma and Ba 2015](<https://arxiv.org/abs/1412.6980> "Adam: A Method for Stochastic Optimization; ICLR").
    - **Nadam** - Modifies Adam's momentum component using Nesterov-style acceleration. The workshop paper reports preliminary comparisons of the update with Adam. [Dozat 2016](<https://openreview.net/pdf?id=OM0jvwB8jIp57ZJjtNEZ> "Incorporating Nesterov Momentum into Adam; ICLR Workshop").
    - **AMSGrad** - Exhibits a convex counterexample to Adam convergence and introduces longer-term gradient memory. Corrects weaknesses in the original analysis under specified optimization assumptions. [Reddi et al. 2018](<https://arxiv.org/abs/1904.09237> "On the Convergence of Adam and Beyond; ICLR").
    - **AdamW** - Decouples weight decay from the adaptive gradient update. L2 regularization and multiplicative weight decay are generally not equivalent in adaptive optimizers. [Loshchilov and Hutter 2019](<https://arxiv.org/abs/1711.05101> "Decoupled Weight Decay Regularization; ICLR").
    - **TAdam** - Uses a Student-t-based update to reduce sensitivity to gradient outliers. [Ilboudo et al. 2020](<https://arxiv.org/abs/2003.00179> "TAdam: A Robust Stochastic Gradient Optimizer; arXiv").
- **RMSprop** - Hinton's lecture 6e describes scaling each gradient by a running root-mean-square of recent gradients. Exponential averaging emphasizes recent magnitudes, unlike AdaGrad's cumulative sum. [docs](<https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf>).
- **Gradient descent variants** - Contrasts full-batch, stochastic, and mini-batch gradients; surveys momentum, adaptive learning rates, optimization difficulties, and parallel training. [Ruder 2016](<https://arxiv.org/abs/1609.04747> "An overview of gradient descent optimization algorithms; arXiv").
- **Large-scale stochastic optimization** - Connects stochastic-gradient theory to practical machine learning, including noise reduction and approximate second-order methods. [Bottou et al. 2018](<https://arxiv.org/abs/1606.04838> "Optimization Methods for Large-Scale Machine Learning; SIAM Review").
- **Adafactor** - Factors second-moment statistics into row and column estimates for matrix parameters, reducing optimizer-state memory. Also studies update clipping and learning-rate choices. [Shazeer and Stern 2018](<https://arxiv.org/abs/1804.04235> "Adafactor: Adaptive Learning Rates with Sublinear Memory Cost; ICML").
- **Shampoo** - Uses tensor-structured matrix preconditioners to capture gradient geometry beyond coordinatewise scaling. Provides regret analysis in the online convex setting. [Gupta et al. 2018](<https://arxiv.org/abs/1802.09568> "Shampoo: Preconditioned Stochastic Tensor Optimization; ICML").
- **When SGD escapes local minima** - Interprets SGD through a smoothed objective. Escape and convergence results rely on assumptions about that objective and the noise; they are not general guarantees for all loss surfaces. [Kleinberg et al. 2018](<https://arxiv.org/abs/1802.06175> "An Alternative View: When Does SGD Escape Local Minima?; ICML").
- **LaProp** - Separates momentum from adaptive gradient normalization. Studies instability caused by coupling these components and proposes a different update order. [Ziyin et al. 2020](<https://arxiv.org/abs/2002.04839> "LaProp: Separating Momentum and Adaptivity in Adam; arXiv").
- **SAM** - Minimizes loss over a neighborhood of parameter values to seek solutions with low loss and low local sharpness. This changes the training objective and gradient computation. [Foret et al. 2021](<https://arxiv.org/abs/2010.01412> "Sharpness-Aware Minimization for Efficiently Improving Generalization; ICLR").
- **Lion** - An optimizer discovered through program search, using momentum and sign-based updates. Its learning-rate scale differs from Adam's, so comparisons require separate tuning. [Chen et al. 2023](<https://arxiv.org/abs/2302.06675> "Symbolic Discovery of Optimization Algorithms; NeurIPS").
- **D-Adaptation** - Adapts step sizes without specifying an initial learning rate; distinguishes convex Lipschitz convergence guarantees from empirical SGD and Adam variants for deep learning. [Defazio and Mishchenko 2023](<https://arxiv.org/abs/2301.07733> "Learning-Rate-Free Learning by D-Adaptation; ICML").
- **Sophia** - Combines momentum with an intermittently estimated diagonal Hessian and clipping. Designed for language-model pretraining; curvature estimation adds its own cost and choices. [Liu et al. 2024](<https://arxiv.org/abs/2305.14342> "Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training; ICLR").
- **Muon for language-model training** - Studies matrix-orthogonalized updates at scale, including weight decay, update scaling, and distributed implementation. [Liu et al. 2025](<https://arxiv.org/abs/2502.16982> "Muon is Scalable for LLM Training; arXiv").
- **Semi-gradient updates** - Stop-gradient operations can produce update fields without a scalar potential. [Flow-map convergence](https://mlpapers.org/generative-models/#flow-matching-and-transport) covers a case with convergent idealized dynamics.

## Training schedules and online control

- **Learned learning-rate controllers** - Learns a policy from training statistics to adjust SGD or RMSprop rates. Transfers from small MNIST networks to tested larger networks and CIFAR-10, but the tested controller did not transfer between base optimizers. [Daniel et al. 2016](<https://doi.org/10.1609/aaai.v30i1.10187> "Learning Step Size Controllers for Robust Neural Network Training; AAAI").
- **SGDR** - Cosine learning-rate decay with periodic warm restarts. Restarts raise the rate while retaining weights; cycle lengths and rate bounds are chosen in advance. [Loshchilov and Hutter 2017](<https://arxiv.org/abs/1608.03983v5> "SGDR: Stochastic Gradient Descent with Warm Restarts; ICLR").
- **Stochastic Weight Averaging** - Averages weights along constant- or cyclic-rate SGD trajectories to produce one model. The paper connects this to geometric ensembling; averaging weights differs from averaging predictions. [Izmailov et al. 2018](<https://arxiv.org/abs/1803.05407v3> "Averaging Weights Leads to Wider Optima and Better Generalization; UAI").
- **Constant rate with cooldown** - Shares a constant-rate training trajectory across stopping budgets through separate cooldown branches. Tuned schedules closely match cosine in the paper's language-model experiments. [Hägele et al. 2024](<https://arxiv.org/abs/2405.18392> "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations; NeurIPS").
- **Warmup and batch scaling** - See [Distributed Learning](https://mlpapers.org/distributed-learning/#scaling-evidence-and-measurement) for the ResNet-50 recipe combining warmup and learning-rate scaling at batch size 8,192; that result does not establish a universal scaling rule.
- **Schedule-Free optimization** - Interpolates base and averaged iterates for gradient evaluation, replacing horizon-dependent decay. Learning rate and momentum still matter. Its AdamW variant won AlgoPerf's 2024 self-tuning track. [Defazio et al. 2024](<https://arxiv.org/abs/2405.15682v4> "The Road Less Scheduled; NeurIPS"); [code](<https://github.com/facebookresearch/schedule_free>).
    - **Nonconvex Schedule-Free theory** - Stationarity rates concern gradient-evaluation iterates of GD/SGD, not the averaged model used for evaluation. Assumes smooth lower-bounded losses; SGD needs unbiased bounded-variance gradients. AdamW is outside this analysis. [Chae and Kim 2026](<https://arxiv.org/abs/2607.09167v1> "Understanding Schedule-Free Methods in Nonconvex Optimization: Rate Guarantees and Escaping Saddles; NeurIPS").

## Learning for downstream decisions

- **Smart predict-then-optimize** - Trains predictions for their downstream decision cost. SPO+ supplies a convex surrogate for linear-objective optimization; shortest-path and portfolio experiments show gains under model misspecification. [Elmachtoub and Grigas 2022](<https://arxiv.org/abs/1710.08005v5> "Smart &quot;Predict, then Optimize&quot;; Management Science").

## Bayesian optimization

- **Bayesian optimization foundations** - Uses a probabilistic surrogate and an acquisition function to choose expensive evaluations. Balances learning about uncertain regions against exploiting promising ones. [Brochu et al. 2010](<https://arxiv.org/abs/1012.2599> "A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning; arXiv").
- **Practical Bayesian optimization** - Studies Gaussian-process choices for hyperparameter tuning, including acquisition functions, evaluation costs, and parallel evaluations. [Snoek et al. 2012](<https://arxiv.org/abs/1206.2944> "Practical Bayesian Optimization of Machine Learning Algorithms; NeurIPS").
- **REMBO** - Uses random embeddings to search high-dimensional objectives with low effective dimensionality. Ambient dimension alone does not determine whether this reduction is suitable. [Wang et al. 2013](<https://www.cs.ubc.ca/~hutter/papers/13-IJCAI-BO-highdim.pdf> "Bayesian Optimization in High Dimensions via Random Embeddings; IJCAI").
    - **Choosing an embedding domain** - Examines the search bounds used with random embeddings. Small domains can exclude useful solutions; large domains can waste evaluations and weaken the dimensionality advantage. [Binois et al. 2019](<https://arxiv.org/abs/1704.05318> "On the choice of the low-dimensional domain for global optimization via random embeddings; Journal of Global Optimization").
- **Bayesian optimization review** - Reviews surrogate models, acquisition strategies, and extensions for automated design and machine learning. Useful for locating assumptions behind a particular BO setup. [Shahriari et al. 2016](<https://www.cs.ox.ac.uk/people/nando.defreitas/publications/BayesOptLoop.pdf> "Taking the Human Out of the Loop: A Review of Bayesian Optimization; Proceedings of the IEEE").
- **BOHB** - Combines model-based configuration proposals with Hyperband's allocation of evaluation budgets. Intended for tuning when lower-budget runs provide useful information. [Falkner et al. 2018](<https://proceedings.mlr.press/v80/falkner18a.html> "BOHB: Robust and Efficient Hyperparameter Optimization at Scale; ICML").
- **HeSBO** - Uses hashing-based embeddings for Bayesian optimization in lower-dimensional subspaces. Analyzes approximation behavior and evaluates the method on high-dimensional problems. [Nayebi et al. 2019](<https://proceedings.mlr.press/v97/nayebi19a.html> "A Framework for Bayesian Optimization in Embedded Subspaces; ICML").
- **SIRBO** - Learns an objective-relevant subspace through supervised dimension reduction, including sliced inverse regression. The embedding is informed by observed objective values. [Zhang et al. 2019](<https://arxiv.org/abs/1907.08953> "High Dimensional Bayesian Optimization via Supervised Dimension Reduction; IJCAI").
- **TuRBO** - Fits local probabilistic models within adaptive trust regions and allocates evaluations across regions, targeting expensive black-box optimization beyond a single global surrogate. [Eriksson et al. 2019](<https://arxiv.org/abs/1910.01739> "Scalable Global Optimization via Local Bayesian Optimization; NeurIPS").
- **SILBO** - Combines evaluated points with unlabeled acquisition candidates to learn an embedding for high-dimensional Bayesian optimization; includes a randomized projection strategy. [Chen et al. 2020](<https://arxiv.org/abs/2005.14601> "Semi-supervised Embedding Learning for High-dimensional Bayesian Optimization; arXiv").

## Perturbation-based gradient estimation

- **SPSA** - Estimates a gradient from two noisy loss evaluations with simultaneous random perturbations of all coordinates. Evaluation count per estimate is independent of dimension; convergence still requires noise, smoothness and step-size assumptions. [Spall 1992](<https://www.jhuapl.edu/spsa/pdf-spsa/spall_tac92.pdf> "Multivariate stochastic approximation using a simultaneous perturbation gradient approximation; IEEE Trans. Automat. Contr.").
- **Neural credit assignment** - Perturbation estimators, random feedback, local losses and forward-mode autodiff supply learning signals with different derivative requirements and costs. See [neural-network methods and comparisons](<https://mlpapers.org/neural-nets/#credit-assignment-and-alternatives-to-backpropagation>).

## Evolutionary and population-based search

- **Differential evolution** - A population-based heuristic for continuous global optimization, including nondifferentiable objectives. [Storn and Price 1997](<https://link.springer.com/article/10.1023/A:1008202821328> "Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces; J. Glob. Optim.").
- **Evolutionary computation** - Reviews search through populations, selection, and variation, including genetic algorithms and evolving programs. These methods optimize a supplied fitness criterion. [Foster 2001](<https://www.nature.com/articles/35076523> "Evolutionary computation; Nature Reviews Genetics").
- **Harmony search** - A heuristic inspired by musical improvisation, illustrated on traveling-salesman and engineering-design problems. [Zong Woo Geem et al. 2001](<https://doi.org/10.1177/003754970107600201> "A New Heuristic Optimization Algorithm: Harmony Search; SIMULATION").
- **Learning classifier systems** - Reviews rule-based learning systems and their evolutionary components. Different implementations address classification, robotics, knowledge discovery, and modeling. [Urbanowicz and Moore 2009](<https://doi.org/10.1155/2009/736398> "Learning Classifier Systems: A Complete Introduction, Review, and Roadmap; JAEA").
- **Memetic algorithms** - Combines evolutionary search with local search components. Reviews discrete, continuous, constrained, multiobjective, and uncertain problems, emphasizing coordination between operators. [Neri and Cotta 2012](<https://www.sciencedirect.com/science/article/pii/S2210650211000691> "Memetic algorithms and memetic computing optimization: A literature review; Swarm Evol. Comput.").
- **CMA-ES** - Covariance-matrix adaptation for stochastic optimization of nonlinear, nonconvex functions over continuous parameters. The tutorial derives the search strategy and its design choices. [Hansen 2016](<https://arxiv.org/abs/1604.00772> "The CMA Evolution Strategy: A Tutorial; arXiv").
- **Population Based Training** - Jointly trains a population of models and adapts their hyperparameter schedules. Combines asynchronous training with selection and exploration across population members. [Jaderberg et al. 2017](<https://arxiv.org/abs/1711.09846> "Population Based Training of Neural Networks; arXiv").

## Local search and swarm methods

- **Metropolis-based optimization** - Approximates a constrained minimum using integral representations and Metropolis sampling. The analysis assumes a closed bounded domain and a unique minimizer. [Pincus 1970](<https://doi.org/10.1287/opre.18.6.1225> "Letter to the Editor—A Monte Carlo Method for the Approximate Solution of Certain Types of Constrained Optimization Problems; Operations Research").
- **Simulated annealing** - Connects optimization to thermal annealing: randomized moves can accept worse states, with acceptance controlled by temperature. Cooling schedule and finite evaluation budget matter. [Kirkpatrick et al. 1983](<https://doi.org/10.1126/science.220.4598.671> "Optimization by Simulated Annealing; Science").
- **Tabu search** - Uses search memory, tabu restrictions, and aspiration criteria to guide local search. Intensification and diversification control revisiting promising regions and exploring new ones. [Glover 1990](<https://doi.org/10.1287/inte.20.4.74> "Tabu Search: A Tutorial; Interfaces").
- **Particle swarm optimization** - Moves candidate solutions using velocities influenced by each particle's best position and the swarm's best position. A derivative-free population heuristic for continuous search. [Kennedy and Eberhart 1995](<https://www.cs.tufts.edu/comp/150GA/homeworks/hw3/_reading6%201995%20particle%20swarming.pdf> "Particle swarm optimization; ICNN").
- **Artificial bee colony** - A numerical optimization heuristic based on honey-bee foraging, with employed, onlooker, and scout roles balancing exploitation of food sources and exploration. [Karaboga 2005](<https://abc.erciyes.edu.tr/pub/tr06_2005.pdf> "An Idea Based on Honey Bee Swarm for Numerical Optimization; technical report").
- **Cuckoo search** - Combines a brood-parasitism-inspired search scheme with Lévy flights for global optimization. The cited work evaluates numerical test functions. [Yang and Deb 2009](<https://arxiv.org/abs/1003.1594> "Cuckoo Search via Levy Flights; 2009 World Congress on Nature &amp; Biologically Inspired Computing (NaBIC)").
- **Firefly algorithm** - Applies the heuristic to nonlinear design optimization and introduces stochastic or singular test functions with known optima. Evidence is restricted to the tested problems. [Yang 2010](<https://arxiv.org/abs/1003.1409> "Firefly Algorithm, Stochastic Test Functions and Design Optimisation; International Journal of Bio-Inspired Computation").
- **Variable neighborhood search** - Systematically changes neighborhoods during descent and perturbation. Reviews basic schemes, parallel implementations, and hybrids for escaping local-search traps. [Hansen et al. 2017](<https://link.springer.com/article/10.1007/s13675-016-0075-x> "Variable neighborhood search: basics and variants; EURO J. Comput. Optim.").

## Bandits and sequential allocation

- **Sequential experimental design** - Studies experiments whose sampling decisions depend on earlier observations. Also explains why optional stopping can invalidate tests designed for fixed sample sizes. [Robbins 1952](<https://projecteuclid.org/journals/bulletin-of-the-american-mathematical-society/volume-58/issue-5/Some-aspects-of-the-sequential-design-of-experiments/bams/1183517370.pdf> "Some aspects of the sequential design of experiments; Bull. Amer. Math. Soc.").
- **Finite-time bandit analysis** - Analyzes exploration-exploitation policies with logarithmic regret uniformly over time for bounded reward distributions. Guarantees concern the stated stochastic bandit setting. [Auer et al. 2002](<https://link.springer.com/article/10.1023/A:1013689704352> "Finite-time Analysis of the Multiarmed Bandit Problem; Mach. Learn.").
- **Empirical bandit comparison** - Compares multi-armed bandit algorithms and introduces the POKER strategy. Results on the paper's reward settings are not a universal ranking of exploration policies. [Vermorel and Mohri 2005](<https://bandit.sourceforge.net/Vermorel2005poker.pdf> "Multi-armed Bandit Algorithms and Empirical Evaluation; ECML").
- **Bayesian probability matching** - Allocates observations according to posterior probabilities that each arm is optimal. Connects Bayesian computation, adaptive experimental design, and bandit heuristics. [Scott 2010](<https://www.economics.uci.edu/files/docs/micro/s11/Scott.pdf> "A modern Bayesian look at the multi‐armed bandit; Appl Stoch Models Bus &amp; Ind").
- **Bandit regret survey** - Covers stochastic and adversarial payoffs, finite actions, and contextual extensions. Regret bounds depend on the feedback model and assumptions about rewards. [Bubeck and Cesa-Bianchi 2012](<https://arxiv.org/abs/1204.5721> "Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems; Foundations and Trends in Machine Learning").
- **Thompson sampling** - Reviews posterior-sampling approaches to sequential decisions, applications with shared information across actions, and settings where the method is or is not effective. [Russo et al. 2018](<https://arxiv.org/abs/1707.02038> "A Tutorial on Thompson Sampling; Foundations and Trends in Machine Learning").

## Multiple objectives and evaluation

- **NSGA-II** - Combines nondominated sorting with elitist selection from parents and offspring, balancing fitness and spread. Approximates a set of trade-offs instead of choosing a single objective weighting. [Deb et al. 2002](<https://doi.org/10.1109/4235.996017> "A fast and elitist multiobjective genetic algorithm: NSGA-II; IEEE Trans. Evol. Computat.").
- **AlgoPerf** - Benchmarks time to a target result across neural-network workloads on fixed hardware. Controls tuning protocols and workload variants to distinguish optimizer gains from evaluation artifacts. [Dahl et al. 2023](<https://arxiv.org/abs/2306.07179> "Benchmarking Neural Network Training Algorithms; arXiv").
- **LLM optimizer comparisons** - In 124M FineWeb runs, longer training narrows D-Muon/SOAP gains over AdamW; AdEMAMix leads. Rankings change with batch size and tuning; the wider study reaches 720M and measures loss, not downstream abilities. [Semenov et al. 2026](<https://arxiv.org/abs/2509.01440v1> "Benchmarking Optimizers for Large Language Model Pretraining; NeurIPS").
- **Diffusion optimizer evaluation** - Across seven optimizers on CelebA-64, denoising losses span 1.8% while FID spans 6.3-15.3. Three seeds per selected configuration; tuning uses shorter runs and some grid boundaries remain unexplored. Loss rankings do not track sample quality. [Bolatov et al. 2026](<https://arxiv.org/abs/2609.23055v1> "Optimizers for Diffusion Models: A Controlled Benchmark; arXiv").

## Software

- **SciPy optimize** - Python solvers for local and global optimization, constraints, linear programming, least squares, root finding, and curve fitting. Solver choice depends on problem structure. [docs](<https://docs.scipy.org/doc/scipy/reference/optimize.html>).
- **CVXPY** - Python modeling language for expressing optimization problems and passing them to numerical solvers. Supports convex modeling and additional structured problem classes. [docs](<https://www.cvxpy.org/>).
- **HiGHS** - Open-source solvers for sparse linear, mixed-integer, and quadratic programming. Provides a C++ library, command-line executable, and language interfaces. [docs](<https://highs.dev/>).
- **Optax** - Composable gradient transformations and optimization algorithms for JAX. Supports assembling custom optimizers from smaller update components. [docs](<https://optax.readthedocs.io/en/latest/>).
- **Sherpa** - Python hyperparameter optimization for expensive iterative models, with local or cluster execution and a visualization dashboard. [Hertel et al. 2020](<https://arxiv.org/abs/2005.04048> "Sherpa: Robust Hyperparameter Optimization for Machine Learning; SoftwareX").
- **pymoo** - Python framework for single- and multiobjective optimization, with constraints, customizable operators, parallel evaluation, and visualization. [Blank and Deb 2020](<https://arxiv.org/abs/2002.04504> "pymoo: Multi-objective Optimization in Python; IEEE Access").
- **ensmallen** - C++ numerical optimization library built around Armadillo. Includes gradient-based and derivative-free optimizers, with callbacks for monitoring and control. [docs](<https://ensmallen.org/>).

## Related topics

- [Neural Networks](https://mlpapers.org/neural-nets/)
- [AutoML](https://mlpapers.org/automl/)
- [Meta-Learning](https://mlpapers.org/meta-learning/)
- [Bayesian Inference](https://mlpapers.org/bayesian-inference/)
- [Reinforcement Learning](https://mlpapers.org/reinforcement-learning/)