# Neural Networks and Deep Learning ## Overviews and approximation - **Perceptron** - Rosenblatt's probabilistic model of recognition, learning, and memory. Studies statistical separability and the capabilities and limitations of early neural systems. [Rosenblatt 1958]( "The perceptron: A probabilistic model for information storage and organization in the brain.; Psychol. Rev."). - **Universal approximation** - Establishes approximation capacity for sufficiently wide feedforward networks with squashing activations. An existence result, not a guarantee of efficient training or generalization. [Hornik et al. 1989]( "Multilayer feedforward networks are universal approximators; Neural Networks"). - **Large multilayer perceptrons** - Studies deep, wide MLPs trained by online backpropagation on augmented handwritten digits using GPUs. Results concern MNIST and the specified training procedure. [Ciresan et al. 2010]( "Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition; Neural Comput."). - **Historical overview** - Surveys deep supervised, unsupervised, reinforcement, and evolutionary learning, organized around credit-assignment paths and their history. [Schmidhuber 2015]( "Deep Learning in Neural Networks: An Overview; Neural Networks"). - **Deep Learning** - Textbook covering feedforward, convolutional, recurrent, and generative networks; optimization, regularization, representation learning, and approximate inference. [Goodfellow et al. 2016]( "Deep Learning; book"). ## Feedforward and convolutional architectures - **GMDH polynomial networks** - Builds multilayer polynomial models, selecting useful combinations before constructing more complex ones. Uses an external criterion to control model complexity. [Ivakhnenko 1971]( "Polynomial Theory of Complex Systems; IEEE Trans. Syst., Man, Cybern."). - **GMDH algorithm families** - Reviews polynomial and nonparametric GMDH algorithms for forecasting, clustering, and system identification. [Ivakhnenko and Ivakhnenko 1995]( "The Review of Problems Solvable by Algorithms of the Group Method of Data Handling (GMDH); Pattern Recognit. Image Anal."). - **Neocognitron** - A self-organizing visual hierarchy of simple- and complex-cell-like layers, designed for pattern recognition tolerant of position changes. [Fukushima 1980]( "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position; Biol. Cybernetics"). - **Backpropagation for digit recognition** - Encodes task constraints in network architecture and learns the mapping from normalized handwritten digit images to classifications. An early end-to-end recognition system. [LeCun et al. 1989]( "Backpropagation Applied to Handwritten Zip Code Recognition; Neural Comput."). - **GPU-trained CNNs** - Presents a configurable GPU implementation of CNNs with supervised feature learning. Evaluates architecture variants on digit and object classification. [Cireşan et al. 2011]( "Flexible, High Performance Convolutional Neural Networks for Image Classification; IJCAI"). - **MLP-Mixer** - Alternates MLP operations across patches and across channels. Demonstrates an image-classification architecture built without convolution or attention. [Tolstikhin et al. 2021]( "MLP-Mixer: An all-MLP Architecture for Vision; NeurIPS"). - **CNN survey** - Reviews convolutional architectures, component choices, and applications across one-, two-, and higher-dimensional inputs. [Li et al. 2022]( "A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects; IEEE Transactions on Neural Networks and Learning Systems"). - **Kolmogorov-Arnold networks** - Learn spline functions on edges. At matched parameters or FLOPs, Yu et al. find MLPs generally better outside formula fitting; B-spline MLPs match or beat KANs on tested formulas. [Liu et al. 2025]( "KAN: Kolmogorov-Arnold Networks; ICLR"), [Yu et al. 2024]( "KAN or MLP: A Fairer Comparison; arXiv"). ## Residual connections - **ResNet** - Learns residual functions relative to layer inputs. Studies how this formulation enables optimization of substantially deeper image-recognition networks. [He et al. 2016]( "Deep Residual Learning for Image Recognition; CVPR"). - **Hyper-Connections** - Replaces a single residual stream with several streams and learned mixing between them. With four streams, an OLMo-7B comparison lowers validation loss and raises average task accuracy from 70.1% to 71.0% at nearly unchanged parameter count. [Zhu et al. 2025]( "Hyper-Connections; ICLR"). ## Sequence models, attention, and implicit depth ### Recurrent sequence models - **LSTM** - Uses memory cells and multiplicative gates to preserve error flow over long delays. The original design targets vanishing gradients in recurrent sequence learning. [Hochreiter and Schmidhuber 1997]( "Long Short-Term Memory; Neural Comput."). - **Bidirectional LSTM** - Combines forward and backward sequence context for framewise phoneme classification. Access to later inputs must match the deployment setting. [Graves and Schmidhuber 2005]( "Framewise phoneme classification with bidirectional LSTM and other neural network architectures; Neural Networks"). - **xLSTM** - Adds exponential gates and scalar or matrix memories to LSTM. Matrix memory supports parallel training. At 1.3B parameters on SlimPajama, extending 2K training contexts to 16K yields lower perplexity than tested Mamba and RWKV-4 baselines. [Beck et al. 2024]( "xLSTM: Extended Long Short-Term Memory; NeurIPS"). - **S4** - Uses a structured state-space parameterization to model long sequences efficiently, reducing its core computation to a Cauchy kernel. [Gu et al. 2022]( "Efficiently Modeling Long Sequences with Structured State Spaces; ICLR"). - **Mamba** - Makes state-space parameters depend on the input, allowing selective retention and forgetting. Uses a hardware-aware recurrent computation with linear scaling in sequence length. [Gu and Dao 2023]( "Mamba: Linear-Time Sequence Modeling with Selective State Spaces; arXiv"). - **Mamba-2** - Connects structured state-space models and attention variants through semiseparable matrices. Uses this duality to derive a revised selective state-space layer. [Dao and Gu 2024]( "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality; ICML"). - **RWKV** - Pairs parallelizable training with recurrent decoding whose state size is independent of context length. Across 12 NLP tasks, models up to 14B parameters achieve average accuracy comparable to Pythia, OPT and BLOOM at similar training compute. [Peng et al. 2023]( "RWKV: Reinventing RNNs for the Transformer Era; Findings of EMNLP"). - **DEER** - Evaluates nonlinear recurrences with parallel Newton solves over the sequence. Dense state Jacobians cost quadratic memory and cubic work in state width; speedups depend on sequence length, width and convergence. [Lim et al. 2024]( "Parallelizing non-linear sequential models over the sequence length; ICLR"). - **Quasi-DEER and ELK** - Quasi-Newton approximations reduce DEER's state-width cost; ELK uses a trust-region/Kalman-smoothing connection to stabilize iterations. Tested chaotic dynamics can still favor sequential evaluation. [Gonzalez et al. 2024]( "Towards Scalable and Stable Parallelization of Nonlinear RNNs; NeurIPS"). - **Zoology** - In matched Pile runs, a repeated-bigram slice accounts for 82% of the average log-loss gap between attention and H3, Hyena or RWKV-v4. Adding sparse, input-dependent attention closes most of the gap in tested convolutional hybrids. [Arora et al. 2024]( "Zoology: Measuring and Improving Recall in Efficient Language Models; ICLR"). - **BASED** - Combines linear attention, local attention and short convolutions. At about 1.3B parameters and 50B Pile tokens, it has similar perplexity to Mamba but improves document extraction and question answering. Full attention still leads on these tasks. [Arora et al. 2024]( "Simple linear attention language models balance the recall-throughput tradeoff; ICML"). ### Fast-weight and delta-rule memory - **Delta-rule memory** - Connects linear attention to fast-weight networks. Replaces additive key-value storage with corrections to the value already retrieved for a key, reducing interference in finite memory. Tests include associative recall, translation and language modeling. [Schlag et al. 2021]( "Linear Transformers Are Secretly Fast Weight Programmers; ICML"). - **Parallel DeltaNet** - Parallelizes delta-rule updates using products of Householder matrices. At 1.3B parameters and 100B tokens, it improves perplexity and average downstream scores over tested Mamba and GLA models. Attention hybrids further improve recall. [Yang et al. 2024]( "Parallelizing Linear Transformers with the Delta Rule over Sequence Length; NeurIPS"). - **Gated DeltaNet** - Combines scalar memory decay with delta-rule corrections, allowing rapid erasure and targeted updates. At 1.3B parameters and 100B tokens, it improves language modeling and retrieval over tested Mamba2 and DeltaNet models. [Yang et al. 2025]( "Gated Delta Networks: Improving Mamba2 with Delta Rule; ICLR"). - **Kimi Delta Attention** - Extends the gated delta rule with channel-wise memory decay. In 16-layer ablations at matched training FLOPs, three KDA layers per latent-attention layer give lower validation perplexity than all-attention, 1:1, 7:1 or 15:1 mixtures. [Kimi Team 2025]( "Kimi Linear: An Expressive, Efficient Attention Architecture; arXiv"). ### Attention - **Transformer** - Builds sequence transduction around attention, removing recurrence and convolution from the architecture. The original encoder-decoder is evaluated on translation and parsing. [Vaswani et al. 2017]( "Attention Is All You Need; NeurIPS"). - **Multi-query attention** - Shares one key and value head across query heads, reducing cache size and memory traffic during autoregressive decoding. Translation and language-model experiments find faster decoding with a small quality loss. [Shazeer 2019]( "Fast Transformer Decoding: One Write-Head is All You Need; arXiv"). - **Grouped-query attention** - Shares keys and values within groups of query heads. Converting T5 checkpoints and training for 5% of the original pretraining compute recovers quality near multi-head attention with decoding speed near multi-query attention. [Ainslie et al. 2023]( "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints; EMNLP"). - **Gated attention** - Output gates let heads suppress residual updates without extreme softmax scores. Training with gates reduces 8-bit quantization loss in BERT, OPT and ViT. For OPT-1.3B, quantized perplexity remains 29.95 versus 12.21 at full precision. [Bondarenko et al. 2023]( "Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing; NeurIPS"). - **Gating at language-model scale** - Per-dimension sigmoid gates add nonlinearity and query-dependent suppression between value and output projections. After YaRN extension, RULER-128K rises from 31.65 to 58.82. Native-32K scores are 79.50 and 79.77. [Qiu et al. 2025]( "Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free; NeurIPS"). - **Rotary position embeddings** - Encodes position through rotations of query and key vectors, making their dot products depend on relative position offsets; used inside attention rather than as added position vectors. [Su et al. 2024]( "RoFormer: Enhanced Transformer with Rotary Position Embedding; Neurocomputing"). - **p-RoPE** - RoPE need not make attention decay with distance. Gemma uses high frequencies for positional heads. At 8K context, Gemma-2B's perplexity slightly improves with the lowest quarter unrotated but worsens with three quarters unrotated. [Barbero et al. 2025]( "Round and Round We Go! What makes Rotary Positional Encodings useful?; ICLR"). - **Multi-head latent attention** - Compresses keys and values into a shared latent vector, with separate positional keys. DeepSeek-V2 ablations reduce cached elements while retaining comparable or better scores than multi-head attention on the four tested benchmarks. [DeepSeek-AI et al. 2024]( "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model; arXiv"). ### Recurrent depth and adaptive computation - **Adaptive Computation Time (ACT)** - Learns how many recurrent updates to perform before each output, with a penalty for extra computation. Improves tested algorithmic tasks, while character-level Wikipedia modelling shows little gain. [Graves 2016]( "Adaptive Computation Time for Recurrent Neural Networks; arXiv"). - **Universal Transformers** - Apply shared attention and transition layers repeatedly to all positions. Improve over Transformers on tested algorithmic and language tasks. Optional per-position halting helps structured tasks but slightly worsens translation. [Dehghani et al. 2019]( "Universal Transformers; ICLR"). - **MoEUT** - Combines shared layers with sparse experts. Tested 44M-1B language models reduce perplexity with fewer arithmetic operations than dense baselines. The reported implementation trains 1.5-2x slower than the dense FlashAttention baseline. [Csordás et al. 2024]( "MoEUT: Mixture-of-Experts Universal Transformers; NeurIPS"). - **Huginn** - Repeats a shared Transformer core before each token. A 3.5B model trained on 800B tokens improves math and coding scores with more iterations. The 180B-token comparison matches parameters and data, not training compute. [Geiping et al. 2025]( "Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach; NeurIPS"). - **Recurrence and compute** - In 116 FineWeb-Edu runs at 20 effective layers, shared loops save parameters but increase validation loss at matched training compute. The trade-off changes with gradient truncation and hyperconnections. [Schwethelm et al. 2026]( "How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models; arXiv"). - **Loop placement** - In dense models up to 1B scale, full-stack loops generally give lower language-model loss at fixed parameters and data. Middle-stack loops make better use of tight training-FLOP budgets. Gains saturate or reverse as more loops are added. [Ekbote et al. 2026]( "How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference; arXiv"). ### Continuous and implicit depth - **Neural ODEs** - Parameterizes hidden-state derivatives with a neural network and computes outputs with an ODE solver. Numerical tolerances and solver behavior become part of model execution. [Chen et al. 2018]( "Neural Ordinary Differential Equations; NeurIPS"). - **Deep equilibrium models** - Finds a weight-tied network's equilibrium by root finding and differentiates it implicitly. Replaces an explicit stack of layers with a fixed-point computation. [Bai et al. 2019]( "Deep Equilibrium Models; NeurIPS"). ### Architecture comparisons - **Architecture modification transfer** - Many published Transformer changes fail to improve a shared T5 setup. GLU activations and RMSNorm transfer more consistently across pretraining, fine-tuning and translation. Most comparisons hold hyperparameters fixed. [Narang et al. 2021]( "Do Transformer Modifications Transfer Across Implementations and Applications?; EMNLP"). - **bux architecture comparisons** - In small code/web models, fine-grained MoE improves loss per active parameter. Sharing attention keys and values trades some quality for cheaper serving. Equal-serving-cost rankings extrapolate from small models to much larger ones. [Pavlichenko 2026]( "2020 to 2026, one change at a time; online study"); [code](). ## Reservoir computing - **Echo state networks** - Drives a fixed recurrent reservoir with inputs and fits a linear readout. The echo-state property makes states determined by input history as initialization effects fade; readout fitting avoids training recurrent weights. [Jaeger 2001]( "The echo state approach to analysing and training recurrent neural networks; GMD technical report"). - **Liquid state machines** - Uses transient responses of a recurrent spiking circuit as temporal features for a trained readout. Its approximation results require both separation by the liquid and sufficient readout capacity. [Maass et al. 2002]( "Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations; Neural Comput."). - **Delay-based reservoirs** - Time-multiplexes inputs through one nonlinear node with delayed feedback, creating virtual reservoir nodes. Reduces physical component count while processing those virtual nodes serially. [Appeltant et al. 2011]( "Information processing using a single dynamical node as complex system; Nat. Commun."). - **Information-processing capacity** - Measures linear-readout reconstruction of orthogonal functions of input history. With i.i.d. inputs and finite fourth moments, the infinite-data sum of capacities is bounded by the number of observed state variables. [Dambre et al. 2012]( "Information Processing Capacity of Dynamical Systems; Sci. Rep."). - **Next-generation reservoir computing** - Fits ridge regression on delayed observations and polynomial features, replacing a random recurrent reservoir. Demonstrates forecasting and attractor reconstruction on Lorenz63 and simulated double-scroll dynamics. [Gauthier et al. 2021]( "Next Generation Reservoir Computing; Nat. Commun."). ## Learning operators between functions - **Operator learning** - See [Scientific ML](https://mlpapers.org/scientific-machine-learning/#learning-operators-and-numerical-updates) for DeepONet, Fourier neural operators and their numerical assumptions. ## Conditional computation - **Sparsely gated mixture of experts** - A learned gate selects a sparse combination of feedforward experts per example. Separates total parameter capacity from the subset activated for each input. [Shazeer et al. 2017]( "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer; ICLR"). - **Switch Transformer** - Simplifies mixture-of-experts routing and addresses communication costs and training instability. Studies sparse conditional computation in language models. [Fedus et al. 2022]( "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity; JMLR"). - **DeepSeekMoE** - Splits experts into smaller units and keeps shared experts active for every token. At matched total and active parameters, 2B-scale ablations find finer routing and shared experts improve most tested benchmarks over coarse top-2 routing. [Dai et al. 2024]( "DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models; ACL"). - **Mixture-of-experts scaling** - See [NLP](https://mlpapers.org/nlp/#scaling-and-in-context-learning) for dense/MoE comparisons at matched parameter and compute budgets, including the trade-off between memory and training tokens. ## Unsupervised and generative networks - **Representation learning** - See [Feature Extraction](https://mlpapers.org/feature-extraction/#learned-representations) for competitive learning, autoencoders and self-supervised objectives. - **Generative networks** - See [Generative Models](https://mlpapers.org/generative-models/) for Boltzmann machines, latent-variable models, autoregressive networks and GANs. - **Sequence generation and forecasting** - TimeGAN generates sequences in [Generative Models](https://mlpapers.org/generative-models/#adversarial-generation). ForGAN predicts future values in [Forecasting](https://mlpapers.org/time-series-forecasting/#global-neural-and-probabilistic-models). ## Bayesian neural networks and uncertainty - **Bayesian neural inference** - See [Bayesian Inference](https://mlpapers.org/bayesian-inference/#neural-network-and-gaussian-process-uncertainty) for neural priors, approximate posteriors and predictive uncertainty. ## Activations and normalization - **Rectified units in RBMs** - Uses noisy rectified linear units to approximate replicated binary units in restricted Boltzmann machines, preserving information about relative input intensity. [Nair and Hinton 2010]( "Rectified Linear Units Improve Restricted Boltzmann Machines; ICML"). - **Deep rectifier networks** - Studies rectifying activations that produce exact zeros, including supervised learning without mandatory unsupervised pretraining. [Glorot et al. 2011]( "Deep Sparse Rectifier Neural Networks; AISTATS"). - **Leaky rectifiers** - Retains a nonzero gradient for negative inputs. Compares rectifier variants and sigmoid-based acoustic models on conversational speech recognition. [Maas et al. 2013]( "Rectifier Nonlinearities Improve Neural Network Acoustic Models; ICML WDLASL Workshop"). - **Maxout** - Takes the maximum over a set of inputs to form a learnable nonlinear unit. Designed and evaluated in conjunction with dropout. [Goodfellow et al. 2013]( "Maxout Networks; ICML"). - **PReLU and rectifier initialization** - Introduces a parametric rectifier and an initialization derived for rectifier networks. Studies their use in deep image classification. [He et al. 2015]( "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification; ICCV"). - **Randomized leaky ReLU** - Randomizes the negative slope during training. Compares standard, leaky, parametric, and randomized rectifiers on image-classification tasks. [Xu et al. 2015]( "Empirical Evaluation of Rectified Activations in Convolutional Network; arXiv"). - **Batch normalization** - Normalizes layer inputs using mini-batch statistics as part of the architecture. Studies effects on training, learning-rate choice, and regularization. [Ioffe and Szegedy 2015]( "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift; ICML"). - **Why batch normalization helps** - Challenges internal covariate shift as the main explanation and analyzes smoother optimization behavior. An analysis of mechanism, not a universal account of every normalization setup. [Santurkar et al. 2018]( "How Does Batch Normalization Help Optimization?; NeurIPS"). - **ELU** - Uses an identity branch for positive inputs and a saturating negative branch. Motivated by shifting average activations closer to zero. [Clevert et al. 2016]( "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs); ICLR"). - **Parametric ELU** - Learns the shape of the exponential-linear activation instead of fixing its parameters. Different layers can learn different nonlinear responses. [Trottier et al. 2017]( "Parametric Exponential Linear Unit for Deep Convolutional Neural Networks; ICMLA"). - **S-shaped ReLU** - Learns a three-segment piecewise-linear activation with four parameters, jointly with the network. Can represent convex and nonconvex shapes. [Jin et al. 2016]( "Deep Learning with S-shaped Rectified Linear Activation Units; AAAI"). - **GELU** - Weights an input by the standard Gaussian cumulative probability at that value. Provides a smooth alternative to ReLU's sign-based gating. [Hendrycks and Gimpel 2016]( "Gaussian Error Linear Units (GELUs); arXiv"). - **Layer normalization** - Computes normalization statistics across a layer for each example, with learned gain and bias. Avoids dependence on other examples in a mini-batch. [Ba et al. 2016]( "Layer Normalization; arXiv"). - **RMSNorm** - Normalizes activations by their root mean square with learned rescaling, omitting LayerNorm's mean subtraction; preserves rescaling invariance without recentering activations. [Zhang and Sennrich 2019]( "Root Mean Square Layer Normalization; NeurIPS"). - **Swish** - Searches activation functions and identifies input times sigmoid of a scaled input. The scale parameter generalizes the SiLU form. [Ramachandran et al. 2017]( "Searching for Activation Functions; arXiv"). - **SELU** - Uses scaled exponential-linear activations to obtain self-normalizing behavior under the paper's assumptions. The result is not a guarantee for arbitrary architectures or initialization. [Klambauer et al. 2017]( "Self-Normalizing Neural Networks; NeurIPS"). - **SiLU** - Multiplies an input by its sigmoid. The paper studies SiLU and its derivative as activations for reinforcement-learning function approximation. [Elfwing et al. 2018]( "Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning; Neural Networks"). - **Mish** - Uses input times tanh of softplus, producing a smooth nonmonotonic activation. Evaluated across image-classification and detection architectures. [Misra 2020]( "Mish: A Self Regularized Non-Monotonic Activation Function; BMVC"). - **Gated feedforward layers** - Compares GLU variants in Transformer feedforward sublayers. Combines two projections by elementwise multiplication, with a nonlinearity applied to one branch. [Shazeer 2020]( "GLU Variants Improve Transformer; arXiv"). ## Training, regularization, and generalization - **Unsupervised pretraining** - Tests how pretraining affects optimization and generalization. Experiments support a regularization interpretation through the basins reached during supervised training. [Erhan et al. 2010]( "Why Does Unsupervised Pre-training Help Deep Learning?; JMLR"). - **Dropout** - Randomly omits feature detectors during training to reduce co-adaptation. Trains shared weights across many perturbed network configurations. [Hinton et al. 2012]( "Improving neural networks by preventing co-adaptation of feature detectors; arXiv"). - **Adaptive dropout** - Learns a binary belief network that determines which hidden activities to suppress. The dropout mechanism is trained jointly with the predictive network. [Ba and Frey 2013]( "Adaptive dropout for training deep neural networks; NeurIPS"). - **Fast dropout** - Approximates dropout's random contributions with a Gaussian distribution, sampling or integrating that approximation instead of repeatedly sampling unit masks. [Wang and Manning 2013]( "Fast dropout training; ICML"). - **Dropout analysis** - Analyzes Bernoulli gating, ensemble averaging, and regularization in linear and nonlinear networks. Distinguishes exact identities from approximations used for nonlinear activations. [Baldi and Sadowski 2014]( "The Dropout Learning Algorithm; Artificial intelligence"). - **Differentially private SGD** - Clips each example's gradient, adds Gaussian noise and accounts for privacy loss over training. Abadi et al. trade accuracy against a stated privacy budget; the protected unit here is an example, not automatically an entire user. [Abadi et al. 2016]( "Deep Learning with Differential Privacy; CCS"). - **Adversarial training** - Optimizes prediction loss against worst-case perturbations in a specified threat set. Madry et al. approximate the inner attack with projected gradients; resistance to tested attacks is not a certificate against every possible perturbation. [Madry et al. 2018]( "Towards Deep Learning Models Resistant to Adversarial Attacks; ICLR"). - **Neural tangent kernel** - Describes function evolution during gradient training through a kernel. In the specified infinite-width limit the kernel becomes fixed; finite feature-learning networks need not follow that regime. [Jacot et al. 2018]( "Neural Tangent Kernel: Convergence and Generalization in Neural Networks; NeurIPS"). - **Lazy training** - Shows how parameter scaling can make training resemble a model's linearization at initialization. Its kernel-like behavior is not a complete explanation of neural feature learning. [Chizat et al. 2019]( "On Lazy Training in Differentiable Programming; NeurIPS"). - **Double descent** - Documents nonmonotonic test error as model size or training duration increases. The observed interpolation regimes qualify simple claims that more capacity always helps or hurts. [Nakkiran et al. 2020]( "Deep Double Descent: Where Bigger Models and More Data Hurt; ICLR"). - **Language-model scaling** - See [NLP](https://mlpapers.org/nlp/#scaling-and-in-context-learning) for model/data allocation, Chinchilla and evidence about data filtering. - **Softmax collapse** - At high confidence, rounding can erase the correct-class gradient while others remain nonzero; under Adam this can trigger Slingshot spikes. A reformulation of cross-entropy in fp32 matches standard cross-entropy in fp64 on tested modular-arithmetic tasks. [Prieto et al. 2025]( "Grokking at the Edge of Numerical Stability; ICLR"), [Liu et al. 2026]( "Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes; NeurIPS (accepted)"), [Barron et al. 2026]( "A Fix for Softmax and Cross-Entropy; arXiv"). - **Time-bomb loss spikes** - Adam shifts logits together, making BF16 lose token-score differences. In 28M language models trained for 16B tokens at high learning rate without weight decay, FP32 output stages or tuned momentum SGD on the head prevent delayed divergence. [Chen et al. 2026]( "The Role of Quantization in Loss Spikes; arXiv"); [code](). ## Credit assignment and alternatives to backpropagation - **Backpropagation** - Applies the chain rule backward through network layers to compute weight gradients. Credit assignment determines the learning signal; an optimizer such as SGD or Adam uses that signal to update weights. [Rumelhart et al. 1986]( "Learning representations by back-propagating errors; Nature"). ### Perturbation-based learning - **Node versus weight perturbation** - Uses loss changes after perturbing activations or weights. In linear teacher-student networks, maximum stable learning rates scale inversely with noise dimension. [Werfel et al. 2003]( "Learning curves for stochastic gradient descent in linear feedforward networks; NeurIPS"). - **MeZO** - Replays [SPSA-style](https://mlpapers.org/optimization/#perturbation-based-gradient-estimation) weight perturbations from seeds, enabling fine-tuning with inference-level memory. Under a local low-effective-rank Hessian bound, its per-step slowdown depends on effective rank rather than parameter count. [Malladi et al. 2023]( "Fine-Tuning Language Models with Just Forward Passes; NeurIPS"). - **EGGROLL** - Uses low-rank factors for weight-space perturbations in evolution strategies, lowering perturbation storage and evaluation cost versus full-rank search. Population averaging can produce higher-rank updates. [Sarkar et al. 2025]( "Evolution Strategies at the Hyperscale; arXiv"). - **Dust** - Applies node perturbation per token in transformers. Large populations give lower loss than backprop with SGD at 100k and 1M tokens; with Adam at 1M, they approach the backprop baseline. Extra draws cost more compute. [Dahal et al. 2026]( "Dust: Pretraining Transformers Without Backpropagation; Q Labs report"). ### Feedback signals and learned targets - **Difference target propagation** - Learns approximate inverse maps to send activation targets backward. A difference correction compensates for imperfect inverses; local fitting and the original top-layer update still use derivatives. [Lee et al. 2015]( "Difference Target Propagation; ECML PKDD"). - **Feedback alignment** - Sends errors through fixed random feedback weights instead of transposed forward weights, retaining local activation derivatives. [Lillicrap et al. 2016]( "Random feedback weights support learning in deep neural networks; Nature Communications"). - **Direct feedback alignment** - Projects output errors directly to each hidden layer through fixed random weights, bypassing feedback alignment's layer-by-layer error relay; updates retain local activation derivatives. [Nøkland 2016]( "Direct Feedback Alignment Provides Learning in Deep Neural Networks; NeurIPS"). - **Synthetic gradients** - Learns predictors of downstream error gradients so modules can update without waiting for downstream computation. Gradient predictors need training signals; this decouples backpropagation rather than eliminating it. [Jaderberg et al. 2017]( "Decoupled Neural Interfaces using Synthetic Gradients; ICML"). ### Local objectives, forward differentiation, and equilibrium learning - **Equilibrium propagation** - Contrasts free and weakly nudged equilibria in an energy-based network. Under smoothness and fixed-point assumptions, the update approaches the objective gradient as nudging vanishes; settling has a computation cost. [Scellier and Bengio 2017]( "Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation; Frontiers in Computational Neuroscience"). - **Equilibrium propagation at ImageNet scale** - Centered nudging with a predictive-coding VGG10 reaches 13.23% top-5 error versus 12.20% for backprop after 50 epochs. Iterative equilibration is required; the result establishes accuracy at this scale, not equal training cost. [Kerjan et al. 2026]( "Training a Predictive Coding Network on ImageNet using Equilibrium Propagation; NeurIPS"). - **Forward-Forward** - Each layer learns to raise a goodness score on positive data and lower it on negative data. Replaces end-to-end error propagation with local objectives; negative-example construction is part of the method. [Hinton 2022]( "The Forward-Forward Algorithm: Some Preliminary Investigations; arXiv"). - **Forward gradients** - Forward-mode autodiff computes a directional derivative; multiplying by its random direction gives an unbiased estimate for independent zero-mean, unit-variance tangent components. Removes reverse-mode differentiation, not derivatives. [Baydin et al. 2022]( "Gradients without Backpropagation; arXiv"). - **Local forward gradients** - Ren et al. reduce estimator variance with activity perturbations and local greedy losses. Forward-mode autodiff supplies the directional derivatives; ImageNet performance improves over tested alternatives but still trails backprop. [Ren et al. 2023]( "Scaling Forward Gradient With Local Losses; ICLR"). ### Comparative evidence - **Scaling evidence** - Bartunov et al. find tested target-propagation and feedback-alignment variants far behind backprop on ImageNet. Results concern their architectures and tuning; direct feedback alignment exceeded their 16GB GPU memory. [Bartunov et al. 2018]( "Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures; NeurIPS"). - **Noise** - A benchmark for training transformers without end-to-end backpropagation. Methods train the same 37.7M-parameter model on 10M tokens, competing on final loss or time to a target loss. The rules allow exact credit through linear maps. [code](). ## Pruning, quantization, and efficient execution - **Optimal Brain Damage** - Uses second-derivative information to remove weights while balancing network complexity against training error. [LeCun et al. 1989]( "Optimal Brain Damage; NeurIPS"). - **Knowledge distillation** - Transfers a teacher model or ensemble's predictions into a single model. Separates the cost of producing training targets from the cost of deploying the student. [Hinton et al. 2014]( "Distilling the Knowledge in a Neural Network; NeurIPS DL Workshop"). - **Weightless neural systems** - Reviews RAM-based neurons addressed by Boolean inputs, including WiSARD and related architectures. Covers capacity analysis, extensions, and applications. [França et al. 2014]( "Advances in Weightless Neural Systems; ESANN"). - **Prune and retrain** - Trains a dense network, removes unimportant connections, then fine-tunes remaining weights. Reduced parameter counts do not alone establish hardware speedups. [Han et al. 2015]( "Learning both Weights and Connections for Efficient Neural Networks; NeurIPS"). - **Binarized networks** - Trains networks with binary weights and activations at runtime, replacing many arithmetic operations with bitwise ones. Actual efficiency depends on implementation and hardware. [Courbariaux et al. 2016]( "Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1; arXiv"). - **Activation checkpointing** - Stores selected forward activations and recomputes intermediate values during backpropagation, trading additional computation for lower training memory. [Chen et al. 2016]( "Training Deep Nets with Sublinear Memory Cost; arXiv"). - **Taylor-based filter pruning** - Approximates the loss change from pruning with a Taylor criterion and interleaves removal with fine-tuning. Evaluated for adapted convolutional networks and classification. [Molchanov et al. 2017]( "Pruning Convolutional Neural Networks for Resource Efficient Inference; ICLR"). - **Compact binary-network training** - Studies low learning rates, PReLU, bipolar regularization, output scaling, and multiple activation binarizations to improve the compression-accuracy trade-off. [Tang et al. 2017]( "How to Train a Compact Binary Neural Network with High Accuracy?; AAAI"). - **L0 regularization** - Uses differentiable expectations over stochastic gates to encourage exact zeros during training. The hard-concrete relaxation enables gradient-based structure learning. [Louizos et al. 2018]( "Learning Sparse Neural Networks through $L_0$ Regularization; ICLR"). - **Lottery ticket hypothesis** - Finds sparse subnetworks whose original initializations permit effective retraining. The experiments support an initialization-sensitive hypothesis, not a universal pruning guarantee. [Frankle and Carbin 2019]( "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks; ICLR"). - **KD-Lib** - A PyTorch library collecting knowledge distillation, pruning, and quantization methods, with support for tuning and experiment logging. [Shah et al. 2020]( "KD-Lib: A PyTorch library for Knowledge Distillation, Pruning and Quantization; arXiv"). - **Parameter-efficient adaptation** - See [Transfer Learning](https://mlpapers.org/transfer-learning/#parameter-efficient-adaptation) for LoRA, QLoRA and comparisons with full fine-tuning. - **FlashAttention** - Tiles exact attention to reduce traffic between GPU memory levels. Changes execution and memory use without replacing dense attention by an approximate attention rule. [Dao et al. 2022]( "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness; NeurIPS"). - **GPTQ** - Uses approximate second-order information for one-shot post-training weight quantization of large Transformers. Low-bit accuracy and speed depend on the model and inference implementation. [Frantar et al. 2023]( "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers; ICLR"). - **AWQ** - Uses activation statistics to identify and scale important weight channels before low-bit weight-only quantization. Avoids backpropagation during calibration. [Lin et al. 2024]( "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration; MLSys"). - **ReplaceMe** - Replaces transformer blocks with linear maps fitted to calibration activations; compatible maps merge into existing projections. Least-squares fitting is closed form; cosine-distance fitting uses optimization. [Shopkhoev et al. 2025]( "ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization; NeurIPS"). - **MoE-to-dense compression** - Selects experts, concatenates their feedforward blocks and distills a dense student. Diversity-aware selection beats tested selection/grouping alternatives across three MoEs; benefits vary with expert pool size and distillation budget. [Kim et al. 2026]( "Pruning and Distilling Mixture-of-Experts into Dense Language Models; NeurIPS"). ## Ensembles - **Neural ensembles** - See [Ensemble Learning](https://mlpapers.org/ensemble-learning/#neural-ensembles) for combination methods, disagreement and accuracy-compute comparisons. ## Related topics - [Optimization](https://mlpapers.org/optimization/) - [Ensemble Learning](https://mlpapers.org/ensemble-learning/) - [Bayesian Inference](https://mlpapers.org/bayesian-inference/) - [Reinforcement Learning](https://mlpapers.org/reinforcement-learning/) - [Interpretability](https://mlpapers.org/interpretability/) - [NLP](https://mlpapers.org/nlp/) - [Computer Vision](https://mlpapers.org/computer-vision/) - [Generative Models](https://mlpapers.org/generative-models/) - [Graph Neural Networks](https://mlpapers.org/graph-neural-networks/)