# Reinforcement Learning ## Temporal differences and action values - **Temporal-difference learning** - Learns predictions from differences between successive predictions, rather than waiting only for final outcomes. Establishes convergence and optimality results for specified cases. [Sutton 1988]( "Learning to predict by the methods of temporal differences; Mach. Learn."). - **Q-learning** - Incrementally estimates action values from experience. Proves convergence under the paper's tabular representation, sampling, and convergence assumptions; this is not a deep-network convergence guarantee. [Watkins and Dayan 1992]( "Q-learning; Mach. Learn."). - **DQN** - Combines convolutional visual representations with Q-learning to estimate future reward from game pixels. The original preprint studies seven Atari games; later DQN evaluations use different protocols and game sets. [Mnih et al. 2013]( "Playing Atari with Deep Reinforcement Learning; NeurIPS DL Workshop"). - **Double DQN** - Separates action selection from action evaluation in the Q-learning target. Reduces overestimation arising from using the same estimates to select and evaluate an action. [van Hasselt et al. 2016]( "Deep Reinforcement Learning with Double Q-learning; AAAI"). - **Dueling networks** - Splits the action-value network into state-value and action-advantage estimators. Shares learning across actions without requiring a different underlying reinforcement-learning algorithm. [Wang et al. 2016]( "Dueling Network Architectures for Deep Reinforcement Learning; ICML"). - **Rainbow** - Combines six DQN extensions and measures their contributions through ablations. Demonstrates that improvements need to be tested jointly; isolated gains need not simply add together. [Hessel et al. 2018]( "Rainbow: Combining Improvements in Deep Reinforcement Learning; AAAI"). - **Prioritized experience replay** - Samples selected stored transitions more often instead of replaying all experience uniformly. Changes which past interactions drive learning, rather than how the agent collects new data. [Schaul et al. 2016]( "Prioritized Experience Replay; ICLR"). - **Replay volume and recency** - In continuous-control off-policy RL, recency-biased replay helps most when few transitions are replayed per environment step. Equal-recency comparisons also favor higher-entropy sampling; the study does not cover Atari. [Korniak et al. 2026]( "When Does Non-Uniform Replay Matter in Reinforcement Learning?; NeurIPS"). ## Return distributions - **Distributional reinforcement learning** - Models the distribution of random returns instead of only their expectation. Develops distributional Bellman analysis and a practical approximate-distribution algorithm, while identifying instability in the control setting. [Bellemare et al. 2017]( "A Distributional Perspective on Reinforcement Learning; ICML"). - **Quantile regression DQN** - Represents the return distribution through quantiles and learns them with quantile regression. Connects approximate distribution learning to a distributional Bellman formulation. [Dabney et al. 2018]( "Distributional Reinforcement Learning with Quantile Regression; AAAI"). ## Policy gradients and actor–critic methods - **REINFORCE** - Derives stochastic-network updates aligned with the gradient of expected reinforcement. Covers immediate rewards and specified delayed-reward settings, and connects the updates with backpropagation. [Williams 1992]( "Simple statistical gradient-following algorithms for connectionist reinforcement learning; Mach. Learn."). - **TRPO** - Optimizes a policy surrogate with a constraint on policy divergence. The practical algorithm approximates a theoretically justified update; monotonic improvement is not an unconditional implementation guarantee. [Schulman et al. 2015]( "Trust Region Policy Optimization; ICML"). - **Generalized advantage estimation** - Uses an exponentially weighted advantage estimator to reduce policy-gradient variance at the cost of bias. Connects advantage estimation to temporal-difference learning with eligibility traces. [Schulman et al. 2016]( "High-Dimensional Continuous Control Using Generalized Advantage Estimation; ICLR"). - **A3C** - Runs parallel actor-learners with asynchronous parameter updates. Studies how diverse concurrent experience stabilizes deep value and actor-critic learning. [Mnih et al. 2016]( "Asynchronous Methods for Deep Reinforcement Learning; ICML"). - **UNREAL** - Adds pixel control, reward prediction and value replay to A3C. On 13 Labyrinth tasks, the authors report a 10x mean reduction in steps needed to reach A3C's final performance. [Jaderberg et al. 2017]( "Reinforcement Learning with Unsupervised Auxiliary Tasks; ICLR"). - **DDPG** - Uses a deterministic policy gradient within an actor-critic architecture for continuous actions. Extends deep Q-learning ideas to settings where enumerating all actions is impractical. [Lillicrap et al. 2016]( "Continuous control with deep reinforcement learning; ICLR"). - **PPO** - Alternates interaction with minibatch optimization of a policy surrogate. Its clipped-ratio variant limits incentives for large updates; clipping is not a hard bound on policy divergence. [Schulman et al. 2017]( "Proximal Policy Optimization Algorithms; arXiv"). - **TD3** - Uses the minimum of two critic estimates and delays policy updates to reduce effects of approximation error. Addresses overestimated values in continuous-action actor-critic learning. [Fujimoto et al. 2018]( "Addressing Function Approximation Error in Actor-Critic Methods; ICML"). - **Soft Actor-Critic** - Optimizes reward together with policy entropy using an off-policy stochastic actor-critic. The entropy term encourages varied actions while the critic estimates task value. [Haarnoja et al. 2018]( "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor; ICML"). - **SAC temperature adaptation** - Extends Soft Actor-Critic with a constrained formulation that tunes the entropy temperature automatically. Studies sensitivity to settings and applications in simulated and physical control. [Haarnoja et al. 2018]( "Soft Actor-Critic Algorithms and Applications; arXiv"). - **Control as inference** - Connects maximum-entropy control with probabilistic inference. Distinguishes exact inference under deterministic dynamics from a variational treatment under stochastic dynamics. [Levine 2018]( "Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review; arXiv"). ## Exploration and goal relabeling - **Density-based pseudo-counts** - Derives visitation-like counts from density models to extend count-based exploration beyond tabular states. Converts estimated novelty into intrinsic rewards. [Bellemare et al. 2016]( "Unifying Count-Based Exploration and Intrinsic Motivation; NeurIPS"). - **Intrinsic curiosity module** - Uses prediction error in an inverse-dynamics-trained feature space as an intrinsic reward. Encourages exploration through failures to predict action consequences rather than raw pixel reconstruction alone. [Pathak et al. 2017]( "Curiosity-driven Exploration by Self-supervised Prediction; ICML"). - **Hindsight experience replay** - Reuses unsuccessful goal-conditioned episodes by relabeling them with goals actually achieved. Recomputes the associated rewards to learn from sparse binary feedback. [Andrychowicz et al. 2017]( "Hindsight Experience Replay; NeurIPS"). - **NoisyNet** - Adds parameterized noise to network weights and learns its parameters with the task objective. Produces exploratory policy variation through the network rather than only random action replacement. [Fortunato et al. 2018]( "Noisy Networks for Exploration; ICLR"). - **Random network distillation** - Rewards prediction error against features from a fixed random network. Introduces a novelty signal and a mechanism for combining intrinsic and task rewards. [Burda et al. 2019]( "Exploration by Random Network Distillation; ICLR"). ## Goal-conditioned representations and planning - **Search on the Replay Buffer** - Builds a graph of visited states with goal-value-based distance estimates, then searches for subgoals a local policy can reach. Planning depends on replay coverage and distance accuracy; evaluated on image navigation. [Eysenbach et al. 2019]( "Search on the Replay Buffer: Bridging Planning and Reinforcement Learning; NeurIPS"). - **Contrastive RL** - Contrasts future states with random states for each state-action pair. The optimal critic encodes discounted future-state occupancy ratios; maximizing its score trains a goal-reaching policy. [Eysenbach et al. 2022]( "Contrastive Learning as Goal-Conditioned Reinforcement Learning; NeurIPS"). - **HIQL** - Learns an offline goal-value function, a high-level policy predicting latent subgoals and a low-level controller. Extends IQL to hierarchical goal reaching; value learning can use action-free trajectories, while controller extraction needs actions. [Park et al. 2023]( "HIQL: Offline Goal-Conditioned RL with Latent States as Actions; NeurIPS"). - **Inference via Interpolation** - Its temporal contrastive parameterization yields a Gauss-Markov chain under Gaussian-marginal and optimal-critic assumptions. Infers latent waypoints from offline trajectories; no online control-policy learning. [Eysenbach et al. 2024]( "Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference; NeurIPS"). - **C-Slerp** - Spherical interpolation of contrastive embeddings supplies adaptive waypoints, improving success in 8-tower and Ant U5 Maze simulations. Its derivation assumes a Bayes-optimal critic, uniform spherical marginals and a small-variance approximation. [Park et al. 2026]( "Hyperspherical Contrastive Representations Unlock Planning and Exploration; arXiv"); [code](). - **Skill objectives and goal criteria** - Repeated occupancy, exact-time arrival and arrival within a window can require different policies. Finite-state theory links skill-information objectives to matching goal criteria; no one diversity objective optimizes all of them. [Modirshanechi et al. 2026]( "Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization; NeurIPS"). ## Memory and temporal abstraction - **Options and Option-Critic** - Options specify initiation, an internal policy and termination. Option-Critic learns option policies and termination functions by gradient methods, enabling temporal abstraction. [Sutton et al. 1999]( "Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning; Artificial Intelligence"), [Bacon et al. 2017]( "The Option-Critic Architecture; AAAI"). - **Deep recurrent Q-networks** - Adds an LSTM to DQN to integrate observations over time. Studies partial observability and changing observation quality; recurrence does not show a systematic advantage on every fully observed task. [Hausknecht and Stone 2015]( "Deep Recurrent Q-Learning for Partially Observable MDPs; arXiv"). - **Successor features** - Separates expected future feature occupancy from reward weights, enabling policy transfer across tasks with shared dynamics and different rewards; paired with generalized policy improvement. [Barreto et al. 2017]( "Successor Features for Transfer in Reinforcement Learning; NeurIPS"). - **HIRO** - Learns high-level goals and lower-level goal-conditioned control using off-policy data. Corrects higher-level replay for changes in the lower-level behavior that alter what a goal accomplishes. [Nachum et al. 2018]( "Data-Efficient Hierarchical Reinforcement Learning; NeurIPS"). - **DIAYN** - Learns diverse skills without task rewards using an information-theoretic objective and maximum-entropy policies. Studies skill pretraining and hierarchical composition for later tasks. [Eysenbach et al. 2019]( "Diversity is All You Need: Learning Skills without a Reward Function; ICLR"). ## Search and self-play - **AlphaZero** - Combines neural policy/value prediction, Monte Carlo tree search, and self-play for board games. Uses the game rules for planning; learning without human games does not mean learning without a simulator. [Silver et al. 2017]( "Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm; arXiv"). ## Learned dynamics and planning ### Learning from imagined experience - **Linear Dyna and prioritized sweeping** - Learns from imagined transitions generated by a world model. Extends Dyna planning and prioritized sweeping to linear function approximation, with convergence results under stated conditions. [Sutton et al. 2008]( "Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping; UAI"). - **Model-Based Policy Optimization** - Uses short model-generated rollouts branching from real observations. Limits reliance on long simulated trajectories, where accumulated model error can undermine policy optimization. [Janner et al. 2019]( "When to Trust Your Model: Model-Based Policy Optimization; NeurIPS"). - **Dreamer** - Learns behavior by propagating value gradients through imagined trajectories in a compact latent world model. Separates learning environmental dynamics from collecting every policy update through real interaction. [Hafner et al. 2020]( "Dream to Control: Learning Behaviors by Latent Imagination; ICLR"). - **DreamerV2** - Uses discrete latent world-model representations and learns behavior from model predictions. Trains the world model separately from the policy and studies both Atari and continuous control. [Hafner et al. 2021]( "Mastering Atari with Discrete World Models; ICLR"). - **DreamerV3** - Combines latent imagination with normalization, balancing, and transformation choices intended to stabilize learning across domains. Evaluates a shared configuration across diverse tasks. [Hafner et al. 2025]( "Mastering Diverse Domains through World Models; Nature"). ### Task-relevant models for planning - **Value Prediction Network** - Predicts rewards, discounts and values along abstract rollouts without reconstructing observations. In stochastic Collect gridworlds, it outperforms the tested observation-prediction model. [Oh et al. 2017]( "Value Prediction Network; NeurIPS"). - **MuZero** - Learns latent dynamics from reward, value and search-policy targets for tree search. AlphaZero instead uses known game rules. MuZero requires neither pixel reconstruction nor matching the encoding of a future observation. [Schrittwieser et al. 2020]( "Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model; Nature"). - **EfficientZero** - Adds a SimSiam-style loss matching predicted latents to encodings of future observations. Removing it cuts median human-normalized score from 1.16 to 0.34 across 26 Atari 100k games. [Ye et al. 2021]( "Mastering Atari Games with Limited Data; NeurIPS"). - **Value equivalence** - Models can produce identical Bellman updates for specified policies and candidate value functions despite different transitions. Expanding these sets reduces which dynamics the model can ignore. [Grimm et al. 2020]( "The Value Equivalence Principle for Model-Based Reinforcement Learning; NeurIPS"). - **Proper value equivalence** - In discounted MDPs, models matching every deterministic policy's value at every state preserve optimal planning despite different dynamics. MuZero's current-policy loss alone does not ensure this. [Grimm et al. 2021]( "Proper Value Equivalence; NeurIPS"). - **TD-MPC / TD-MPC2** - [Continuous-control planning](https://mlpapers.org/model-predictive-control/#latent-planning-and-comparative-evidence) with latent models and terminal value estimates. TD-MPC couples reward and value prediction to matching future encodings. - **Uncertainty-aware receding-horizon control** - See [MPC](https://mlpapers.org/model-predictive-control/#learning-dynamics-costs-and-feasible-behavior) for PETS and probabilistic dynamics ensembles. ## Learning from fixed datasets - **Batch-Constrained Q-learning** - Restricts candidate actions to remain close to those supported by a fixed dataset. Addresses extrapolation errors that arise when offline policy learning evaluates poorly supported actions. [Fujimoto et al. 2019]( "Off-Policy Deep Reinforcement Learning without Exploration; ICML"). - **Behavior-regularized actor-critic** - Organizes offline RL around regularizing policy behavior relative to logged data. Uses shared comparisons and ablations to examine which algorithmic choices matter. [Wu et al. 2019]( "Behavior Regularized Offline Reinforcement Learning; arXiv"). - **Conservative Q-learning** - Adds a value regularizer to Bellman-error training to learn conservative estimates from offline data. Lower-bound results have assumptions; conservative training does not guarantee deployment performance. [Kumar et al. 2020]( "Conservative Q-Learning for Offline Reinforcement Learning; NeurIPS"). - **Cal-QL** - Calibrates conservative offline value estimates against a reference policy to improve initialization for subsequent online learning; addresses the offline-to-online transition. [Nakamoto et al. 2023]( "Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning; NeurIPS"). - **Implicit Q-learning** - Fits an expectile state-value estimate, backs it into a Q-function, and extracts a policy through advantage-weighted cloning. Avoids querying unseen actions during value learning. [Kostrikov et al. 2021]( "Offline Reinforcement Learning with Implicit Q-Learning; arXiv"). - **Decision Transformer** - Predicts actions autoregressively from desired return, past states, and actions. Recasts offline policy learning as conditional sequence modeling; requesting a return does not guarantee achieving it. [Chen et al. 2021]( "Decision Transformer: Reinforcement Learning via Sequence Modeling; NeurIPS"). - **Diffuser** - Plans by iteratively denoising trajectories, combining learned generation with conditioning on goals or rewards. Moves part of trajectory optimization into the learned model. [Janner et al. 2022]( "Planning with Diffusion for Flexible Behavior Synthesis; ICML"). ## Off-policy evaluation - **Sequential doubly robust evaluation** - Combines importance weighting with value-model corrections to estimate a target policy's return from logged behavior. Jiang and Li reduce variance in studied tasks; requires policy probabilities and action support. [Jiang and Li 2016]( "Doubly Robust Off-policy Value Evaluation for Reinforcement Learning; ICML"). - **Fitted Q Evaluation** - Repeatedly regresses Bellman targets for a fixed policy on logged transitions, then estimates its value. Le et al.'s error bounds require coverage and function-class conditions; fitting low regression error alone does not certify a new policy. [Le et al. 2019]( "Batch Policy Learning under Constraints; ICML"). ## Demonstrations and preference feedback - **DAgger** - Aggregates expert labels on observations encountered by the learner's policy. Addresses distribution shift between expert demonstrations and the states visited after the learner makes mistakes. [Ross et al. 2011]( "A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning; AISTATS"). - **GAIL** - Learns a policy through adversarial comparison with expert behavior, without a supplied task reward. The original method still collects policy rollouts; it is not purely offline behavioral cloning. [Ho and Ermon 2016]( "Generative Adversarial Imitation Learning; NeurIPS"). - **Human preference rewards** - Learns task objectives from human comparisons of trajectory segments and trains policies using that feedback. Replaces hand-written task rewards with learned preferences, not with direct access to human intent. [Christiano et al. 2017]( "Deep reinforcement learning from human preferences; NeurIPS"). - **Learning with demonstrations and long horizons** - Combines a transformed Bellman operator, temporal consistency, and human demonstrations for Atari. Separates reward-scale handling, long-horizon learning, and exploration contributions. [Pohlen et al. 2018]( "Observe and Look Further: Achieving Consistent Performance on Atari; arXiv"). ### Robotic imitation - **ACT** - A conditional VAE predicts action chunks and averages overlapping predictions. Without this averaging, increasing chunks from 1 to 100 steps raises mean success from 1% to 44% across two simulated bimanual tasks, then tapers. Chunking also helps tested baselines. [Zhao et al. 2023]( "Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware; RSS"); [code](). - **Diffusion Policy** - Denoises action sequences and executes a prefix before replanning. The better of its CNN and Transformer variants matches or beats tested baselines on eight simulated tasks. On BlockPush, the CNN completes both blocks in 11% of trials versus BET's 71%. [Chi et al. 2023]( "Diffusion Policy: Visuomotor Policy Learning via Action Diffusion; RSS"); [code](). - **OpenVLA** - Fine-tunes a 7B vision-language model on 970k robot episodes to emit discretized action tokens. Adapted to new tasks, it beats Diffusion Policy on diverse language-conditioned tasks, while Diffusion Policy remains competitive on narrower ones. [Kim et al. 2025]( "OpenVLA: An Open-Source Vision-Language-Action Model; CoRL"); [code](). - **OpenVLA-OFT** - On LIBERO with OpenVLA's inputs, parallel decoding of 8-step chunks raises success from 76.5% to 90.2%. Continuous L1 regression reaches 95.3%, versus 95.4% for 50-step diffusion. Demonstrations use a consistent strategy per task. [Kim et al. 2025]( "Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success; RSS"); [code](). - **Embodied reasoning and control** - In robot-policy ablations, movement and trajectory guidance help more than high-level reasoning alone. Scaling noisy embodied CoT pretraining can reduce success; ERVLA combines grounding, reasoning dropout and an action policy. [Sun et al. 2026]( "Revisiting Embodied Chain-of-Thought for Generalizable Robot Manipulation; NeurIPS"). ## Multiple agents and constrained control - **MADDPG** - Extends actor-critic learning to account for other agents' actions during training. Studies cooperative and competitive interaction, where other learning agents make the environment nonstationary. [Lowe et al. 2017]( "Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments; NeurIPS"). - **Constrained Policy Optimization** - Optimizes reward subject to expected-cost constraints through a policy-search update. Theoretical near-constraint satisfaction depends on assumptions and does not certify physical safety of a learned controller. [Achiam et al. 2017]( "Constrained Policy Optimization; ICML"). - **Evolution strategies for control** - Optimizes policy parameters through black-box evaluations and perturbations. Provides a control-learning baseline that does not require value functions or policy-gradient credit assignment through actions. [Salimans et al. 2017]( "Evolution Strategies as a Scalable Alternative to Reinforcement Learning; arXiv"). - **QMIX** - Combines per-agent values with a monotonic mixing network for centralized training and decentralized action choice. The monotonicity restriction makes local greedy choices consistent with the joint value model. [Rashid et al. 2018]( "QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning; ICML"). - **Multi-agent RL review** - Organizes challenges including nonstationarity, partial observability, communication, and centralized versus decentralized training. Surveys approaches and their application settings. [Nguyen et al. 2020]( "Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications; IEEE Trans. Cybern."). ## Evaluation and datasets - **Reproducibility in deep RL** - Shows how nondeterminism, hyperparameters, and reporting choices complicate algorithm comparisons. Calls for standardized protocols and uncertainty reporting rather than isolated best-run scores. [Henderson et al. 2018]( "Deep Reinforcement Learning that Matters; AAAI"). - **Arcade Learning Environment protocols** - Reviews Atari evaluation practices and introduces sticky actions as environmental stochasticity. Makes protocol differences explicit when comparing game-playing agents. [Machado et al. 2018]( "Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents; Journal of Artificial Intelligence Research"). - **Off-policy evaluation benchmark** - Stress-tests methods that estimate a policy's performance from data collected by other policies. Studies how experimental conditions affect estimator reliability and provides the COBS benchmark. [Voloshin et al. 2019]( "Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning; arXiv"). - **D4RL** - Benchmarks offline learning with human, controller-generated, multitask, and mixed-policy datasets. Tests limitations hidden by datasets collected only from partially trained RL agents. [Fu et al. 2020]( "D4RL: Datasets for Deep Data-Driven Reinforcement Learning; arXiv"). - **Statistical uncertainty in RL benchmarks** - Shows that few-run point estimates can change conclusions about algorithm rankings. Recommends interval estimates, performance profiles, and robust aggregates such as the interquartile mean. [Agarwal et al. 2021]( "Deep Reinforcement Learning at the Edge of the Statistical Precipice; NeurIPS"). ## Software - **Stable-Baselines3** - Provides Python implementations with consistent interfaces, documentation, and comparisons to reference codebases. A practical baseline implementation collection; the cited paper describes its tested release. [Raffin et al. 2021]( "Stable-Baselines3: Reliable Reinforcement Learning Implementations; JMLR"). - **CleanRL** - Provides self-contained algorithm implementations in single files, including training details. Supports inspection and modification of an individual algorithm without tracing a large shared framework. [Huang et al. 2022]( "CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms; JMLR"). ## Related topics - [Feature extraction](https://mlpapers.org/feature-extraction/) - [Model predictive control](https://mlpapers.org/model-predictive-control/) - [Neural Networks](https://mlpapers.org/neural-nets/) - [Optimization](https://mlpapers.org/optimization/) - [Modeling](https://mlpapers.org/models/)