--- title: "Generative Models" aliases: ["Generative AI", "Deep generative models"] tags: - generative - deep-learning - unsupervised --- # Generative Models ## Overview - **Flow matching and diffusion** - Derives diffusion and flow matching from stochastic and ordinary differential equations, connecting training objectives, guidance, and image/video architectures. A mathematical overview rather than a new model comparison. [Holderrieth and Erives 2025]( "An Introduction to Flow Matching and Diffusion Models; arXiv"). ## Latent-variable models - **Variational autoencoders** - Learns a generative model and an approximate posterior together by optimizing a variational lower bound. Reparameterization enables stochastic gradients through continuous latent samples. [Kingma and Welling 2014]( "Auto-Encoding Variational Bayes; ICLR"). - **Importance-weighted autoencoders** - Uses multiple posterior samples and importance weighting to tighten the VAE likelihood bound. Changes the training objective while retaining a generative network and recognition model. [Burda et al. 2016]( "Importance Weighted Autoencoders; ICLR"). - **VAEs for images, labels, and captions** - Combines an image encoder and generative decoder with label or caption models. Supports learning from images with or without associated annotations. [Pu et al. 2016]( "Variational Autoencoder for Deep Learning of Images, Labels and Captions; NeurIPS"). - **VAE foundations and extensions** - Reviews approximate inference, variational training and extensions of deep latent-variable models. [Doersch 2016]( "Tutorial on Variational Autoencoders; arXiv"), [Kingma and Welling 2019]( "An Introduction to Variational Autoencoders; Found. Trends ML"). - **VQ-VAE** - Learns discrete latent codes through vector quantization and fits a learned prior over them. Separates representation learning from modeling the distribution of codes for generation. [van den Oord et al. 2017]( "Neural Discrete Representation Learning; NeurIPS"). - **VQGAN and latent Transformers** - Learns a compact vocabulary of image constituents with a convolutional model, then models their composition with a Transformer. Supports spatial and class conditioning for image synthesis. [Esser et al. 2021]( "Taming Transformers for High-Resolution Image Synthesis; CVPR"). ## Autoregressive and masked generation - **NADE** - Decomposes a joint distribution into tractable neural conditional distributions. Provides normalized probabilities for high-dimensional binary vectors without an intractable partition function. [Larochelle and Murray 2011]( "The Neural Autoregressive Distribution Estimator; AISTATS"). - **Deep autoregressive networks** - Adds autoregressive connections to stochastic hidden layers, enabling exact ancestral sampling. Learns through an approximate variational/description-length objective. [Gregor et al. 2014]( "Deep AutoRegressive Networks; ICML"). - **MADE** - Masks network connections so each output depends only on earlier inputs in a chosen ordering. Produces autoregressive conditional probabilities with a feedforward network and supports multiple orderings. [Germain et al. 2015]( "MADE: Masked Autoencoder for Distribution Estimation; ICML"). - **PixelRNN** - Models discrete image pixels sequentially using two-dimensional recurrent layers. Assigns probabilities to raw pixel values conditioned on preceding pixels. [van den Oord et al. 2016]( "Pixel Recurrent Neural Networks; ICML"). - **Conditional PixelCNN** - Uses gated convolutional image-density models conditioned on labels or learned embeddings. Can act as a conditional generator or as an autoregressive decoder within an autoencoder. [van den Oord et al. 2016]( "Conditional Image Generation with PixelCNN Decoders; NeurIPS"). - **WaveNet** - Generates raw audio autoregressively, conditioning each sample on earlier waveform values. Additional conditioning supports text-to-speech and speaker-specific synthesis. [van den Oord et al. 2016]( "WaveNet: A Generative Model for Raw Audio; arXiv"). - **MaskGIT** - Trains a bidirectional Transformer to predict masked image tokens. Generation fills tokens in parallel and refines them iteratively, replacing fixed raster-order decoding. [Chang et al. 2022]( "MaskGIT: Masked Generative Image Transformer; CVPR"). - **Autoregressive generation with diffusion losses** - Models continuous token distributions with a diffusion loss inside autoregressive generation, avoiding a discrete vector-quantized tokenizer; includes masked autoregressive image generation. [Li et al. 2024]( "Autoregressive Image Generation without Vector Quantization; NeurIPS"). ## Adversarial generation - **GANs** - Trains a generator against a discriminator that distinguishes generated from observed samples. The idealized minimax analysis assumes unrestricted functions; finite-network training need not reach that solution. [Goodfellow et al. 2014]( "Generative Adversarial Networks; NeurIPS"). - **DCGAN** - Uses constrained convolutional generator and discriminator architectures for adversarial image generation. Studies learned visual representations and their use in downstream tasks. [Radford et al. 2016]( "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks; ICLR"). - **Wasserstein GAN** - Uses a constrained critic to estimate a Wasserstein objective between generated and observed distributions. The original implementation enforces the critic constraint through weight clipping. [Arjovsky et al. 2017]( "Wasserstein GAN; ICML"). - **WGAN-GP** - Replaces WGAN's weight clipping with a penalty on the critic's input-gradient norm. Addresses optimization problems caused by clipping; it does not establish convergence for every GAN architecture. [Gulrajani et al. 2017]( "Improved Training of Wasserstein GANs; NeurIPS"). - **Spectral normalization** - Normalizes discriminator weights to stabilize adversarial training. Offers a reusable normalization technique instead of changing the generator's latent representation. [Miyato et al. 2018]( "Spectral Normalization for Generative Adversarial Networks; ICLR"). - **StyleGAN** - Introduces a style-based generator with control over synthesis at different scales and separate stochastic variation. Studies latent interpolation and separation of visual attributes. [Karras et al. 2019]( "A Style-Based Generator Architecture for Generative Adversarial Networks; CVPR"). - **StyleGAN3** - Traces texture sticking to aliasing in the generator and redesigns signal processing to reduce dependence on pixel coordinates. Studies translation- and rotation-equivariant synthesis configurations. [Karras et al. 2021]( "Alias-Free Generative Adversarial Networks; NeurIPS"). - **TimeGAN** - Jointly learns an embedding space with supervised temporal and adversarial objectives to generate time series that preserve temporal relationships. [Yoon et al. 2019]( "Time-series Generative Adversarial Networks; NeurIPS"). - **Adaptive discriminator augmentation** - Adjusts augmentation during GAN training to counter discriminator overfitting with limited data. Applies to training from scratch and fine-tuning without changing the loss or network architecture. [Karras et al. 2020]( "Training Generative Adversarial Networks with Limited Data; NeurIPS"). ## Invertible density models - **NICE** - Maps observations invertibly to a factorized latent distribution with tractable inverse and Jacobian determinant. Trains by exact log-likelihood and samples through the inverse transformation. [Dinh et al. 2015]( "NICE: Non-linear Independent Components Estimation; ICLR Workshop"). - **Real NVP** - Uses invertible, non-volume-preserving transformations for tractable density estimation. Supports exact likelihood evaluation and latent inversion under the specified model. [Dinh et al. 2017]( "Density estimation using Real NVP; ICLR"). - **Masked Autoregressive Flow** - Stacks autoregressive transformations to increase density-model flexibility. Connects autoregressive density estimation with normalizing flows and generalizes Real NVP. [Papamakarios et al. 2017]( "Masked Autoregressive Flow for Density Estimation; NeurIPS"). - **Glow** - Introduces invertible 1×1 convolutions within a normalizing flow. Combines tractable likelihood and latent inversion with parallelizable image synthesis. [Kingma and Dhariwal 2018]( "Glow: Generative Flow with Invertible 1x1 Convolutions; NeurIPS"). - **Neural spline flows** - Uses monotonic rational-quadratic splines as invertible elementwise transforms. Adds flexibility to coupling and autoregressive flows while retaining analytic inversion. [Durkan et al. 2019]( "Neural Spline Flows; NeurIPS"). - **FFJORD** - Defines an invertible transformation through an ODE and estimates the Jacobian trace with Hutchinson's estimator. Removes determinant-based architectural restrictions; numerical integration still has a cost. [Grathwohl et al. 2019]( "FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models; ICLR"). ## Score learning and diffusion - **Diffusion probabilistic models** - Gradually destroys data structure through a forward diffusion process and learns a reverse process that restores it. Establishes the noising-and-reversal construction for deep generative modeling. [Sohl-Dickstein et al. 2015]( "Deep Unsupervised Learning using Nonequilibrium Thermodynamics; ICML"). - **Noise-conditional score networks** - Estimates score fields at multiple Gaussian-noise levels and samples through annealed Langevin dynamics. Noise perturbations make score learning practical around low-dimensional data manifolds. [Song and Ermon 2019]( "Generative Modeling by Estimating Gradients of the Data Distribution; NeurIPS"). - **DDPM** - Connects diffusion training to denoising score matching through a weighted variational objective. Learns iterative denoising for image generation without adversarial training. [Ho et al. 2020]( "Denoising Diffusion Probabilistic Models; NeurIPS"). - **Improved DDPM** - Learns reverse-process variances and modifies diffusion training to improve likelihood and sampling efficiency. Studies likelihood, sample quality, and distribution coverage separately. [Nichol and Dhariwal 2021]( "Improved Denoising Diffusion Probabilistic Models; ICML"). - **Score-based SDEs** - Unifies diffusion and score models through forward and reverse stochastic differential equations. Derives a probability-flow ODE and predictor-corrector sampling procedures. [Song et al. 2021]( "Score-Based Generative Modeling through Stochastic Differential Equations; ICLR"). - **Elucidated diffusion models** - Separates diffusion design choices in training, network preconditioning, and sampling. Studies how changes can improve both newly trained models and sampling from existing score networks. [Karras et al. 2022]( "Elucidating the Design Space of Diffusion-Based Generative Models; NeurIPS"). ## Conditioning and latent architectures - **Classifier guidance** - Uses gradients from a classifier to steer conditional diffusion sampling. Trades sample diversity against fidelity; the reported image-generation comparisons depend on architecture and guidance settings. [Dhariwal and Nichol 2021]( "Diffusion Models Beat GANs on Image Synthesis; NeurIPS"). - **Classifier-free guidance** - Combines conditional and unconditional score estimates from joint training. Steers generation without a separate classifier, with a quality-diversity tradeoff controlled at sampling time. [Ho and Salimans 2022]( "Classifier-Free Diffusion Guidance; arXiv"). - **Latent diffusion** - Performs diffusion in a pretrained autoencoder's latent space. Cross-attention supports text and other conditioning, while latent compression reduces the dimensionality handled by the denoiser. [Rombach et al. 2022]( "High-Resolution Image Synthesis with Latent Diffusion Models; CVPR"). - **Diffusion Transformers** - Replaces the U-Net in latent image diffusion with a Transformer operating on latent patches. Studies scaling through network size and token count rather than introducing a new diffusion objective. [Peebles and Xie 2023]( "Scalable Diffusion Models with Transformers; ICCV"). - **Rectified-flow Transformers** - Studies training-time sampling for rectified flow and a Transformer with separate image/text weights and bidirectional token interactions. Combines a transport objective with multimodal architecture design. [Esser et al. 2024]( "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis; ICML"). - **Diffusion Forcing** - Trains a causal sequence model with independent noise levels per token, combining next-token prediction with diffusion denoising so future tokens need not share the past's noise level. [Chen et al. 2024]( "Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion; NeurIPS"). ## Flow matching and transport - **Diffusion Schrödinger bridges** - Uses approximate iterative proportional fitting to learn a stochastic bridge between data and a reference distribution, connecting score-based generation to entropy-regularized transport. [De Bortoli et al. 2021]( "Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling; NeurIPS"). - **Rectified flow** - Fits an ODE to straight interpolations between samples from two distributions. Repeated rectification seeks straighter learned trajectories that can be integrated with fewer steps. [Liu et al. 2022]( "Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow; arXiv"). - **Flow matching** - Trains continuous normalizing flows by regressing vector fields along conditional probability paths, without simulating trajectories during training. Supports diffusion paths and other choices of transport path. [Lipman et al. 2023]( "Flow Matching for Generative Modeling; ICLR"). - **Stochastic interpolants** - Constructs a time-dependent density between base and target distributions and learns its velocity with a quadratic objective. Avoids backpropagating through an ODE solver during training. [Albergo and Vanden-Eijnden 2023]( "Building Normalizing Flows with Stochastic Interpolants; ICLR"). - **MeanFlow** - Learns average rather than instantaneous flow velocity using an identity linking the two. Targets one-step generation trained from scratch, without requiring a pretrained teacher or distillation. [Geng et al. 2025]( "Mean Flows for One-step Generative Modeling; NeurIPS"). - **Stop-gradient convergence** - Solves nonconservative training dynamics for MeanFlow and proves convergence to the true flow map under infinite-capacity, continuous-time optimization. Improved MeanFlow and slim distillation proofs assume perfect flow-matching pretraining. [Shen et al. 2026]( "How I learned to stop worrying and love StopGrads: Stationarity, Convergence, and a case study on Flow Map Learning; NeurIPS"). - **Generator matching** - Learns infinitesimal Markov generators from conditional probability paths using Bregman losses. Unifies flow, diffusion and jump processes; conditional training has the same parameter gradients as the intractable marginal objective. [Holderrieth et al. 2025]( "Generator Matching: Generative modeling with arbitrary Markov processes; ICLR"). - **Drifting** - Trains a one-pass generator with stopped-gradient attraction to data and repulsion between generated samples. Reports FID 1.54 on 50,000 class-conditional ImageNet-256 samples; the setup requires a pretrained feature encoder. [Deng et al. 2026]( "Generative Modeling via Drifting; NeurIPS"). - **Conservative drifting fields** - Sharp kernel normalization turns the drift into a kernel-density score difference. The potential freezes the current generated distribution, so it changes between updates; this does not establish descent on one fixed objective. [Franz et al. 2026]( "Drifting Fields are not Conservative; NeurIPS"). - **Strong stochastic flow maps** - Learns an additive-noise SDE solution conditioned on the initial state and a Brownian path approximation. Targets pathwise consistency, unlike matching only transition distributions; assumes regular coefficients and approximates the solution map. [McCallum et al. 2026]( "Strong Stochastic Flow Maps; NeurIPS"). - **Wasserstein-tilted flow maps** - Fine-tunes a pretrained flow map with transport-based reward regularization. Moves samples rather than reweighting them as KL tilting does; comparisons exclude prior flow-map distillation, and greater reward can reduce diversity. [Mammadov et al. 2026]( "WTF?! Simulation-Free Reinforcement Learning with Wasserstein-Tilted Flow Maps; NeurIPS"). ## Sampling and distillation - **DDIM** - Constructs non-Markovian diffusion processes sharing the DDPM training objective. Changes the sampling procedure to trade computation against sample quality without requiring a new training objective. [Song et al. 2021]( "Denoising Diffusion Implicit Models; ICLR"). - **DPM-Solver** - Analytically handles the linear part of the diffusion ODE and numerically approximates the remaining integral. Provides higher-order sampling without retraining the denoising model. [Lu et al. 2022]( "DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps; NeurIPS"). - **Progressive distillation** - Distills a deterministic diffusion sampler into one using half as many steps, then repeats. Transfers a teacher's sampling behavior into a sequence of progressively shorter samplers. [Salimans and Ho 2022]( "Progressive Distillation for Fast Sampling of Diffusion Models; ICLR"). - **Consistency models** - Learns a direct mapping from noise to data for one-step generation, with optional multistep refinement. Supports both distillation from diffusion models and standalone training. [Song et al. 2023]( "Consistency Models; ICML"). - **Latent consistency models** - Distills guided latent diffusion into a model predicting probability-flow ODE solutions in latent space. Reduces iterative sampling while retaining conditioning from the pretrained generator. [Luo et al. 2023]( "Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference; arXiv"). - **Efficient Adjoint Matching** - Redesigns diffusion drift and terminal cost to obtain closed-form adjoint targets. SD3.5-M reward tuning reaches comparable PickScore in up to four times fewer GPU-hours than Adjoint Matching; approximate sampling still uses a pretrained model. [Shin et al. 2026]( "Efficient Adjoint Matching for Fine-tuning Diffusion Models; NeurIPS"). - **Colored noise sampling** - Redistributes sampling noise across frequency bands while preserving total variance. On unguided ImageNet-256 SiT-XL/2 at 250 steps, FID falls from 8.26 to 6.27 versus white-noise SDE sampling; gains with guidance are smaller. [Davidson et al. 2026]( "Colored Noise Diffusion Sampling; NeurIPS"). - **Fréchet-distance training** - Uses queues or running moments to estimate a large-population distance while differentiating only the current batch. Post-trains image generators without a teacher; optimizing Inception FID and improving human preference are separate comparisons. [Yang et al. 2026]( "Representation Fréchet Loss for Visual Generation; NeurIPS"). - **Score-query lower bounds** - Worst-case diffusion sampling needs roughly sqrt(d) adaptive queries under polynomially accurate score oracles, even for smoothed bounded distributions. This oracle-model result does not rule out learned one-step generators or distillation. [Xun and Price 2026]( "Query Lower Bounds for Diffusion Sampling; NeurIPS"). - **Lanczos covariance sampling** - Uses Jacobian-vector products to sample full reverse covariance without dense matrices. At matched time, FID improves on CIFAR-10/CelebA but worsens on ImageNet-64 versus OCM; the ideal path-KL rate assumes exact moments. [Akhtar et al. 2026]( "The Value of Covariance Matching in Gaussian DDPMs and the Lanczos Sampler; NeurIPS"). ## Discrete diffusion and flows - **D3PM** - Extends diffusion to discrete states with structured transition matrices, including absorbing states and embedding-based transitions. Combines a variational bound with an auxiliary cross-entropy objective. [Austin et al. 2021]( "Structured Denoising Diffusion Models in Discrete State-Spaces; NeurIPS"). - **Discrete flow matching** - Defines probability paths and sampling rules for discrete variables using learned posterior predictions. Supports non-autoregressive generation without treating tokens as continuous image pixels. [Gat et al. 2024]( "Discrete Flow Matching; NeurIPS"). - **LLaDA** - Trains a language diffusion model with forward masking and a Transformer that predicts masked tokens during reverse generation. Uses a likelihood lower bound and supports supervised instruction fine-tuning. [Nie et al. 2025]( "Large Language Diffusion Models; NeurIPS"). - **Branching Flows** - Adds learned splitting and deletion rates to generator matching, allowing sequence length to change during generation. Supports discrete, continuous and manifold-valued elements, including unknown-length segments between fixed flanking regions. [Billera et al. 2026]( "Branching Flows: Discrete, Continuous, and Manifold Flow Matching with Splits and Deletions; NeurIPS"). ## Energy-based generation - **Boltzmann machines** - Studies stochastic networks with symmetric connections, energy-based relaxation, and learning of hidden representations. Distinct from sequence-processing RNNs such as LSTM. [Hinton and Sejnowski 1986]( "Learning and Relearning in Boltzmann Machines; book chapter"). - **Neural energy-based models** - Scales MCMC-based training of neural energy models to images and trajectories. Studies generation, composition, and reconstruction through a learned energy function rather than an explicit feedforward generator. [Du and Mordatch 2019]( "Implicit Generation and Generalization in Energy-Based Models; arXiv"). ## Evaluation and memorization - **Inception Score** - Evaluates generated images through classifier confidence and diversity of predicted classes. Does not compare generated samples directly against a held-out real-image distribution. [Salimans et al. 2016]( "Improved Techniques for Training GANs; NeurIPS"). - **Fréchet Inception Distance** - Compares Gaussian approximations to real and generated image features using their means and covariances. Measures agreement in the chosen feature space rather than every aspect of the image distribution. [Heusel et al. 2017]( "GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium; NeurIPS"). - **Finite-sample FID bias** - Chong and Forsyth show that FID bias depends on the evaluated model, so equal sample counts do not remove ranking distortions. They extrapolate toward infinite-sample scores; sample budget and estimator should accompany FID comparisons. [Chong and Forsyth 2020]( "Effectively Unbiased FID and Inception Score and where to find them; CVPR"). - **Kernel Inception Distance** - Measures squared MMD between Inception features using a polynomial kernel. Its U-statistic estimates this discrepancy without finite-sample bias; the chosen representation and kernel still determine which distributional differences are measured. [Bińkowski et al. 2018]( "Demystifying MMD GANs; ICLR"). - **Distribution precision and recall** - Separates generated-sample quality from coverage of the target distribution. Exposes distinct failure modes that a single distribution-distance score can combine. [Sajjadi et al. 2018]( "Assessing Generative Models via Precision and Recall; NeurIPS"). - **Improved precision and recall** - Builds nonparametric representations of real and generated feature manifolds to assess quality and coverage separately. Also studies how truncation changes the generated distribution. [Kynkäänniemi et al. 2019]( "Improved Precision and Recall Metric for Assessing Generative Models; NeurIPS"). - **Density and coverage** - Revisits failures of precision/recall estimators, including sensitivity to outliers and identical-distribution comparisons. Proposes density and coverage metrics for more informative diagnosis. [Naeem et al. 2020]( "Reliable Fidelity and Diversity Metrics for Generative Models; ICML"). - **Perceptual metric reassessment** - Finds weak agreement between common generation metrics and human realism judgments in the studied settings. Examines feature-extractor choice and shows that existing metrics can miss memorization. [Stein et al. 2023]( "Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models; NeurIPS"). - **Extracting training images** - Demonstrates extraction of memorized training examples from studied diffusion models using generation and filtering. Synthetic output is not, by itself, evidence that training records remain private. [Carlini et al. 2023]( "Extracting Training Data from Diffusion Models; arXiv"). ## Related topics - [Neural Networks](https://mlpapers.org/neural-nets/) - [Bayesian Inference](https://mlpapers.org/bayesian-inference/) - [Computer Vision](https://mlpapers.org/computer-vision/) - [NLP](https://mlpapers.org/nlp/)