# Scientific Machine Learning ## Problems and mathematical structure - **Solution learning** - Approximates a PDE solution from equations, boundary/initial conditions and observations. The original PINN formulation fits a coordinate-to-state function; solving a new problem can require another optimization. [Raissi et al. 2017]( "Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations; arXiv"). - **Operator learning** - Learns maps between function spaces, such as coefficients to solution fields. Shared parameters across discretizations distinguish the operator formulation from a fixed-grid predictor; approximation and numerical discretization remain separate errors. [Kovachki et al. 2023]( "Neural Operator: Learning Maps Between Function Spaces; JMLR"). - **Hybrid models** - Combines known differential equations with trainable missing terms. The supplied physics restricts what must be learned; fitting observed trajectories does not uniquely identify an unknown mechanism. [Rackauckas et al. 2020]( "Universal Differential Equations for Scientific Machine Learning; arXiv"). ## Neural solvers and equation constraints - **Physics-informed neural networks** - Uses automatic differentiation to penalize differential-equation residuals alongside data and boundary losses. Residual penalties enforce equations approximately at sampled points; they are not exact conservation constraints. [Raissi et al. 2017]( "Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations; arXiv"). - **Deep BSDE solvers** - Recasts suitable parabolic PDEs as backward stochastic differential equations. Neural networks approximate solution gradients along simulated paths; terminal-condition mismatch supplies the training loss. [E et al. 2017]( "Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations; Commun. Math. Stat."). - **Deep Ritz** - Minimizes a variational energy over neural trial functions. Unlike strong-form residual fitting, it uses the problem's variational formulation; the chosen functional and admissible boundary treatment determine the solution sought. [E and Yu 2018]( "The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems; Communications in Mathematics and Statistics"). - **Deep Galerkin Method** - Trains a neural solution on randomly sampled space-time points using PDE, initial and boundary residuals. Avoids a tensor-product mesh; reported high-dimensional examples do not imply dimension-independent accuracy for arbitrary PDEs. [Sirignano and Spiliopoulos 2018]( "DGM: A deep learning algorithm for solving partial differential equations; J. Comput. Phys."). - **hp-VPINNs** - Combines neural trial functions with local polynomial test functions and domain decomposition. Weak residuals and h/p refinement connect physics-informed training to variational numerical methods. [Kharazmi et al. 2021]( "hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition; CMAME"). - **Residual-adaptive refinement** - Adds collocation points where PDE residuals are large. DeepXDE uses this to focus PINN training; a small residual still needs a stability argument to imply small solution error. [Lu et al. 2021]( "DeepXDE: A deep learning library for solving differential equations; SIAM Rev."). ## Training failures and error analysis - **PINN failure modes** - Convection, reaction and diffusion examples expose optimization failures despite sufficient network expressivity. Curriculum regularization and sequential time training improve the tested cases; adding a PDE penalty alone is insufficient. [Krishnapriyan et al. 2021]( "Characterizing possible failure modes in physics-informed neural networks; NeurIPS"). - **Unbalanced loss gradients** - Wang et al. trace training failures to stiffness and unequal backpropagated gradients across loss terms. Gradient-statistics-based balancing improves their examples; the equation and boundary losses need compatible optimization scales. [Wang et al. 2021]( "Understanding and mitigating gradient pathologies in physics-informed neural networks; SIAM Journal on Scientific Computing"). - **PINN neural tangent kernels** - Infinite-width analysis reveals unequal convergence rates for residual and boundary losses. Eigenvalue-based reweighting addresses this imbalance in experiments; the limiting-kernel theory is not a finite-network convergence guarantee. [Wang et al. 2022]( "When and why PINNs fail to train: A neural tangent kernel perspective; Journal of Computational Physics"). - **Time-ordered PINN training** - Weights later-time residuals according to earlier-time errors. This respects the order of an initial-value problem and improves chaotic-system benchmarks; "causal" here refers to temporal evolution, not causal discovery. [Wang et al. 2022]( "Respecting causality is all you need for training physics-informed neural networks; arXiv"). - **Residual-to-solution error bounds** - De Ryck et al. bound Navier-Stokes PINN errors using training error, network size and quadrature. Regularity and PDE-specific stability enter the result; approximation capacity alone does not guarantee successful optimization. [De Ryck et al. 2023]( "Error estimates for physics informed neural networks approximating the Navier-Stokes equations; IMA Journal of Numerical Analysis"). - **PINN loss curvature** - Links differential-operator conditioning to loss curvature. Adam+L-BFGS beats either alone; Nyström-preconditioned Newton-CG improves stalled solutions in the tested PDEs. This compares PINN optimizers, not neural and classical solvers. [Rathore et al. 2024]( "Challenges in Training PINNs: A Loss Landscape Perspective; ICML"). ## Learning operators and numerical updates - **Reduced-order operator inference** - Fits linear or low-order polynomial dynamics by least squares on projected trajectories, without full solver operators. A reduced state basis limits the model; estimating dynamics adds information beyond compressing snapshots. [Peherstorfer and Willcox 2016]( "Data-driven operator inference for nonintrusive projection-based model reduction; CMAME"). - **Learned discretizations** - Learns spatial-derivative approximations inside a numerical time integrator. Bar-Sinai et al. obtain accurate coarse-grid dynamics on tested 1D equations; the learned component augments a solver rather than replacing its entire structure. [Bar-Sinai et al. 2019]( "Learning data driven discretizations for partial differential equations; PNAS"). - **Solver-in-the-loop correction** - Backpropagates through solver rollouts to train state corrections. Earlier corrections change later inputs, unlike fixed-trajectory training. Gains depend on dynamics: the forced-Burgers experiment shows only a small advantage. [Um et al. 2020]( "Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers; NeurIPS"). - **Particle graph simulators** - Represents particles as graph nodes and learns interactions by message passing. Training with perturbed states reduces accumulated rollout error in the reported simulations; long-horizon performance differs from one-step fit. [Sanchez-Gonzalez et al. 2020]( "Learning to Simulate Complex Physics with Graph Networks; ICML"). - **DeepONet** - A branch network encodes input-function samples; a trunk network encodes output locations. Their combination predicts an output function. The operator approximation theorem does not bound optimization or generalization error. [Lu et al. 2021]( "DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators; Nature Machine Intelligence"). - **Fourier Neural Operator** - Learns integral kernels through Fourier-mode transformations. Demonstrates resolution transfer on Burgers, Darcy and Navier-Stokes tasks; shared weights across grids do not guarantee accuracy on unseen physical regimes. [Li et al. 2021]( "Fourier Neural Operator for Parametric Partial Differential Equations; ICLR"). - **Physics-Informed Neural Operator** - Combines operator training on coarse data with PDE constraints evaluated at finer resolution. Connects FNO-style amortized prediction with equation-based refinement; separate pretraining cost from per-instance inference. [Li et al. 2024]( "Physics-Informed Neural Operator for Learning Partial Differential Equations; ACM / IMS Journal of Data Science"). - **MeshGraphNets** - Passes messages over simulation meshes to predict physical updates, with adaptive meshing in studied systems. Evaluated on cloth, solids and fluids; rollout accuracy and remeshing behavior matter beyond one-step prediction. [Pfaff et al. 2021]( "Learning Mesh-Based Simulation with Graph Networks; ICLR"). - **PDE pretraining: Poseidon** - Pretrains a multiscale operator transformer on fluid equations, then adapts to 15 tasks. Uses time conditioning and semigroup-based data pairing; transfer to new PDEs includes downstream training, not universal zero-shot solving. [Herde et al. 2024]( "Poseidon: Efficient Foundation Models for PDEs; NeurIPS"). - **Sparse molecular Hessians** - Derives a local potential's Hessian sparsity from its interaction graph, then uses coloring and Hessian-vector products for exact recovery. Linear time and storage require fixed cutoff, fixed message-passing depth and bounded local density. [Thiede et al. 2026]( "Colour me shocked: Exact Molecular Hessians from local MLIPs in O(N) time using sparse differentiation!; NeurIPS"). - **Meshfree exterior calculus** - Builds conservative operators on point clouds and learns a local flux law. Transfer bounds separate discretization and flux error, but require controlled stability constants and new edge-feature coverage; one solution need not identify the flux. [Shaffer et al. 2026]( "A meshfree exterior calculus for generalizable and data-efficient learning of physics from point clouds; NeurIPS"). ## Dynamics, conservation and discovery - **Equation discovery** - See [Symbolic Regression](https://mlpapers.org/symbolic-regression/#sparse-dynamical-system-identification) for SINDy, PDE-FIND and sparse-discovery follow-ups. - **Neural ODEs** - Parameterizes a continuous-time vector field and integrates it with an ODE solver. The solver, tolerances and gradient calculation become part of the learned system; a neural vector field need not preserve physical invariants. [Chen et al. 2018]( "Neural Ordinary Differential Equations; NeurIPS"). - **Latent ODEs** - Uses continuous latent dynamics and an ODE-RNN inference network for irregular observations. Can model observation times with a Poisson process; irregular sampling support does not itself identify the underlying physical state. [Rubanova et al. 2019]( "Latent ODEs for Irregularly-Sampled Time Series; arXiv"). - **Hamiltonian neural networks** - Learns a scalar Hamiltonian and derives dynamics through canonical derivatives. Conservation follows for the learned continuous system; numerical integration can still introduce energy drift. [Greydanus et al. 2019]( "Hamiltonian Neural Networks; NeurIPS"). - **Lagrangian neural networks** - Learns a Lagrangian and obtains dynamics from Euler-Lagrange equations. Uses positions and velocities without requiring canonical momenta; the learned dynamics and numerical solver still need validation. [Cranmer et al. 2020]( "Lagrangian Neural Networks; ICLR DDE Workshop"). - **Reservoir dynamics models** - Fixed recurrent states and delayed polynomial features provide alternatives to fitting a neural vector field. See [reservoir computing](https://mlpapers.org/neural-nets/#reservoir-computing) for the methods and chaotic-system evidence. ## Data assimilation and digital twins - **Adjoint variational assimilation** - Fits a dynamically constrained trajectory to observations across time. A forward model run and backward adjoint compute gradients with respect to the initial state, avoiding one perturbation run per control variable. [Le Dimet and Talagrand 1986]( "Variational algorithms for analysis and assimilation of meteorological observations: theoretical aspects; Tellus A"). - **Ensemble Kalman filter** - Propagates an ensemble through the dynamics and uses its sample covariances to update states from observations. The analysis approximates nonlinear Bayesian filtering; finite ensembles also introduce sampling error. [Evensen 1994]( "Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics; J. Geophys. Res. Oceans"), [Evensen 2003]( "The Ensemble Kalman Filter: theoretical formulation and practical implementation; Ocean Dyn."). - **Filtering connections** - See [Kalman filtering](https://mlpapers.org/models/#classical-forecasting-and-latent-states) for linear state estimation and [Sequential Monte Carlo](https://mlpapers.org/bayesian-inference/#sequential-monte-carlo) for particle filtering and smoothing. Ensemble covariance updates and weighted-particle approximations retain different information about uncertainty. - **Ensemble Kalman inversion** - Repurposes ensemble updates for derivative-free inverse-problem optimization. Estimates remain in the span of the initial ensemble, making its choice a restriction on recoverable parameters or fields. [Iglesias et al. 2013]( "Ensemble Kalman Methods for Inverse Problems; Inverse Problems"). - **Probabilistic digital twins** - Uses a dynamic decision network to update uncertain asset states and guide control. The UAV study calibrates on experimental data; in-flight damage tracking and planning use synthetic observations. [Kapteyn et al. 2021]( "A Probabilistic Graphical Model Foundation for Enabling Predictive Digital Twins at Scale; Nat. Comput. Sci."). ## Inverse problems and uncertainty - **Simulation-based inference** - Uses simulations when likelihood evaluation is unavailable, including learned likelihoods, likelihood ratios and posteriors. Inference concerns parameters under a simulator; posterior confidence does not validate the simulator itself. [Cranmer et al. 2020]( "The frontier of simulation-based inference; PNAS"). - **Bayesian PINNs** - Combines equation constraints and noisy observations in Bayesian inference. HMC gives better uncertainty estimates than tested variational/dropout alternatives in the paper; calibration remains specific to likelihood, prior and model adequacy. [Yang et al. 2021]( "B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data; J. Comput. Phys."). ## Sequence-to-function genomics - **Basenji** - Uses dilated convolutions to predict quantitative chromatin and transcription profiles from DNA. Enlarging the receptive field improves held-out prediction; variant scores align with measured expression associations across 19 tissues. [Kelley et al. 2018]( "Sequential regulatory activity prediction across chromosomes with convolutional neural networks; Genome Res."); [code](). - **Enformer** - Uses attention to integrate regulatory DNA up to 100,000 bases away. Across held-out genes, mean correlation with measured transcription rises from 0.81 with Basenji2 to 0.85; enhancer prioritization also improves against CRISPR perturbation data. [Avsec et al. 2021]( "Effective gene expression prediction from sequence by integrating long-range interactions; Nat. Methods"); [code](). - **Borzoi** - Extends Enformer to predict RNA-seq coverage from 524,000-base DNA windows. Coverage supports scoring effects on expression, splicing and RNA processing; adding chromatin-accessibility training improves expression-variant and enhancer-linking predictions. [Linder et al. 2025]( "Predicting RNA-seq coverage from DNA sequence as a unifying model of gene regulation; Nat. Genet."); [code](). - **AlphaGenome** - Jointly predicts expression, splicing and chromatin features from one million DNA bases, with outputs up to single-base resolution. Matches or exceeds the strongest tested external models on 25 of 26 variant-effect benchmarks. [Avsec et al. 2026]( "Advancing regulatory variant effect prediction with AlphaGenome; Nature"); [code](). - **Variant-effect underestimation** - On selected GTEx whole-blood genes, AlphaGenome often gives candidate causal variants weak effects relative to neighboring variants. Underestimation persists in a 1-Mb input check. [Drusinsky and Pollard 2026]( "Causal variant underestimation is a major overlooked driver of sequence-to-function model underperformance; bioRxiv"). ## Comparative evidence - **PDEBench** - Compares forward and inverse PDE tasks with FNO, U-Net and PINN baselines. Metrics include boundary, conservation and frequency-dependent errors; a single aggregate field error can conceal physically relevant failures. [Takamoto et al. 2022]( "PDEBENCH: An Extensive Benchmark for Scientific Machine Learning; NeurIPS"). - **Personal gene-expression prediction** - Across 421 people, Enformer and three other models predict differences between genes but have near-zero mean correlation for differences between individuals. Some genes show reversed predicted effects. [Huang et al. 2023]( "Personal transcriptome variation is poorly explained by current genomic deep learning models; Nat. Genet."). - **PINNs versus finite elements** - FEM was generally faster at equal or better accuracy in the tested PDEs; 1D Schrödinger results were inconclusive. Faster PINN point evaluation appeared only for 3D Poisson, after training. [Grossmann et al. 2024]( "Can Physics-Informed Neural Networks beat the Finite Element Method?; IMA Journal of Applied Mathematics"). - **Mass-balance audit** - In two steady-state 1D groundwater cases, PINNs selected from 10,800 configurations each fit hydraulic heads well but retain larger mass-balance errors than finite volumes. Accurate fields did not ensure local or global conservation. [Mamud et al. 2024]( "Quantifying local and global mass balance errors in physics-informed neural networks; Scientific Reports"). - **The Well** - Provides 16 diverse simulation datasets with common interfaces. Baselines expose differences across physical systems; evaluate rollouts and dataset-specific dynamics rather than treating performance on one fluid equation as general simulation ability. [Ohana et al. 2024]( "The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning; NeurIPS"). - **Perturbation-response prediction** - Tested deep models did not outperform simple baselines on gene perturbations. A cell-line follow-up finds gains with metrics focused on perturbation effects. Additive baselines remain strong for near-additive double perturbations. [Ahlmann-Eltze et al. 2025]( "Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines; Nat. Methods"), [Miller et al. 2026]( "Deep learning perturbation models can outperform baselines on calibrated metrics; Nat. Biotechnol."). - **Neural-solver breakeven** - Counts solves needed to repay data-generation and task-training cost against an error-matched classical solver. Some tested periodic PDE/model pairs need hundreds of thousands of calls; others never break even. Foundation-model pretraining is excluded. [Zhang et al. 2026]( "Breakeven complexity: A new perspective on neural partial differential equation solvers; NeurIPS"). ## Software - **DeepXDE** - Implements PINNs, operator networks and adaptive sampling for differential equations. Supports forward and inverse problems; equation residuals, boundary conditions and sampling rules remain user-specified. [docs](). ## Related topics - [Geometric deep learning](https://mlpapers.org/geometric-deep-learning/) - [Symbolic regression](https://mlpapers.org/symbolic-regression/) - [Model predictive control](https://mlpapers.org/model-predictive-control/) - [Bayesian inference](https://mlpapers.org/bayesian-inference/) - [Neural nets](https://mlpapers.org/neural-nets/) - [Evaluation](https://mlpapers.org/evaluation/)