# Model Evaluation and Comparison ## Evaluation targets - **Fitted model versus learning procedure** - Performance of one trained model differs from average performance over new training sets. Bates et al. establish this distinction for cross-validation in ordinary least squares. [Bates et al. 2023]( "Cross-validation: what does it estimate and how well does it do it?; JASA"). - **Selection versus assessment** - Optimizing a noisy validation criterion can overfit it; evaluating the selected winner on that same criterion is optimistic. Cawley and Talbot find selection effects comparable to algorithm differences. [Cawley and Talbot 2010]( "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation; JMLR"). ## Splitting and resampling - **.632+ bootstrap** - Combines training and out-of-bag error with a weight adjusted for overfitting. Efron and Tibshirani study classification error estimation; this is an error estimator, not a confidence interval. [Efron and Tibshirani 1997]( "Improvements on Cross-Validation: The 632+ Bootstrap Method; JASA"). - **Rolling-origin evaluation** - Repeatedly train on earlier observations and test later horizons. Match forecast horizons to deployment; random folds can expose information unavailable at prediction time. [Tashman 2000]( "Out-of-sample tests of forecasting accuracy: an analysis and review; IJF"), [Hyndman and Athanasopoulos 2021]( "Forecasting: Principles and Practice; book"). - **Group-aware validation** - GroupKFold and leave-one-group-out prevent the same subject, device or group from appearing on both sides of a split. Group choice defines whether evaluation tests new observations or generalization to new groups. [docs](). - **Clonal pseudoreplication** - Shared clones make null perturbations look reproducible in random cell halves. In one T-cell donor, STATE's effect correlation is 0.26 versus a 0.08 baseline, then below baseline across transductions. Clone effects are smaller in tested cell lines. [Huang 2026]( "Shared clones make random cell halves look reproducible in pooled screens; arXiv"). - **Cross-validation** - Holdout, k-fold and leave-one-out trade training-set size, computation and dependence among error estimates. Arlot and Celisse distinguish risk estimation from selecting a model; the best procedure depends on the objective. [Arlot and Celisse 2010]( "A survey of cross-validation procedures for model selection; Statist. Surv."). - **Stratification** - Preserves approximate class proportions across folds. Cross-validation documentation warns that this engineering safeguard can shrink observed fold variability; it does not remove sampling uncertainty. [docs](). - **Leakage-safe preprocessing** - Fit imputation, scaling, selection and learned representations within training splits. Pipelines help enforce this; preprocessing or selecting a pipeline using test outcomes still leaks information. [docs](). - **Nested cross-validation for selection** - Tune preprocessing, features and hyperparameters inside each outer training fold; assess the resulting procedure on its outer test fold. Outer results must not drive further selection. [Cawley and Talbot 2010]( "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation; JMLR"). - **Data thinning** - Under convolution-closed models, splits each observation into independent parts for clustering/PCA validation without holding out rows. Gaussian thinning needs the correct variance/covariance; misspecification can induce dependence. [Neufeld et al. 2024]( "Data Thinning for Convolution-Closed Distributions; JMLR"). ## Uncertainty in reported performance - **Dependent folds** - Bengio and Grandvalet prove no universally unbiased estimator of k-fold CV variance exists. Ignoring error correlations can underestimate uncertainty; extra folds are not independent replications. [Bengio and Grandvalet 2004]( "No Unbiased Estimator of the Variance of K-Fold Cross-Validation; JMLR"). - **Small-sample evaluation** - Varoquaux shows wide prediction-accuracy uncertainty in small neuroimaging samples. Standard errors across folds can understate it; resampling cannot substitute for collecting independent observations. [Varoquaux 2018]( "Cross-validation failure: small sample sizes lead to large error bars; NeuroImage"). - **Training and search variability** - Bouthillier et al. find data sampling, initialization and hyperparameter choices materially affect comparisons on five tasks. Repeating one fitted model's test evaluation misses these sources. [Bouthillier et al. 2021]( "Accounting for Variance in Machine Learning Benchmarks; MLSys"). - **Stratified bootstrap intervals** - Resample runs within benchmark tasks to quantify uncertainty in aggregate scores. Specify whether tasks themselves are fixed or sampled; the resampling unit determines the uncertainty represented. [Agarwal et al. 2021]( "Deep Reinforcement Learning at the Edge of the Statistical Precipice; NeurIPS"). - **Confidence sequences** - Builds intervals valid simultaneously over time, allowing data-dependent stopping. Martingale concentration bounds require stated tail or boundedness assumptions; repeated monitoring costs wider intervals. [Howard et al. 2021]( "Time-uniform, nonparametric, nonasymptotic confidence sequences; Ann. Statist."); [code](). - **Nested CV for uncertainty** - Bates et al. estimate prediction-error variance using nested resampling; reported interval coverage improves where naive CV intervals fail. This objective differs from nesting for hyperparameter selection. [Bates et al. 2023]( "Cross-validation: what does it estimate and how well does it do it?; JASA"). - **Post-collection selective inference** - Conditions on both adaptive query choices and a later chosen contrast. Gives selective p-values under Gaussian noise with known variance; the implemented search requires tractable selection events, illustrated with TPE and GP-UCB. [Nishino et al. 2026]( "Post-ADC Inference: Valid Inference After Active Data Collection; NeurIPS"). ## Statistical inference using predictions - **Prediction-powered inference** - Uses labeled data to correct prediction-based population estimates and intervals. The basic setup needs i.i.d. samples and an independent predictor; the paper's basic intervals are asymptotic, and weak predictions can widen them. [Angelopoulos et al. 2023]( "Prediction-powered inference; Science"); [code](). - **PPI++** - Tunes the weight of prediction-based corrections, with zero weight recovering labeled-only inference. Optimizes asymptotic variance under regularity assumptions; this is not a finite-sample improvement guarantee. [Angelopoulos et al. 2026]( "PPI++: Efficient Prediction-Powered Inference; The Annals of Applied Statistics"). - **Finite-sample prediction assistance** - In mean estimation, learning the PPI++ correction weight adds variance. Weakly correlated predictions can worsen error versus labeled-only estimation, even with arbitrarily many unlabeled samples; the threshold depends on label count. [Mani et al. 2026]( "No Free Lunch: Non-Asymptotic Analysis of Prediction-Powered Inference; ICML"). - **Calibrated prediction-powered inference** - Calibrates model scores on labeled data before pooling predictions for mean estimation. Linear calibration matches PPI++ at first order; isotonic calibration optimizes within monotone score transformations under regularity assumptions. [van der Laan and van der Laan 2026]( "Calibeating Prediction-Powered Inference; NeurIPS"). ## Metrics and scoring rules - **Thresholded classification** - Precision, recall and F-scores evaluate decisions at a chosen threshold; macro and pooled averaging weight classes differently. Report the threshold-selection procedure. [docs](). - **ROC versus precision-recall** - Davis and Goadrich relate both curve spaces. ROC-AUC optimization need not optimize PR-AUC; precision depends on class prevalence, so comparisons need the class mix. [Davis and Goadrich 2006]( "The relationship between Precision-Recall and ROC curves; ICML"). - **Point-prediction losses** - Squared error targets the conditional mean; absolute error targets a conditional median. Choosing a loss specifies the prediction being evaluated, not only its units. [docs](). - **Proper scoring rules** - Brier and logarithmic scores assess predictive probabilities; CRPS assesses predictive distributions. Propriety rewards truthful distributions in expectation. [Gneiting and Raftery 2007]( "Strictly Proper Scoring Rules, Prediction, and Estimation; JASA"). - **Interval scores** - Combine interval width with penalties for observations outside the interval. Coverage alone rewards arbitrarily wide intervals and misses useful sharpness. [Gneiting and Raftery 2007]( "Strictly Proper Scoring Rules, Prediction, and Estimation; JASA"). - **Calibration-error estimation** - Binning, sample size and estimator bias affect reported calibration. Kumar et al. develop more accurate estimators and scaling-binning calibration; smooth reliability plots alone are insufficient evidence. [Kumar et al. 2019]( "Verified Uncertainty Calibration; NeurIPS"). - **Reject option and selective prediction** - Chow derives a Bayes-optimal error-reject trade-off from true class posteriors. Geifman-El-Yaniv bounds selective risk under IID calibration. Coverage is the acceptance rate; risk is error among accepted predictions. [Chow 1957]( "An optimum character recognition system using decision functions; IRE Transactions on Electronic Computers"), [Chow 1970]( "On optimum recognition error and reject tradeoff; IEEE Transactions on Information Theory"), [Geifman and El-Yaniv 2017]( "Selective Classification for Deep Neural Networks; NeurIPS"). ## Predictive calibration - **Multiclass probability calibration** - Transforms classifier scores into multiclass probability estimates. [Zadrozny and Elkan 2002]( "Transforming classifier scores into accurate multiclass probability estimates; KDD"). - **Predicting good probabilities** - Compares distortions in classifier probabilities and correction by Platt scaling or isotonic regression; calibration method and calibration-set size affect results. [Niculescu-Mizil and Caruana 2005]( "Predicting good probabilities with supervised learning; ICML"). - **Nearly-isotonic regression** - Relaxes monotonicity through a convex penalty and computes a path of fits; intermediate solutions help examine whether strict monotonicity is appropriate. [Tibshirani et al. 2011]( "Nearly-Isotonic Regression; Technometrics"). - **Ensembles of calibration models** - Studies ensembles of near-isotonic regression for binary probability calibration and compares them with linear-trend calibration; relaxes strict isotonicity. [Pakdaman Naeini and Cooper 2017]( "Binary classifier calibration using an ensemble of piecewise linear regression models; Knowl Inf Syst"). - **Beta calibration** - Introduces a parametric calibration family for binary classifiers that includes the identity map and accommodates skewed score distributions. [Kull et al. 2017]( "Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers; AISTATS"). - **Temperature scaling** - Evaluates modern neural-network calibration and a single-parameter post-processing baseline; effectiveness on the evaluated datasets does not guarantee calibration after distribution shift. [Guo et al. 2017]( "On Calibration of Modern Neural Networks; ICML"). - **CRUDE** - Calibrates regression uncertainty using empirical errors; assumes a common residual-distribution shape after shifting by predicted means and scaling by predicted standard deviations. [Zelikman et al. 2020]( "CRUDE: Calibrating Regression Uncertainty Distributions Empirically; ICML UDL Workshop"). - **Label-free calibration transfer** - Projects predictions toward compatibility with a calibrated reference using unlabeled paired outputs. Proper-loss improvement requires a calibrated reference and failure of mutual calibration; reference errors can offset the gain. [Kong et al. 2026]( "Calibration without Ground Truth; NeurIPS"). ## Prediction-set coverage and risk control - **Conformal prediction under covariate shift** - Weights calibration scores by target-to-training covariate density ratios while holding the conditional response law fixed. Exact known-weight guarantees do not automatically carry over to inaccurately estimated ratios. [Tibshirani et al. 2019]( "Conformal Prediction Under Covariate Shift; NeurIPS"). - **Split conformal prediction** - Calibrates a fitted predictor's scores on held-out data to form prediction sets. Exchangeability supports finite-sample marginal coverage; report set size and subgroup coverage; the guarantee is not conditional on every input. [Angelopoulos and Bates 2023]( "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification; Foundations and Trends in Machine Learning"). - **Conformal risk control** - Calibrates a threshold to bound expected loss over calibration and test draws. Requires exchangeable, bounded, non-increasing, right-continuous losses and an achievable target; the guarantee is not per input. [Angelopoulos et al. 2024]( "Conformal Risk Control; ICLR"); [code](). - **Anytime-valid conformal risk control** - Bounds calibration-conditional risk with high probability simultaneously as IID calibration data accumulate. Requires a fixed nested set family and bounded, monotone, right-continuous loss; does not cover arbitrary online model updates. [Hultberg et al. 2026]( "Anytime-Valid Conformal Risk Control; NeurIPS"). ## Statistical comparisons - **Comparisons across datasets** - Wilcoxon signed-rank compares two methods; Friedman with post-hoc tests compares several. Datasets are the comparison units; correlated CV folds are not interchangeable with independent datasets. [Demšar 2006]( "Statistical Comparisons of Classifiers over Multiple Data Sets; JMLR"). - **Permutation tests** - Shuffle labels to test class structure, or shuffle features within classes to test exploited dependencies. The permitted shuffles define the tested null hypothesis. [Ojala and Garriga 2010]( "Permutation Tests for Studying Classifier Performance; JMLR"). - **Bayesian comparisons** - Estimate probabilities of superiority, practical equivalence and inferiority under a statistical model. Specify the equivalence region and dependence model; these differ from null-hypothesis p-values. [Benavoli et al. 2017]( "Time for a Change: a Tutorial for Comparing Multiple Classifiers Through Bayesian Analysis; JMLR"). ## Group fairness criteria - **Equalized odds and opportunity** - Equalized odds matches error rates across groups conditional on the outcome; equal opportunity requires equal true-positive rates. Hardt et al. construct post-processing rules; the criteria use observed labels and group membership. [Hardt et al. 2016]( "Equality of Opportunity in Supervised Learning; NeurIPS"). - **Conflicting fairness criteria** - Chouldechova shows that predictive-value parity and equal false-positive/negative rates conflict at unequal base rates for nondegenerate imperfect classifiers. A fairness target must specify which errors and populations matter. [Chouldechova 2017]( "Fair prediction with disparate impact: A study of bias in recidivism prediction instruments; Big Data"). ## Benchmarks, reuse and distribution shift - **Adaptive holdout reuse** - Repeated feedback can overfit a test set. Dwork et al. use controlled, randomized access to limit this effect; guarantees belong to the access procedure, not unrestricted repeated leaderboard inspection. [Dwork et al. 2015]( "Generalization in Adaptive Data Analysis and Holdout Reuse; NeurIPS"). - **Uncertainty under shift** - Ovadia et al. find in-distribution post-hoc calibration insufficient under tested shifts; some model-averaging methods fare better. Evaluate predictive accuracy and calibration separately. [Ovadia et al. 2019]( "Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift; NeurIPS"). - **Robust benchmark summaries** - Interquartile mean and performance profiles expose information hidden by one mean or median. Agarwal et al. show uncertainty-aware analyses change conclusions on studied few-run RL benchmarks. [Agarwal et al. 2021]( "Deep Reinforcement Learning at the Edge of the Statistical Precipice; NeurIPS"). - **Natural distribution shifts** - WILDS benchmarks hospital, location, time and other shifts. Standard training and tested shift-handling methods retain substantial out-of-distribution gaps; the split defines the generalization claim. [Koh et al. 2021]( "WILDS: A Benchmark of in-the-Wild Distribution Shifts; ICML"). - **Learning curves** - Compare held-out performance against training-set size. Viering and Loog document diverse shapes, including nonmonotonic curves; extrapolating a fitted power law is an additional modeling assumption. [Viering and Loog 2023]( "The Shape of Learning Curves: a Review; IEEE Transactions on Pattern Analysis and Machine Intelligence"). - **LLM-as-judge evaluation** - MT-Bench studies agreement with human preferences alongside position, verbosity and self-preference biases. A model judge approximates judgments in a tested setting; it is not a factual oracle or a universal replacement for human assessment. [Zheng et al. 2023]( "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena; NeurIPS"). - **AgentAbstain** - 263 paired tasks test both acting and abstaining. Best paired accuracy is 59.5% across 17 agents. One rollout per task; failed runs are excluded, and GPT-5.4 judges verbal responses. The score does not assess full task success. [Liu et al. 2026]( "AgentAbstain: Do LLM Agents Know When Not to Act?; NeurIPS"). - **Leaderboard manipulation** - Offline attacks on LMArena data change Bradley-Terry rankings through strategic votes, added model copies and prompt selection. Unchanged models can move when competitors change; these are simulated attacks, not demonstrated live compromise. [Fleisig et al. 2026]( "Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation; NeurIPS"). - **Select-LLM** - Chooses queries to reduce uncertainty about the best model. Across 23 datasets, needs fewer reference labels than adapted active-selection baselines. Uses precomputed model outputs and reliable references; savings exclude model-inference cost. [Durmazkeser et al. 2026]( "Large Language Model Selection with Limited Annotations; NeurIPS"). - **Verbalized evaluation awareness** - Across eight open models and four benchmarks, inserting awareness sentences mid-trace or removing them changes answers little. Removal excludes unsuccessful edits; this tests verbalized awareness, not hidden evaluation detection or gaming. [Knecht et al. 2026]( "Evaluation Awareness in Language Models Has Limited Effect on Behaviour; NeurIPS"). ## Software - **scikit-learn** - Splitters, scoring functions and pipelines support evaluation workflows. The chosen split, search and scorer still determine what is estimated. [docs](), [docs](), [docs](). - **rliable** - Implementation of interval estimates, interquartile means, probability of improvement and performance profiles for benchmark score arrays. Run/task structure and score normalization remain explicit inputs to the analysis. [code](). - **baycomp** - Bayesian comparison tools for classifiers on one or multiple datasets; returns probabilities for superiority, practical equivalence and inferiority. The caller supplies the equivalence threshold. [docs](). ## Related topics - [AutoML](https://mlpapers.org/automl/) - [Feature selection](https://mlpapers.org/feature-selection/) - [Anomaly detection](https://mlpapers.org/anomaly-detection/) - [Bayesian inference](https://mlpapers.org/bayesian-inference/) - [Interpretability](https://mlpapers.org/interpretability/) - [Transfer learning](https://mlpapers.org/transfer-learning/)