# Feature Selection and Variable Selection ## Selection objectives - **Predictive selection** - Chooses a small or inexpensive input subset for a specified learner and prediction task. Its risk-cost trade-off need not identify a unique set of generating variables. [Guyon and Elisseeff 2003]( "An Introduction to Variable and Feature Selection; JMLR"). - **Minimal-optimal versus all-relevant** - A small subset sufficient for prediction and the full set of relevant variables are different targets; redundant variables can matter to the latter. [Nilsson et al. 2007]( "Consistent Feature Selection for Pattern Recognition in Polynomial Time; JMLR"). - **Conditional discovery** - Model-X knockoffs test whether the target is independent of a variable given all remaining inputs. This target differs from keeping every useful substitute. [Candes et al. 2018]( "Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection; JRSS B"). - **Unsupervised selection** - Usefulness is defined by a chosen surrogate, such as reconstruction, neighborhoods or cluster structure. Preserving it does not establish utility for an unspecified target. [Li et al. 2017]( "Feature Selection: A Data Perspective; ACM CSUR"). - **Acquisition** - Static selection uses one subset; an adaptive policy chooses the next measurement from values already observed. This differs from selecting an explanation after seeing all inputs. [Covert et al. 2023]( "Learning to Maximize Mutual Information for Dynamic Feature Selection; ICML"). ### Interactions and interchangeable variables - **XOR example** - For independent fair bits, Y = X1 XOR X2 is independent of either input alone, yet determined by their pair. Even nonlinear univariate dependence scores can miss joint information. [Brown et al. 2012]( "Conditional Likelihood Maximisation: A Unifying Framework for Information Theoretic Feature Selection; Journal of Machine Learning Research"). - **Duplicate example** - If X1 = X2 = Y, either input predicts perfectly, and each is conditionally uninformative given the other. Changing between these subsets need not change predictions. [Statnikov et al. 2013]( "Algorithms for Discovery of Multiple Markov Boundaries; JMLR"). ## Comparative evidence - **Tabular feature-selection benchmark** - On 12 datasets with MLP/FT-Transformer predictors, Deep Lasso improves prediction with corrupted or second-order added features. Products of original variables can be useful; rejecting every added feature is not the objective. [Cherepanova et al. 2023]( "A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning; NeurIPS"). - **SelectArena** - Across 71 datasets and five learners, tree importance and F-tests offer strong prediction/runtime trade-offs. Selection uses fixed subset sizes and a one-hour cap, with random filling/subsampling for interrupted searches; validity measures rejection of added noise. [Schäfer et al. 2026]( "SelectArena: A Modern Benchmark for Feature Selection on Real-world Tabular Data; arXiv"). ## Filter criteria and screening ### Marginal screening - **Variance threshold** - Removes features with low training-set variance without using labels; scale affects the result. [docs](). - **Univariate tests** - Chi-squared for non-negative counts/frequencies; ANOVA F for class differences; linear-regression F for marginal linear association. Individual scores can miss interactions. [docs](), [docs](), [docs](). - **Sure independence screening (SIS/ISIS)** - Screens marginal correlations before subset fitting; ISIS repeats screening. Signal/design assumptions give high-probability retention of all signals, while extra variables may remain. [Fan and Lv 2008]( "Sure Independence Screening for Ultra-High Dimensional Feature Space; JRSS B"). ### Information, redundancy, and interactions - **Relief** - Updates feature weights from nearby same-class and different-class examples. Relief-family review covers interaction sensitivity and extensions. [Kira and Rendell 1992]( "The Feature Selection Problem: Traditional Methods and a New Algorithm; AAAI"), [Urbanowicz et al. 2018]( "Relief-Based Feature Selection: Introduction and Review; Journal of Biomedical Informatics"). - **MI estimation** - Discrete and continuous features require appropriate treatment; mislabeling their type can give incorrect estimates. Nonlinear dependence estimation does not make marginal scoring interaction-aware. [docs](). - **Joint mutual information (JMI)** - Scores candidates jointly with already selected variables. Brown et al. derive JMI and related criteria as approximations to conditional-likelihood selection. [Yang and Moody 1999]( "Data Visualization and Feature Selection: New Algorithms for Nongaussian Data; NeurIPS"), [Brown et al. 2012]( "Conditional Likelihood Maximisation: A Unifying Framework for Information Theoretic Feature Selection; Journal of Machine Learning Research"). - **FCBF (fast correlation-based filter)** - Uses symmetrical uncertainty to screen relevance and remove redundant features. [Yu and Liu 2003]( "Feature Selection for High-Dimensional Data: A Fast Correlation-Based Filter Solution; ICML"). - **Consistency-based selection** - Penalizes identical selected-feature patterns with conflicting labels. Dash and Liu compare search strategies using this criterion. [Dash and Liu 2003]( "Consistency-based search in feature selection; Artificial Intelligence"). - **Mutual information** - A dependence quantity; Kraskov et al. provide an estimator. Selection criteria include mRMR/JMI, while forward search constructs a subset. These are separate choices. [Kraskov et al. 2004]( "Estimating Mutual Information; Physical Review E"), [Brown et al. 2012]( "Conditional Likelihood Maximisation: A Unifying Framework for Information Theoretic Feature Selection; Journal of Machine Learning Research"). - **mRMR** - Balances feature-target mutual information against redundancy among selected features. [Hanchuan Peng et al. 2005]( "Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy; TPAMI"). - **INTERACT** - Removes features using changes in class inconsistency, targeting interactions that marginal ranking can miss. [Zhao and Liu 2007]( "Searching for Interacting Features; IJCAI"). ## Conditional independence and Markov boundaries - **Koller-Sahami** - Eliminates features using approximate Markov blankets and information-theoretic redundancy criteria. [Koller and Sahami 1996]( "Toward Optimal Feature Selection; ICML"). - **Grow-Shrink** - Grows a candidate blanket and then removes conditionally irrelevant variables. [Margaritis and Thrun 1999]( "Bayesian Network Induction via Local Neighborhoods; NeurIPS"). - **IAMB** - Incremental association-based blanket discovery, with forward inclusion and backward pruning. [Tsamardinos et al. 2003]( "Algorithms for Large Scale Markov Blanket Discovery; FLAIRS"). - **Max-Min blanket discovery** - Identifies parents/children and extends the search to a target's Markov blanket. A predictive blanket alone does not identify causal direction. [Tsamardinos et al. 2003]( "Time and sample efficient discovery of Markov blankets and direct causal relations; KDD"). - **Markov blankets** - Seek variables that render the target conditionally independent of the remaining variables. Discovery depends on conditional-independence tests and distributional assumptions. [Fu and Desmarais 2010]( "Markov Blanket based Feature Selection: A Review of Past Decade; WCE"). - **Markov boundary** - An inclusion-minimal blanket. Multiple boundaries can exist, yielding alternative sufficient subsets; disagreement need not identify a uniquely wrong selection. [Statnikov et al. 2013]( "Algorithms for Discovery of Multiple Markov Boundaries; JMLR"). - **FOCI** - Greedily adds variables using a conditional-dependence statistic. Its sufficient-set recovery result needs detectable improvement at every insufficient subset; purely joint signals can defeat this forward-search condition. [Azadkia and Chatterjee 2021]( "A simple measure of conditional dependence; The Annals of Statistics"). ## Subset search and cardinality optimization - **Floating search** - Alternates inclusion and exclusion to revisit earlier choices. Adaptive floating search varies the search depth. [Pudil et al. 1994]( "Floating search methods in feature selection; Pattern Recognit. Lett."), [Somol et al. 1999]( "Adaptive floating search methods in feature selection; Pattern Recognition Letters"). - **Sequential selection** - Greedily adds or removes features according to cross-validation scores. SequentialFeatureSelector supports forward and backward search. [docs](). - **Population-based subset search** - Genetic search and particle swarms explore subsets scored by a chosen criterion; that criterion defines the selection objective. [Miao and Niu 2016]( "A Survey on Feature Selection; Procedia Computer Science"), [Kennedy and Eberhart 1995]( "Particle swarm optimization; ICNN"), [Tu et al. 2007]( "Feature Selection using PSO-SVM; IAENG IJCS"). - **Wrappers** - Evaluate candidate subsets using a particular learner's predictive performance. Search and validation are part of the method. [Kohavi and John 1997]( "Wrappers for feature subset selection; Artificial Intelligence"). - **Recursive feature elimination (RFE)** - Repeatedly fits a model and removes low-ranked inputs. RFE and RFECV implement elimination and cross-validated subset-size selection. [Guyon et al. 2002]( "Gene Selection for Cancer Classification using Support Vector Machines; Mach. Learn."); [docs](), [docs](). - **Best-subset optimization** - Mixed-integer formulations constrain feature count directly. Solver bounds can certify global optimality or quantify a remaining gap when stopped early. [Bertsimas et al. 2016]( "Best Subset Selection via a Modern Optimization Lens; The Annals of Statistics"). - **Fast L0 selection** - Coordinate descent and local combinatorial search optimize L0-penalized objectives. Their local optimality conditions differ from a global MIO certificate. [Hazimeh and Mazumder 2020]( "Fast Best Subset Selection: Coordinate Descent and Local Combinatorial Optimization Algorithms; Operations Research"). ## Embedded sparsity and model-based selection ### Penalized and Bayesian models - **Spike-and-slab** - Mixture priors represent exclusion and inclusion of regression coefficients. George and McCulloch develop computational approaches. [Mitchell and Beauchamp 1988]( "Bayesian Variable Selection in Linear Regression; JASA"), [George and McCulloch 1997]( "Approaches for Bayesian Variable Selection; Statistica Sinica"). - **Lasso** - An L1 penalty yields sparse regression coefficients, coupling variable selection with fitting. [Tibshirani 1996]( "Regression Shrinkage and Selection Via the Lasso; JRSS B"). - **Adaptive lasso** - Weights each coefficient's L1 penalty using an initial estimate. Zou establishes selection consistency and oracle asymptotics under design, initialization and tuning conditions; this is not a finite-sample guarantee of correct support. [Zou 2006]( "The Adaptive Lasso and Its Oracle Properties; JASA"). - **Lasso support recovery** - Sparse prediction does not establish correct variable recovery. Selection consistency requires additional assumptions, notably restrictions on predictor dependence. [Zhao and Yu 2006]( "On Model Selection Consistency of Lasso; JMLR"). - **Elastic net** - Combines L1 and L2 penalties, encouraging groups of correlated predictors to enter together. [Zou and Hastie 2005]( "Regularization and variable selection via the elastic net; JRSS B"). - **Group lasso** - Selects predefined groups of coefficients, such as all encoded levels of a categorical variable. This differs from elastic net's grouping of correlated predictors. [Yuan and Lin 2005]( "Model Selection and Estimation in Regression with Grouped Variables; JRSS B"). - **HSIC Lasso** - Uses feature-wise kernels to select variables with strong nonlinear dependence on the response and limited redundancy with one another; a sparse alternative to linear Lasso screening. [Yamada et al. 2014]( "High-Dimensional Feature Selection by Feature-Wise Kernelized Lasso; Neural Comput."). ### Trees, importance, and subset rules - **Decision trees** - Split-based importance sums impurity reductions attributed to each input; thresholding can select a subset. These scores can favor high-cardinality inputs. [docs](), [docs](). - **Gradient boosting** - Sequential function fitting can supply feature-importance rankings. Importance thresholding produces a subset. [Friedman 2001]( "Greedy function approximation: A gradient boosting machine.; Ann. Statist."), [Mason et al. 1999]( "Boosting Algorithms as Gradient Descent; NeurIPS"); [docs](). - **Random forests** - Ensembles provide importance scores used for ranking and selection. Boulesteix et al. review practical pitfalls. A ranking still needs a subset-selection rule. [Breiman 2001]( "Random Forests; Mach. Learn."), [Boulesteix et al. 2012]( "Overview of random forest methodology and practical guidance with emphasis on computational biology and bioinformatics; WIREs Data Min & Knowl"). - **Boruta** - Targets all-relevant selection by comparing random-forest importances with randomized shadow features. [Kursa et al. 2010]( "Boruta – A System for Feature Selection; Fundamenta Informaticae"), [Kursa and Rudnicki 2010]( "Feature Selection with the Boruta Package; J. Stat. Soft."); [code](). - **VSURF** - Forest importance and stepwise selection produce subsets for interpretation and prediction. [Genuer et al. 2010]( "Variable selection using Random Forests; Pattern Recognition Letters"), [Genuer et al. 2015]( "VSURF: An R Package for Variable Selection Using Random Forests; The R Journal"). ### Differentiable selectors - **Concrete autoencoders** - Learn which original inputs to retain through a relaxed discrete selector and reconstruction decoder. [Balın et al. 2019]( "Concrete Autoencoders: Differentiable Feature Selection and Reconstruction; ICML"); [code](). - **Stochastic gates (STG)** - Learn input gates jointly with a predictor using a continuous relaxation of an L0 feature-count penalty. [Yamada et al. 2020]( "Feature Selection using Stochastic Gates; ICML"); [code](). - **LassoNet** - Ties nonlinear feature use to sparse linear skip connections through a hierarchical constraint; produces a regularization path of global subsets. [Lemhadri et al. 2021]( "LassoNet: A Neural Network with Feature Sparsity; JMLR"); [code](). - **Deep Lasso** - Applies a group penalty to loss gradients with respect to input features, extending linear sparsity to input sensitivity. Gradient norms also rank features. [Cherepanova et al. 2023]( "A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning; NeurIPS"). ## Unsupervised and graph-based selection - **FSSEM** - Searches feature subsets alongside expectation-maximization clustering; examines likelihood and scatter criteria and their dimensionality biases. [Dy and Brodley 2004]( "Feature Selection for Unsupervised Learning; JMLR"). - **Laplacian Score** - Ranks features by preservation of local graph structure; applicable without class labels. [He et al. 2005]( "Laplacian Score for Feature Selection; NeurIPS"). - **Principal feature analysis** - Uses PCA-derived structure to choose representative original variables. Its output is an input subset rather than principal-component coordinates. [Lu et al. 2007]( "Feature selection using principal feature analysis; ACM Multimedia"). - **Spectral feature selection (SPEC)** - Evaluates features through a similarity graph's spectrum, covering supervised and unsupervised criteria. [Zhao and Liu 2007]( "Spectral feature selection for supervised and unsupervised learning; ICML"). - **Multi-cluster feature selection (MCFS)** - Combines a spectral embedding with sparse regression to select features preserving cluster structure. [Cai et al. 2010]( "Unsupervised feature selection for multi-cluster data; KDD"). ## Resampling, error control and post-selection inference - **Ensemble selection** - Aggregates feature preferences across perturbed datasets or selectors. Aggregation alone supplies no generic FDR guarantee. [Saeys et al. 2008]( "Robust Feature Selection Using Ensemble Feature Selection Techniques; ECML PKDD"), [Shen et al. 2019]( "A Novel Weighted Combination Method for Feature Selection using Fuzzy Sets; FUZZ-IEEE"). - **Stability selection** - Thresholds selection frequencies across subsamples. Its original bound controls the expected number of false selections under exchangeability and other assumptions. [Meinshausen and Buehlmann 2010]( "Stability Selection; JRSS B"). - **Complementary-pairs stability selection** - Pairs disjoint half-samples and bounds selection of variables with low selection probabilities, without the original exchangeability requirement. [Shah and Samworth 2012]( "Variable selection with error control: Another look at Stability Selection; JRSS B"). - **Artificial controls, different targets** - Boruta compares forest importances with shuffled shadows for all-relevant selection; model-X uses exchangeable controls for conditional discovery with FDR control under its assumptions. [Kursa and Rudnicki 2010]( "Feature Selection with the Boruta Package; J. Stat. Soft."), [Candes et al. 2018]( "Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection; JRSS B"). - **Knockoff+** - Selects against synthetic control variables with finite-sample false discovery rate control in the paper's Gaussian linear-model setting. [Barber and Candès 2015]( "Controlling the false discovery rate via knockoffs; Ann. Statist."). - **Post-selection inference** - Conditions on lasso selection to form intervals for selected-model coefficients. Exact results use a fixed design and Gaussian errors with known variance. [Lee et al. 2016]( "Exact post-selection inference, with application to the lasso; Ann. Statist."). - **Model-X knockoffs** - Permits arbitrary response models; FDR control requires exchangeability and knockoffs independent of the response conditional on the inputs. [Candes et al. 2018]( "Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection; JRSS B"); [docs](). - **Deep knockoffs** - Learns approximate model-X knockoff generators by encouraging pairwise exchangeability. Generator diagnostics matter: approximate sampling alone does not establish exact false-discovery control. [Romano et al. 2019]( "Deep Knockoffs; JASA"). - **Knockoff robustness** - Flexibility in the response model relies on modeling the input distribution. Errors in estimated feature conditionals can inflate FDR; robustness bounds quantify this dependence. [Barber et al. 2020]( "Robust inference with knockoffs; The Annals of Statistics"). ## Instancewise selection - **Instancewise selection** - L2X learns a feature subset for each example to explain a fixed predictor; this differs from one global subset used for all observations. [Chen et al. 2018]( "Learning to Explain: An Information-Theoretic Perspective on Model Interpretation; ICML"). - **INVASE** - A selector, predictor and baseline network learn instance-specific feature subsets through actor-critic training; the selected variables and subset size can change between observations. [Yoon et al. 2019]( "INVASE: Instance-wise Variable Selection using Neural Networks; ICLR"). ## Cost-sensitive and adaptive acquisition - **Dynamic feature selection** - Learns a policy approximating greedy conditional-mutual-information acquisition from observed inputs. Recovery of the greedy policy assumes optimal training; it is not a globally optimal acquisition guarantee. [Covert et al. 2023]( "Learning to Maximize Mutual Information for Dynamic Feature Selection; ICML"). ## Evaluation and comparative evidence ### Validation and selection stability - **Selection leakage** - Choosing features before resampling can bias estimated prediction error. Ambroise and McLachlan demonstrate the problem in microarray classification; pipeline guidance keeps selection inside training folds. [Ambroise and McLachlan 2002]( "Selection bias in gene extraction on the basis of microarray gene-expression data; PNAS"); [docs](). - **Stability measurement** - Compares scores, ranks, or subsets under data perturbations. Nogueira, Sechidis, Brown provide a chance-corrected measure, intervals, and tests. [Kalousis et al. 2006]( "Stability of feature selection algorithms: a study on high-dimensional spaces; Knowl Inf Syst"), [Nogueira et al. 2018]( "On the Stability of Feature Selection Algorithms; Journal of Machine Learning Research"). - **Search overfitting** - A noisy validation criterion can overfit selector, subset-size and predictor choices. Nested resampling or an untouched test set must evaluate the complete search, including learned preprocessing. [Cawley and Talbot 2010]( "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation; JMLR"). - **MUVR** - Combines variable selection with repeated double cross-validation in multivariate modeling. [Shi et al. 2018]( "Variable selection and validation in multivariate modelling; Bioinformatics"); [code](). ### Correlated features and importance bias - **Forest importance bias** - Strobl et al. show preference for variables with more categories or different measurement scales in studied forest importance measures. Importance is not an unbiased relevance test. [Strobl et al. 2007]( "Bias in random forest variable importance measures: Illustrations, sources and a solution; BMC Bioinformatics"). - **Conditional permutation importance** - Permutes within a conditioning scheme to account for predictor dependence. Gregorutti et al. analyze correlation effects and recursive elimination. [Strobl et al. 2008]( "Conditional variable importance for random forests; BMC Bioinformatics"), [Gregorutti et al. 2016]( "Correlation and variable importance in random forests; Statistics and Computing"). - **Held-out permutation importance** - Measures a fitted model's score loss after permuting a feature. Correlated substitutes can mask importance; the result depends on the model and score. [docs](). ## Task-specific selection and method recommendation - **Selector recommendation** - Learns from dataset metafeatures and previous selector evaluations. [Wang et al. 2013]( "A Feature Subset Selection Algorithm Automatic Recommendation Method; Journal of Artificial Intelligence Research"), [Parmezan et al. 2017]( "Metalearning for choosing feature selection algorithms in data mining: Proposal of a new framework; Expert Systems with Applications"), [Shen et al. 2020]( "A Novel Meta Learning Framework for Feature Selection using Data Synthesis and Fuzzy Similarity; FUZZ-IEEE"). - **Uplift modeling** - Selects inputs for heterogeneous treatment-effect prediction rather than ordinary outcome prediction. Zhao et al. develop task-specific statistical and information-theoretic criteria. [Zhao et al. 2022]( "Feature Selection Methods for Uplift Modeling and Heterogeneous Treatment Effect; IFIP AICT"). ## Software - **Python (classical selection)** - scikit-learn provides filters, model thresholds, RFE/RFECV, and sequential selection; scikit-feature collects research implementations. [docs](), [code](). - **Python (learned selection and error control)** - Concrete autoencoders, STG, LassoNet, knockpy. [code](), [code](), [code](), [docs](). - **Supporting tools** - feature-selector removes missingness/constant/collinearity-based columns; these are data-quality rules. BoostARoota implements XGBoost-based shadow comparisons. [code](), [code](). - **R (search and filters)** - FSinR and praznik; copent supplies supporting copula/transfer-entropy estimation. [Aragón-Royón et al. 2020]( "FSinR: an exhaustive package for feature selection; arXiv"), [Ma 2020]( "copent: Estimating Copula Entropy and Transfer Entropy in R; arXiv"); [code](), [code](), [code](). - **R (model-based selection)** - Boruta, VSURF, glmnet, spikeSlabGAM, knockoff. [Scheipl 2011]( "spikeSlabGAM: Bayesian Variable Selection, Model Choice and Regularization for Generalized Additive Mixed Models in R; J. Stat. Soft."); [code](), [code](), [code](), [code](), [code](). - **L0Learn** - Approximate L0-regularized regression and classification through coordinate descent and local combinatorial search; C++ implementation with R/Python interfaces. [Hazimeh et al. 2023]( "L0Learn: A Scalable Package for Sparse Learning using L0 Regularization; JMLR"). - **Archived software** - fscaret was archived by CRAN in August 2025; retained for historical reference. [code](). ## Surveys and further reading - **An Introduction to Variable and Feature Selection** - Relevance, redundancy, ranking, subset search, and validation. [Guyon and Elisseeff 2003]( "An Introduction to Variable and Feature Selection; JMLR"). - **Feature-selection survey** - Reviews filter, wrapper and embedded approaches. [Miao and Niu 2016]( "A Survey on Feature Selection; Procedia Computer Science"). - **Feature Selection: A Data Perspective** - Conventional, structured, heterogeneous, and streaming data. [Li et al. 2017]( "Feature Selection: A Data Perspective; ACM CSUR"). - **Feature Selection and Feature Extraction in Pattern Analysis** - Selection of existing variables and construction of new representations. [Ghojogh et al. 2019]( "Feature Selection and Feature Extraction in Pattern Analysis: A Literature Review; arXiv"). - **Attribution rankings** - Man-Chan compare permutation, LIME and SHAP rankings for random forests; reported stability differences do not establish predictive gains from selecting those features. [Man and Chan 2020]( "The best way to select features?; The Journal of Financial Data Science"). ## Connections - **Representation learning** - Ordinary autoencoders construct latent features. Concrete autoencoders instead constrain the encoder to select original inputs. [Baldi 2012]( "Autoencoders, Unsupervised Learning, and Deep Architectures; Proceedings of ICML Workshop on Unsupervised and Transfer Learning"), [Kingma and Welling 2019]( "An Introduction to Variational Autoencoders; Found. Trends ML"), [Balın et al. 2019]( "Concrete Autoencoders: Differentiable Feature Selection and Reconstruction; ICML"). ## Related topics - [Feature extraction](https://mlpapers.org/feature-extraction/) - [Interpretability](https://mlpapers.org/interpretability/) - [Causal inference](https://mlpapers.org/causal-inference/) - [Ensemble learning](https://mlpapers.org/ensemble-learning/) - [AutoML](https://mlpapers.org/automl/)