# Interpretability and Explainable AI ## Explanatory targets - **Predictor and behavior** - Inspect the scoring rule, explain one output, or measure predictive performance. Removal-based explanations distinguish output contributions from loss-based importance; these targets need different comparisons. [Covert et al. 2021]( "Explaining by Removing: A Unified Framework for Model Explanation; JMLR"). - **Training data** - Influence methods estimate how training observations affect predictions or loss. Koh and Liang approximate parameter responses; deletion-and-retraining is a separate validation target. [Koh and Liang 2017]( "Understanding Black-box Predictions via Influence Functions; ICML"). - **Representation and computation** - Probes test decodable information; causal abstractions test proposed computations through aligned interventions. Accessible information alone does not establish its use. [Hewitt and Liang 2019]( "Designing and Interpreting Probes with Control Tasks; EMNLP"), [Geiger et al. 2021]( "Causal Abstractions of Neural Networks; NeurIPS"). - **Decision changes** - Recourse concerns feasible actions and their downstream effects. Intervening on model activations tests the program; intervening on people or physical systems requires assumptions about the world. [Karimi et al. 2020]( "Algorithmic Recourse: from Counterfactual Explanations to Interventions; arXiv"). ## Interpretable models - **Additive models and interactions** - GAMs sum feature functions on a link/score scale; GA²M adds pairwise terms. EBMs learn them by boosting. Additivity on that scale need not imply additive probability changes. [Lou et al. 2013]( "Accurate intelligible models with pairwise interactions; KDD"); [docs](). - **RETAIN** - Uses reverse-time, two-level attention and computes visit/variable contributions with attention held fixed. Interpretation relies on this architecture's contribution calculation. [Choi et al. 2016]( "RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism; NeurIPS"); [code](). - **Rule lists** - Ordered conditions followed by a default prediction. CORELS certifies optimal regularized training risk over its supplied rule candidates when optimization completes. [Angelino et al. 2018]( "Learning Certifiably Optimal Rule Lists for Categorical Data; JMLR"). - **Sparse scoring systems** - Learn small integer coefficients with sparsity and application-specific constraints. The score is the predictor itself. [Ustun and Rudin 2019]( "Learning Optimized Risk Scores; JMLR"). - **Neural additive models** - Jointly train one neural network per input and sum their outputs on the score/link scale. The original architecture exposes feature functions without explicit cross-feature interaction terms. [Agarwal et al. 2021]( "Neural Additive Models: Interpretable Machine Learning with Neural Nets; NeurIPS"). ## Feature effects and global importance - **Individual conditional expectation (ICE)** - Plots a separate prediction curve for each observation, exposing heterogeneity hidden by average partial dependence. [Goldstein et al. 2015]( "Peeking Inside the Black Box: Visualizing Statistical Learning with Plots of Individual Conditional Expectation; Journal of Computational and Graphical Statistics"). - **Partial dependence (PDP)** - Averages model predictions while varying selected inputs and retaining other observed values. Correlated inputs can produce unrealistic combinations. [docs](). - **Model class reliance** - Ranges over feature importance across a specified class of sufficiently accurate predictors. Different models can rely on different inputs; the class and performance tolerance define the range. [Fisher et al. 2019]( "All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously; JMLR"). - **Accumulated local effects (ALE)** - Accumulates locally averaged prediction differences, reducing extrapolation caused by dependent inputs. Feature effects describe the fitted model, not causal effects. [Apley and Zhu 2020]( "Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models; JRSS B"). - **Permutation importance** - Measures a fitted model's score change after shuffling one input. Results depend on the model, score, evaluation data, and correlated substitutes. [docs](). - **SAGE** - Assigns Shapley values to features using their contribution to expected predictive loss reduction. This measures predictive power rather than averaging local prediction attributions. [Covert et al. 2020]( "Understanding Global Feature Contributions With Additive Importance Measures; NeurIPS"). - **Removal-based explanations** - Covert et al. connect SHAP, LIME, permutation tests and other methods through three choices: how features are removed, which behavior is measured, and how removal effects are summarized. [Covert et al. 2021]( "Explaining by Removing: A Unified Framework for Model Explanation; JMLR"). - **Dependence and extrapolation** - Breaking feature relationships can make importance and effect plots reflect behavior in sparsely observed regions. Hooker and Mentch analyze this failure and alternatives. [Hooker et al. 2021]( "Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance; Statistics and Computing"). - **Conditional subgroup importance** - Forms interpretable subgroups with trees, then permutes within them to approximate conditional feature importance; also adapts PDPs. [Molnar et al. 2023]( "Model-agnostic Feature Importance and Effects with Dependent Features -- A Conditional Subgroup Approach; Data Min Knowl Disc"). ## Attribution and local explanations ### Feature attribution - **Layer-wise relevance propagation (LRP)** - Redistributes a prediction backward through layers using relevance-propagation rules. The propagation rule determines the decomposition. [Bach et al. 2015]( "On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation; PLoS ONE"). - **SHAP** - Decomposes an individual prediction into additive Shapley contributions. Aggregation summarizes contributions over chosen observations; it does not turn them into loss-based importance. [Lundberg and Lee 2017]( "A Unified Approach to Interpreting Model Predictions; NeurIPS"); [code](). - **TreeSHAP** - Computes tree-model attributions under specified assumptions about missing or dependent features. Background data and perturbation mode affect the explanation. [Lundberg et al. 2018]( "Consistent Individualized Feature Attribution for Tree Ensembles; arXiv"); [docs](). - **Shapley-value choices** - The value function and treatment of absent features define the attribution problem; different constructions can disagree. [Sundararajan and Najmi 2020]( "The Many Shapley Values for Model Explanation; ICML"). - **Duplicate-input example (constructed)** - Let X2=X1 be a fair bit and f(x)=x1. At (1,1), conditional Shapley gives (0.25,0.25); marginal replacement gives (0.5,0). Game definitions change whether an unused but informative input receives credit. [Sundararajan and Najmi 2020]( "The Many Shapley Values for Model Explanation; ICML"). - **Integrated Gradients** - Integrates input gradients along a straight path from a baseline to the observation. Attributions sum to the output difference; the baseline is part of the explanation. [Sundararajan et al. 2017]( "Axiomatic Attribution for Deep Networks; ICML"). - **DeepLIFT** - Propagates activation differences relative to reference activations to assign input contributions. [Shrikumar et al. 2017]( "Learning Important Features Through Propagating Activation Differences; ICML"); [code](). - **Attribution guarantees** - IG completeness constrains a sum; Shapley axioms characterize an allocation for a specified game. Neither property alone identifies a unique internal computation. [Sundararajan et al. 2017]( "Axiomatic Attribution for Deep Networks; ICML"), [Sundararajan and Najmi 2020]( "The Many Shapley Values for Model Explanation; ICML"). - **Grad-CAM** - Weights convolutional feature maps with target gradients to produce a coarse spatial attribution map. A heatmap's visual appeal does not establish faithfulness. [Selvaraju et al. 2019]( "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization; IJCV"). ### Local surrogates and rules - **LIME** - Fits an interpretable surrogate around one prediction using weighted perturbed samples. Sampling, neighborhood weights, and surrogate capacity shape the result. [Ribeiro et al. 2016]( ""Why Should I Trust You?": Explaining the Predictions of Any Classifier; KDD"); [code](). - **MeLIME** - Extends LIME with data-distribution-aware sampling and alternative local interpretable models. Its changes address the perturbation and surrogate-fitting stages. [Botari et al. 2020]( "MeLIME: Meaningful Local Explanation for Machine Learning Models; arXiv"). - **Anchors** - Searches for rules that preserve a prediction with high probability under a specified perturbation distribution. Precision depends on that distribution. [Ribeiro et al. 2018]( "Anchors: High-Precision Model-Agnostic Explanations; AAAI"); [code](). ### Formal sufficient explanations - **Abductive explanations** - Use a model/domain encoding and sound solver to find assignments preserving a prediction for all admissible completions. Subset-minimal means none can be removed; minimum-cardinality means the fewest assignments. [Ignatiev et al. 2019]( "Abduction-Based Explanations for Machine Learning Models; AAAI"). ## Counterfactual explanations and recourse - **Counterfactual explanations** - Search for nearby input changes that produce a desired model output. A changed prediction alone does not establish an actionable intervention. [Wachter et al. 2018]( "Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR; Harvard Journal of Law & Technology"). - **Actionable recourse** - Optimizes changes to actionable inputs for linear classifiers. Karimi et al. distinguish input edits from interventions with downstream causal effects. [Ustun et al. 2019]( "Actionable Recourse in Linear Classification; FAT*"), [Karimi et al. 2020]( "Algorithmic Recourse: from Counterfactual Explanations to Interventions; arXiv"). - **Diverse counterfactuals** - DiCE balances proximity and diversity to offer several ways to change a prediction. Feature constraints specify permitted changes. [Mothilal et al. 2020]( "Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations; FAccT"); [code](). ## Training-data attribution - **Influence functions** - Approximate how upweighting a training example changes a test prediction or loss using derivatives of the fitted model. Deep-network approximations require validation. [Koh and Liang 2017]( "Understanding Black-box Predictions via Influence Functions; ICML"). - **Influence-estimation failures** - Network depth, regularization, and test-point choice affect approximation accuracy in the investigated neural networks. Reported influence is not an exact retraining result. [Basu et al. 2021]( "Influence Functions in Deep Learning Are Fragile; ICLR"). - **TracIn** - Estimates training-example influence using gradient inner products across saved training checkpoints. This differs from a single-checkpoint Hessian approximation. [Pruthi et al. 2020]( "Estimating Training Data Influence by Tracing Gradient Descent; NeurIPS"). - **Datamodels** - Predict a target example's model output from which training examples were included. Ilyas et al. fit these surrogates using many subset-trained models, enabling dataset counterfactuals; the training algorithm and subset distribution define the target. [Ilyas et al. 2022]( "Datamodels: Predicting Predictions from Training Data; arXiv"). - **TRAK** - Approximates training-data effects using randomly projected model gradients and a kernel-based calculation. Evaluated against retraining-based prediction changes across vision and language tasks; these scores remain approximations to dataset interventions. [Park et al. 2023]( "TRAK: Attributing Model Behavior at Scale; ICML"). ## Concepts and representations - **Visual representations** - Feature visualization optimizes inputs for activation; Network Dissection measures unit alignment with labeled concepts (2017). Activation patterns and semantic alignment do not by themselves test causal use. [Olah et al. 2017]( "Feature Visualization; Distill"), [Bau et al. 2017]( "Network Dissection: Quantifying Interpretability of Deep Visual Representations; CVPR"). - **TCAV** - Learns concept directions from examples and measures prediction sensitivity through directional derivatives. Results depend on concept examples and the chosen representation layer. [Kim et al. 2018]( "Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV); ICML"). - **ProtoPNet** - Predicts by matching image regions to learned prototypes associated with training-image patches. Prototype similarity forms part of the predictor, rather than a separate local surrogate. [Chen et al. 2019]( "This Looks Like That: Deep Learning for Interpretable Image Recognition; NeurIPS"). - **Probing with controls** - Test what a representation makes decodable while checking what the probe itself can memorize. Decodability alone does not establish that the original model uses that information. [Hewitt and Liang 2019]( "Designing and Interpreting Probes with Control Tasks; EMNLP"). - **Concept bottleneck models** - Predict annotated concepts before predicting the target; editing predicted concept values changes downstream predictions. These are interventions inside the model. [Koh et al. 2020]( "Concept Bottleneck Models; ICML"). - **Concept leakage** - Soft concept outputs can encode task information beyond their labels, even with sequential training. Mahinpei et al. demonstrate this on synthetic and MNIST tasks; concept supervision does not ensure semantic isolation. [Mahinpei et al. 2021]( "Promises and Pitfalls of Black-Box Concept Learning Models; arXiv"). ## Comparing representations - **SVCCA** - Reduces activation matrices by SVD, then compares retained subspaces with canonical correlations. Measures shared representation across layers or models on the same examples. [Raghu et al. 2017]( "SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability; NeurIPS"). - **CKA** - Compares centered example-similarity matrices. Linear CKA is invariant to orthogonal transforms and uniform scaling, but not arbitrary invertible transforms; it compares representation geometry, not causal model use. [Kornblith et al. 2019]( "Similarity of Neural Network Representations Revisited; ICML"). - **Representation-similarity evidence** - ReSi finds no consistently best measure across its tests and graph, language and vision models. Its prediction-grounded tests use last-layer features; rankings depend on what counts as similar. [Klabunde et al. 2025]( "ReSi: A Comprehensive Benchmark for Representational Similarity Measures; ICLR"). ## Mechanistic interpretability - **Causal abstraction** - Aligns neural representations with variables in a proposed causal model and tests corresponding interchange interventions. This evaluates a specified computational hypothesis, not causal relations in the world. [Geiger et al. 2021]( "Causal Abstractions of Neural Networks; NeurIPS"). ### Candidate representations - **Superposition** - Controlled toy models store sparse features in overlapping directions, yielding polysemantic neurons. This motivates sparse dictionaries; it does not establish that a learned dictionary recovers a real network's true features. [Elhage et al. 2022]( "Toy Models of Superposition; arXiv"). - **Sparse autoencoders** - Reconstruct activations as sparse combinations of learned directions, seeking features distributed across neurons. [Bricken et al. 2023]( "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning; Transformer Circuits Thread"), [Gao et al. 2024]( "Scaling and evaluating sparse autoencoders; arXiv"). - **End-to-end sparse autoencoders** - Train sparse dictionaries through the remaining model. On GPT-2 small and TinyStories-1M, output-preserving training needs fewer active features at matched model loss. Downstream reconstruction limits changes to intermediate computation. [Braun et al. 2024]( "Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning; NeurIPS"). - **Transcoders** - Approximate dense feed-forward layers with sparse networks, enabling feature-level circuit tracing. Tests on 120M-1.4B language models find comparable sparsity, fidelity and interpretability to sparse autoencoders. [Dunefsky et al. 2024]( "Transcoders Find Interpretable LLM Feature Circuits; NeurIPS"). - **Feature absorption** - A child concept can absorb a parent concept in an SAE, causing the parent latent to miss examples. Larger dictionaries or greater sparsity need not resolve this failure. [Chanin et al. 2025]( "A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders; NeurIPS"). - **Native feature baselines** - Feed-forward key/value features compete with SAEs and transcoders on the tested interpretability measures. Dictionary widths and available checkpoints constrain the comparison. [Ye et al. 2025]( "Transformer Key-Value Memories Are Nearly as Interpretable as Sparse Autoencoders; NeurIPS"). - **SAE seed sensitivity** - Different initializations on the same model/data yield different feature dictionaries. Overlap depends on architecture and matching; one fitted SAE is not a unique feature inventory. [Paulo and Belrose 2025]( "Sparse Autoencoders Trained on the Same Data Learn Different Features; arXiv"). - **Neurons versus transcoders** - On mainly English data, sparsified neurons need a learned output correction for competitive replacement KL; transcoders reconstruct better. Neurons match top-activation concept tests but trail across broader activation quantiles. [Shabalin and Piotrowski 2026]( "Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise; NeurIPS InterpScience Workshop"). ### Interventions and circuit discovery - **Transformer circuit framework** - Decomposes attention computation into query-key and output-value paths, including interactions across layers. The initial analysis studies small attention-only models. [Elhage et al. 2021]( "A Mathematical Framework for Transformer Circuits; Transformer Circuits"). - **Circuit discovery case study** - Identifies attention-head roles in GPT-2 small's indirect-object identification task and tests faithfulness, completeness, and minimality. Findings concern that model and task. [Wang et al. 2022]( "Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small; arXiv"). - **ACDC** - Removes computational edges through activation-patching tests while seeking to retain a specified behavior. The recovered circuit depends on the dataset, intervention and metric. [Conmy et al. 2023]( "Towards Automated Circuit Discovery for Mechanistic Interpretability; NeurIPS"). - **Activation patching** - Replaces internal activations between runs to test their effect on model behavior. Zhang and Nanda show that corruption choices and evaluation metrics can change localization results. [Zhang and Nanda 2024]( "Towards Best Practices of Activation Patching in Language Models: Metrics and Methods; ICLR"). - **Sparse Feature Circuits / SHIFT** - Humans select task-irrelevant SAE circuit features for ablation, then retrain the classifier head. Marks et al. improve profession classification on gender-balanced biographies in their case study. [Marks et al. 2025]( "Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models; ICLR"). - **Circuit tracing** - Builds prompt-specific attribution graphs using transcoders that approximate model components. Reconstruction error and frozen attention limit the original method. [Ameisen et al. 2025]( "Circuit Tracing: Revealing Computational Graphs in Language Models; Transformer Circuits"). - **Attention-path extensions** - Traces feature interactions through attention query-key computations omitted by the original circuit-tracing method. [Kamath et al. 2025]( "Tracing Attention Computation Through Feature Interactions; Transformer Circuits"). - **Neuron circuits** - MLP activations with RelP attribution yield sparse circuits competitive with SAE circuits on tested tasks. Activation site and attribution rule both matter; sparsity does not establish monosemanticity. [Arora et al. 2026]( "Language Model Circuits Are Sparse in the Neuron Basis; arXiv"). - **Patching interaction effects** - Noising and denoising can measure different effects because a patched component interacts with unpatched pathways. A mediation decomposition isolates this interaction; a small individual patching effect need not imply an irrelevant component. [Vaidyanathan et al. 2026]( "The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching; NeurIPS"). - **INNSteer** - Learns an invertible transformation from paired behavioral activations. A fixed shift in its latent space becomes an input-dependent nonlinear edit after inversion. Measures behavioral control, not recovery of a causal explanation. [Nguyen and Le 2026]( "Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior; NeurIPS"). - **TopKLoRA circuits** - Sparse adapters yield smaller known-trigger backdoor circuits in 30 matched comparisons with dense LoRA on two models. Removing circuits from multi-layer adapters preserves instruction-following in an LLM-judged test. [Masiak et al. 2026]( "Interpretable sparse fine-tuning with TopKLoRA enables faithful circuit discovery; arXiv"). ### Mechanism evaluation - **MIB** - On four tasks and five models, attribution and mask optimization lead circuit localization; supervised distributed alignment search leads causal-variable localization. Tested SAE features do not outperform neurons on the latter track. [Mueller et al. 2025]( "MIB: A Mechanistic Interpretability Benchmark; ICML"). - **SAEBench** - Evaluates sparse autoencoders across eight metrics spanning interpretation, disentanglement, and downstream uses; reconstruction gains need not improve other scores. [Karvonen et al. 2025]( "SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability; ICML"); [code](). - **SAEBench reliability** - Chanin's audit tests reseed noise, synthetic ground-truth agreement and training-trajectory discrimination. Targeted Probe Perturbation and Spurious Correlation Removal fail multiple tests at canonical settings. [Chanin 2026]( "Are Sparse Autoencoder Benchmarks Reliable?; arXiv"). - **Circuit-score misranking** - Geng et al. find intervention scores favoring equally sized circuits with worse held-out behavioral agreement. Restoring intact-model signals reverses selected failures; mechanism identity remains unverified. [Geng et al. 2026]( "Are We Recovering Mechanisms? Objective-Level Recovery Gaps in Mechanistic Interpretability; arXiv"). - **Neural Chameleons** - Fine-tuning to evade benign-concept probes transfers to unseen safety probes when triggered. Monitors are trained on the final frozen checkpoint; ensembles mitigate but do not eliminate evasion. This tests trained evasion, not spontaneous concealment. [McGuinness et al. 2026]( "Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Activation Monitors; NeurIPS"). ## Evaluation and comparative evidence - **Evaluation objectives** - Distinguish application-grounded, human-grounded, and functionally grounded evaluation. A proxy score and usefulness to a person answer different questions. [Doshi-Velez and Kim 2017]( "Towards A Rigorous Science of Interpretable Machine Learning; arXiv"). - **Sources of disagreement** - Separate explainer approximation noise, changes from model fitting, and different accurate models. Changing the target is distinct from estimating the same target unreliably. [Sundararajan and Najmi 2020]( "The Many Shapley Values for Model Explanation; ICML"), [Fisher et al. 2019]( "All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously; JMLR"). - **Human usefulness** - Transparent, few-feature apartment-price models improved prediction simulation, but not advice-following; transparency sometimes impaired error correction. 3,800 participants; effects depended on presentation. [Poursabzi-Sangdeh et al. 2021]( "Manipulating and Measuring Model Interpretability; CHI"). ### Faithfulness, stability, and perturbations - **Saliency sanity checks** - Randomize model parameters or training labels to test whether explanations depend on learned behavior. Some tested methods remained visually similar after randomization. [Adebayo et al. 2018]( "Sanity Checks for Saliency Maps; NeurIPS"). - **Infidelity and sensitivity** - Compare attributed changes with output changes under perturbations, and measure explanation variation. A constant explanation can minimize sensitivity without being informative. [Yeh et al. 2019]( "On the (In)fidelity and Sensitivity for Explanations; NeurIPS"). - **Remove-and-retrain (ROAR)** - Masks highly ranked inputs and retrains to assess retained predictive information. This tests a different question from perturbing a fixed predictor. [Hooker et al. 2019]( "A Benchmark for Interpretability Methods in Deep Neural Networks; NeurIPS"). - **Faithfulness versus plausibility** - An explanation's agreement with model behavior differs from whether a person finds it convincing. Jacovi and Goldberg distinguish these evaluation targets. [Jacovi and Goldberg 2020]( "Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?; ACL"). - **Adversarial explanations** - Classifiers can behave differently on real and perturbed inputs, concealing biased predictions from LIME/SHAP in the studied attacks. [Slack et al. 2020]( "Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods; Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society"). - **Out-of-distribution perturbations** - Masking can make feature-importance scores and evaluation metrics reflect behavior outside the training distribution. Perturbation design is part of the evaluation. [Hase et al. 2021]( "The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations; NeurIPS"). ### Attention and language explanations - **Attention as explanation** - Jain and Wallace find alternative attention weights with similar predictions; Wiegreffe and Pinter challenge the inference and propose additional tests (2019). The result depends on architecture and evaluation. [Jain and Wallace 2019]( "Attention is not Explanation; NAACL"), [Wiegreffe and Pinter 2019]( "Attention is not not Explanation; EMNLP"). - **Interpretation pitfalls** - Feature dependence, interactions, poor generalization, uncertainty, and unjustified causal readings can invalidate conclusions. [Molnar et al. 2022]( "General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models; Lecture Notes in Computer Science xxAI - Beyond Explainable AI"); [docs](). - **Chain-of-thought faithfulness** - Models can use prompt hints or biases without reporting them in their explanations. Chen et al. test reasoning models. Findings are conditional on the tested tasks and hints. [Turpin et al. 2023]( "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting; NeurIPS"), [Chen et al. 2025]( "Reasoning Models Don't Always Say What They Think; arXiv"). ## Software - **Interpretable models** - InterpretML provides Explainable Boosting Machines and post-hoc methods; framework paper. [Nori et al. 2019]( "InterpretML: A Unified Framework for Machine Learning Interpretability; arXiv"); [code](). - **Feature effects and attribution** - scikit-learn for PDP/ICE and permutation importance; SHAP, LIME, ELI5 for model explanations. [docs](), [code](), [code](), [docs](). - **Neural-network attribution** - Captum provides input, neuron, and layer attribution plus evaluation metrics for PyTorch; DeepLIFT supplies reference-based propagation. [Kokhlikyan et al. 2020]( "Captum: A unified and generic model interpretability library for PyTorch; arXiv"); [docs](), [code](). - **Explanation toolkits** - AIX360 and Alibi collect multiple explanation methods; DiCE generates counterfactual examples. [code](), [code](), [code](). - **Mechanistic analysis** - TransformerLens exposes activations and interventions; SAELens trains and analyzes sparse autoencoders; circuit-tracer constructs attribution graphs and supports interventions. [code](), [code](), [code](). - **Explanation evaluation** - Quantus collects metrics for neural-network explanations; metric assumptions and perturbation choices determine what is evaluated. [Hedström et al. 2023]( "Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond; JMLR"); [code](). - **TSViz** - Combines input/filter influence, filter clustering, and inverse optimization to inspect convolutional time-series models. [Siddiqui et al. 2019]( "TSViz: Demystification of Deep Learning Models for Time-Series Analysis; IEEE Access"). ## Surveys and perspectives - **Interpretable models versus post-hoc explanations** - Rudin argues for directly interpretable predictors in high-stakes settings where suitable models can be built. [Rudin 2019]( "Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead; Nature Machine Intelligence"). - **Explanatory Model Analysis** - Model diagnostics and local/global explanations, with R and Python examples. [Biecek and Burzykowski 2021]( "Explanatory Model Analysis; book"). - **Interpretable Machine Learning** - Molnar. Interpretable models, feature effects, attribution, examples, and limitations. [Molnar 2025]( "Interpretable Machine Learning: A Guide for Making Black Box Models Explainable; book"). - **Already Clean** - Goodfire research essay argues that better visualizations and concept descriptions explain apparent polysemanticity. Its neuron examples motivate direct baselines; broad cleanliness claims remain preliminary. [Cammarata 2026]( "Neural Networks Are Already Clean; Goodfire essay"). ### Applications - **Glucose forecasting** - Applies RETAIN to longitudinal glucose prediction and examines its explanations. Evidence concerns the studied type-2 diabetes cohort. [De Bois et al. 2020]( "Interpreting Deep Glucose Predictive Models for Diabetic People Using RETAIN; Lecture Notes in Computer Science Pattern Recognition and Artificial Intelligence"). ## Related topics - [Feature selection](https://mlpapers.org/feature-selection/) - [Feature extraction](https://mlpapers.org/feature-extraction/) - [Causal inference](https://mlpapers.org/causal-inference/) - [Neural networks](https://mlpapers.org/neural-nets/) - [Ensemble learning](https://mlpapers.org/ensemble-learning/) - [Models](https://mlpapers.org/models/)