# Feature Extraction and Representation Learning 🔒 Full text at this link requires payment or institutional access. Unmarked links may be unchecked. ## Surveys - **Representation Learning: A Review and New Perspectives** - Reviews probabilistic models, autoencoders and manifold learning as approaches to learning representations. [Bengio et al. 2013]( "Representation Learning: A Review and New Perspectives; TPAMI"). - **A Survey of Dimensionality Reduction Techniques** - Categorizes dimensionality-reduction techniques and explains their mathematical foundations. [Sorzano et al. 2014]( "A survey of dimensionality reduction techniques; arXiv"). - **Feature Selection and Feature Extraction in Pattern Analysis** - Reviews the theory, applications, and numerical implementations of feature selection and extraction methods. [Ghojogh et al. 2019]( "Feature Selection and Feature Extraction in Pattern Analysis: A Literature Review; arXiv"). ## Fixed and domain-informed features ### Text and image descriptors - **Text vectorization** - Sparse word/character n-grams, TF-IDF, and feature hashing. [docs](). - **Image descriptors** - SIFT describes local keypoints; HOG summarizes local gradient orientations. [Lowe 2004]( "Distinctive Image Features from Scale-Invariant Keypoints; IJCV"), [Dalal and Triggs 2005]( "Histograms of Oriented Gradients for Human Detection; CVPR"). ### Signal and time-series transforms - **Time-series statistics** - tsfresh combines a large feature bank with relevance filtering; catch22 provides 22 selected time-series characteristics. [Christ et al. 2018]( "Time Series FeatuRe Extraction on basis of Scalable Hypothesis tests (tsfresh – A Python package); Neurocomputing"), [Lubba et al. 2019]( "catch22: CAnonical Time-series CHaracteristics: Selected through highly comparative time-series analysis; Data Min Knowl Disc"); [code](), [code](). - **Wavelet scattering** - Cascades of wavelet transforms, nonlinearities, and averaging. Kymatio implements scattering for 1D, 2D, and 3D inputs. [Andreux et al. 2020]( "Kymatio: Scattering Transforms in Python; JMLR"); [code](). - **Convolutional transforms** - ROCKET, MiniROCKET, and MultiROCKET produce convolution-response statistics for linear classification; extraction can expand dimensionality. [Dempster et al. 2020]( "ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels; Data Mining and Knowledge Discovery"), [Dempster et al. 2021]( "MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classification; KDD"), [Tan et al. 2022]( "MultiRocket: Multiple pooling operators and transformations for fast and effective time series classification; Data Mining and Knowledge Discovery"); [code](). ## Tabular feature construction - **OpenFE** - Expands tabular features with predefined operators, scores their incremental predictive value using feature boosting, and prunes candidates in two stages. Generation and supervised selection are coupled. [Zhang et al. 2023]( "OpenFE: Automated Feature Generation with Expert-level Performance; ICML"). - **TabPrep** - Jointly tunes feature generators and downstream models, unlike OpenFE's separate feature selection. With four model families and post-hoc ensembling, improves the best score on 19 of 51 TabArena datasets against newer model baselines. [Tschalzev et al. 2026]( "TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks; NeurIPS"). ## Linear and statistical representations ### Component analysis and factorization - **PCA (principal component analysis)** - Orthogonal projection maximizing retained variance of centered data. [Pearson 1901]( "LIII. On lines and planes of closest fit to systems of points in space; Philosophical Magazine"); [docs](). - **Dual PCA** - Computes PCA through the sample Gram matrix. [Ghojogh and Crowley 2019]( "Unsupervised and Supervised Principal Component Analysis: Tutorial; arXiv"). - **Incremental PCA** - Batchwise PCA updates. [docs](), [code](). - **Truncated SVD / latent semantic analysis** - Low-rank approximation without PCA's centering step, allowing sparse document-term inputs. [Deerwester et al. 1990]( "Indexing by latent semantic analysis; J. Am. Soc. Inf. Sci."); [docs](). - **NMF (non-negative matrix factorization)** - Additive factorization of non-negative data. [Lee and Seung 1999 🔒]( "Learning the parts of objects by non-negative matrix factorization; Nature"); [docs](). - **Factor analysis** - A linear latent-factor model with feature-specific Gaussian noise. [docs](). - **ICA (independent component analysis)** - Linear separation of statistically independent sources. [Hyvärinen and Oja 2000]( "Independent component analysis: algorithms and applications; Neural Networks"), [Hyvärinen et al. 2001]( "Independent Component Analysis; book"); [docs](). - **Topic LDA (latent Dirichlet allocation)** - Document representations as topic mixtures. [Blei et al. 2003]( "Latent Dirichlet Allocation; JMLR"), [Hoffman et al. 2010]( "Online Learning for Latent Dirichlet Allocation; NeurIPS"). - **Robust PCA / principal component pursuit** - Separates low-rank structure from sparse corruption (`X = L + S`), with recovery guarantees under rank, incoherence, and corruption assumptions. [Candes et al. 2011]( "Robust Principal Component Analysis?; JACM"). ### Sparse representations - **Sparse PCA** - Sparse component loadings, selecting which input variables contribute to each component. [Zou et al. 2006]( "Sparse Principal Component Analysis; JCGS"); [docs](). - **Dictionary learning / sparse coding** - Sparse sample codes over learned, possibly overcomplete dictionaries; sparsity applies to the codes rather than PCA loadings. [Mairal et al. 2010]( "Online Learning for Matrix Factorization and Sparse Coding; JMLR"); [docs](). ### Supervised and multiview projections - **Fisher LDA (linear discriminant analysis)** - Supervised projection maximizing between-class relative to within-class scatter. [Fisher 1936]( "The Use of Multiple Measurements in Taxonomic Problems; Annals of Eugenics"), [Rao 1948]( "The Utilization of Multiple Measurements in Problems of Biological Classification; JRSS B"), [Martinez and Kak 2001]( "PCA versus LDA; IEEE Trans. Pattern Anal. Machine Intell."); [docs](). - **Supervised PCA** - Response-based predictor screening followed by PCA. [Bair et al. 2006]( "Prediction by Supervised Principal Components; JASA"). - **CCA / PLS** - Canonical correlation analysis maximizes correlation between projected views; partial least squares extracts cross-covariance-based components, including for regression. [docs](). ## Kernel methods and random projections - **Kernel PCA** - Component analysis in an implicit kernel feature space. [Schölkopf et al. 1998]( "Nonlinear Component Analysis as a Kernel Eigenvalue Problem; Neural Comput."); [docs](). - **Kernel PCA extensions** - Kernel PCA for novelty detection and robust kernel PCA. [Hoffmann 2007]( "Kernel PCA for Novelty Detection; Pattern Recognition"), [Nguyen and De la Torre 2008]( "Robust Kernel Principal Component Analysis; Advances in Neural Information Processing Systems"). - **Kernel Fisher discriminant analysis** - Supervised class separation in a kernel feature space. [Mika et al. 1999]( "Fisher discriminant analysis with kernels; NNSP"). - **Nyström approximation** - Sampled data landmarks approximate the kernel matrix; an alternative to Fourier sampling. [Williams and Seeger 2000]( "Using the Nyström Method to Speed Up Kernel Machines; NeurIPS"); [docs](). - **Random projections** - Gaussian or sparse random maps approximately preserve Euclidean distances under Johnson-Lindenstrauss conditions. [Achlioptas 2003]( "Database-friendly random projections: Johnson-Lindenstrauss with binary coins; JCSS"); [docs](). - **Random Fourier features** - Sampled Fourier maps approximate shift-invariant kernels. Unlike distance-preserving random projections, these construct nonlinear features and may expand dimension. [Rahimi and Recht 2007]( "Random Features for Large-Scale Kernel Machines; NeurIPS"). ## Manifold learning and visualization ### Distance and spectral methods - **MDS (multidimensional scaling)** - Fits coordinates to dissimilarities or their ordering. [Kruskal 1964]( "Multidimensional Scaling by Optimizing Goodness of Fit to a Nonmetric Hypothesis; Psychometrika"), [Little et al. 2018]( "Exact Cluster Recovery via Classical Multidimensional Scaling; arXiv"), [Little et al. 2022]( "An analysis of classical multidimensional scaling with applications to clustering; Information and Inference: A Journal of the IMA"); [docs](). - **Sammon mapping** - Weighted distance reconstruction emphasizing smaller input distances. [Sammon 1969]( "A Nonlinear Mapping for Data Structure Analysis; IEEE Trans. Comput."). - **SOM (self-organizing maps)** - Updates grid-organized prototypes and their neighbors. [Kohonen 1982 🔒]( "Self-organized formation of topologically correct feature maps; Biol. Cybern."). - **Isomap** - MDS on graph-approximated geodesic distances. [Tenenbaum et al. 2000]( "A Global Geometric Framework for Nonlinear Dimensionality Reduction; Science"); [docs](). - **LLE (locally linear embedding)** - Fits coordinates using fixed local linear reconstruction weights. [Roweis and Saul 2000]( "Nonlinear Dimensionality Reduction by Locally Linear Embedding; Science"); [docs](). - **HLLE (Hessian eigenmapping)** - Manifold coordinates from local Hessian estimates. Supported by scikit-learn's LLE interface. [Donoho and Grimes 2003]( "Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data; PNAS"). - **Laplacian eigenmaps / spectral embedding** - Graph-local similarity represented by Laplacian eigenvectors. [Belkin and Niyogi 2003]( "Laplacian Eigenmaps for Dimensionality Reduction and Data Representation; Neural Comput."). - **Maximum variance unfolding / semidefinite embedding** - Maximizes spread while preserving local distances through semidefinite optimization. [Weinberger et al. 2005]( "Nonlinear Dimensionality Reduction by Semidefinite Programming and Kernel Matrix Factorization; AISTATS"). - **Diffusion maps** - Spectral coordinates reflecting random-walk connectivity across scales. [Coifman and Lafon 2006]( "Diffusion maps; ACHA"). - **PHATE** - Embeds distances between transformed diffusion probabilities to visualize continuous and branching structure. [Moon et al. 2019 🔒]( "Visualizing structure and transitions in high-dimensional biological data; Nat Biotechnol"); [code](). ### Neighborhood objectives and learned mappings - **t-SNE** - Local neighborhood probability matching. [van der Maaten and Hinton 2008]( "Visualizing Data using t-SNE; JMLR"), [van der Maaten 2014]( "Accelerating t-SNE using Tree-Based Algorithms; Journal of Machine Learning Research"); [docs](), [docs](). - **Extensions** - Tree-SNE for hierarchical clustering; LEt-SNE for hyperspectral imagery. [Robinson and Pierce-Hoffman 2020]( "Tree-SNE: Hierarchical Clustering and Visualization Using t-SNE; arXiv"), [Shukla et al. 2019]( "LEt-SNE: A Hybrid Approach To Data Embedding and Visualization of Hyperspectral Imagery; arXiv"); [code](). - **LargeVis** - Approximate neighborhood-graph construction and probabilistic layout for large collections. [Tang et al. 2016]( "Visualizing Large-scale and High-dimensional Data; WWW"). - **UMAP** - Embeds a fuzzy neighborhood graph. [McInnes et al. 2018]( "UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction; arXiv"); [code](), [docs](). - **Parametric UMAP** - Trains a neural encoder with the UMAP objective. [Sainburg et al. 2021]( "Parametric UMAP embeddings for representation and semi-supervised learning; Neural Comput."); [docs](). - **densMAP** - Adds relative local-density preservation to UMAP. [Narayan et al. 2021 🔒]( "Assessing single-cell transcriptomic variability through density-preserving data visualization; Nat Biotechnol"); [docs](). - **MAPLE** - Adds self-supervised manifold modeling to UMAP using maximum manifold capacity representations. [Huang et al. 2026]( "MAPLE: Self-Supervised Learning-Enhanced Nonlinear Dimensionality Reduction for Visual Analysis; TVCG"), [Huang et al. 2026]( "MAPLE: Self-Supervised Learning-Enhanced Nonlinear Dimensionality Reduction for Visual Analysis; IEEE Trans. Visual. Comput. Graphics"). - **TriMap** - Relative similarity preservation through triplets. [Amid and Warmuth 2019]( "TriMap: Large-scale Dimensionality Reduction Using Triplets; arXiv"); [code](). - **PaCMAP** - Combines neighbor, mid-near, and distant pairs to preserve structure at different scales. [Wang et al. 2021]( "Understanding How Dimension Reduction Tools Work: An Empirical Approach to Deciphering t-SNE, UMAP, TriMAP, and PaCMAP for Data Visualization; JMLR"); [code](). - **LocalMAP** - Adjusts the neighborhood graph during optimization to improve local cluster separation. Implemented in the PaCMAP repository. [Wang et al. 2025]( "Dimension Reduction with Locally Adjusted Graphs; AAAI"). - **ParamRepulsor** - Hard-negative mining and stronger repulsion address loss of local detail in parametric embeddings. [Huang et al. 2024]( "Navigating the Effect of Parametrization for Dimensionality Reduction; NeurIPS"); [code](). - **New observations** - Ordinary UMAP and openTSNE place new points relative to a fitted embedding. Parametric UMAP instead evaluates a learned encoder; new-point support alone does not imply a neural mapping. [docs](), [docs](). ## Learned representations ### Reconstruction and masking - **Autoencoders** - Encoder-decoder learning through reconstruction. [Hinton and Salakhutdinov 2006]( "Reducing the Dimensionality of Data with Neural Networks; Science"). - **Denoising** - Clean-input reconstruction from corrupted observations. [Vincent et al. 2010]( "Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion; JMLR"). - **Variational** - Probabilistic latent-variable modeling with an approximate posterior. [Kingma and Welling 2014]( "Auto-Encoding Variational Bayes; ICLR"). - **Masked** - Reconstructs hidden image patches from visible ones. [He et al. 2022]( "Masked Autoencoders Are Scalable Vision Learners; CVPR"); [code](). ### Contrastive learning - **Contrastive Predictive Coding** - Predicts future latent representations from autoregressive context using a contrastive loss and negative samples; learns sequential features without reconstructing raw observations. [van den Oord et al. 2018]( "Representation Learning with Contrastive Predictive Coding; arXiv"). - **SimCLR** - Learns image representations by contrasting augmented views through a nonlinear projection head. ImageNet linear evaluation favors encoder features before the head over its output. [Chen et al. 2020]( "A Simple Framework for Contrastive Learning of Visual Representations; ICML"); [code](). - **Alignment and uniformity** - Relates contrastive learning to positive-pair alignment and uniform spread on the unit sphere. The analysis is asymptotic; normalization alone does not guarantee uniform learned features. [Wang and Isola 2020]( "Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere; ICML"). ### Self-distillation and redundancy reduction - **BYOL** - An online network predicts a moving target network's representation without explicit negative pairs. [Grill et al. 2020]( "Bootstrap your own latent: A new approach to self-supervised Learning; NeurIPS"). - **DINO** - Self-distillation without labels using a momentum teacher; the precursor to [DINOv2/v3](#images-and-video). [Caron et al. 2021]( "Emerging Properties in Self-Supervised Vision Transformers; ICCV"). - **Barlow Twins** - Drives cross-correlation between two augmented views toward identity, combining agreement with redundancy reduction. This objective also underlies [TLDR](#representation-compression). [Zbontar et al. 2021]( "Barlow Twins: Self-Supervised Learning via Redundancy Reduction; ICML"). - **VICReg** - Separates view agreement, per-feature variance control, and covariance regularization. Unlike Barlow Twins' cross-correlation objective, it explicitly constrains each branch's variance. [Bardes et al. 2022]( "VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning; ICLR"); [code](). ### Prediction in representation space - **I-JEPA** - Predicts representations of masked image regions from visible context, rather than reconstructing pixels as in MAE. [Assran et al. 2023]( "Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture; CVPR"). - **JEPA learning bias** - In deep linear models, JEPA's feature-learning order depends on regression coefficients and input-target covariance; masked reconstruction is led by covariance. Theory assumes diagonal covariances, small aligned initialization and stop-gradient targets. [Littwin et al. 2024]( "How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks; NeurIPS"). - **LeJEPA** - Combines a joint-embedding predictive objective with sketched isotropic Gaussian regularization of embeddings. [Balestriero and LeCun 2025]( "LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics; arXiv"). - **JEPA predictive closure** - In CIFAR-10 sequences with three-dimensional SIGReg latents and no stop-gradient, an unpredictable cue becomes more decodable when it helps predict other features. Altering predictive links shifts which feature sets are represented. [Levy et al. 2026]( "Predictive Closure: A Set-Level Characterization of JEPA’s Feature Selection; arXiv"); [code](). - **Spectral Ghost** - Develops a spectral account of representation sufficiency connecting component analysis and self-supervised objectives. [Dai et al. 2026]( "Spectral Ghost in Representation Learning: from Component Analysis to Self-Supervised Learning; arXiv"). ## Identifiability and disentanglement - **Limits of unsupervised disentanglement** - Locatello et al. show that recovering independent factors requires inductive assumptions. Across seven datasets, stronger disentanglement does not consistently reduce downstream sample needs; factor recovery is a distinct goal. [Locatello et al. 2019]( "Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations; ICML"). - **Identifiable VAE / nonlinear ICA** - Conditions a factorized latent prior on an observed auxiliary variable. Identifiability holds up to specified transformations under variation and regularity conditions; an ordinary VAE with an unconditional prior lacks this guarantee. [Khemakhem et al. 2020]( "Variational Autoencoders and Nonlinear ICA: A Unifying Framework; AISTATS"). ## Extraction by modality ### Images and video - **DINOv2** - Scales self-supervised learning to produce reusable image-level and patch-level features. [Oquab et al. 2024]( "DINOv2: Learning Robust Visual Features without Supervision; TMLR"); [code](). - **DINOv3** - Gram anchoring addresses degradation of dense features during extended self-supervised training. [Siméoni et al. 2026]( "DINOv3; NeurIPS"); [code](). - **V-JEPA 2.1** - Dense prediction over visible and masked tokens, plus intermediate-layer self-supervision, extends JEPA features to spatially structured image/video representations. [Mur-Labadia et al. 2026]( "V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning; ECCV"); [code](). - **FINO** - Adapts visual representations to scientific domains using discrete or continuous metadata alongside self-supervised learning. [Gardès et al. 2026]( "Who Needs Labels? Adapting Vision Foundation Models With the Metadata You Already Have; NeurIPS"). ### Text and retrieval #### Sentence and document vectors - **Sentence-BERT** - Siamese/triplet training makes BERT sentence vectors usable with cosine similarity, avoiding joint query-document encoding. [Reimers and Gurevych 2019]( "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks; EMNLP"); [code](). - **E5** - Contrastive pretraining on weakly supervised text pairs; evaluated both without labeled-data fine-tuning and after supervised fine-tuning. [Wang et al. 2022]( "Text Embeddings by Weakly-Supervised Contrastive Pre-training; arXiv"); [code](). - **Qwen3 Embedding** - Adapts language-model backbones for instruction-aware multilingual text embeddings. [Zhang et al. 2025]( "Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models; arXiv"); [code](). #### Document context and chunk pooling - **Late chunking** - Encodes the document before pooling spans into chunk vectors. Across three encoders and four BEIR datasets, mean nDCG@10 rises from 52.2 to 54.0 with 256-token chunks; individual settings include ties and regressions. [Günther et al. 2024]( "Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models; arXiv"). - **pplx-embed-context** - Combines diffusion-pretrained bidirectional encoders, contrastive learning, and late chunking for document-context-aware passage vectors. [Eslami et al. 2026]( "Diffusion-Pretrained Dense and Contextual Embeddings; ACL Industry"). #### Sparse and multivector scoring - **ColBERT** - Independently encodes queries and passages into token vectors; sums each query token's maximum similarity to a passage token. Late interaction retains multiple vectors, whereas late chunking pools each chunk. [Khattab and Zaharia 2020]( "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT; SIGIR"); [code](). - **BGE-M3** - Jointly supports dense vectors, sparse lexical weights, and token multivectors through self-knowledge distillation. [Chen et al. 2024]( "M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation; Findings of ACL"); [code](). - **Multivector expressivity** - Proves a worst-case representation-size separation between Chamfer similarity and single-vector inner products; this is not an empirical retrieval ranking. [Jayaram 2026]( "Multi-Vector Embeddings are Provably More Expressive than Single Vector Embeddings; arXiv"). ### Audio and time series - **wav2vec 2.0** - Masks latent speech representations and predicts jointly learned quantized targets with a contrastive loss. Unlabeled audio pretraining precedes supervised speech-recognition fine-tuning. [Baevski et al. 2020]( "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations; NeurIPS"). - **TS2Vec** - Contrastive learning of contextual timestamp features, pooled for subsequences or whole series. [Yue et al. 2022]( "TS2Vec: Towards Universal Representation of Time Series; AAAI"); [code](). - **BEATs** - Iterates acoustic tokenization and masked audio-token prediction to learn general audio representations. [Chen et al. 2023]( "BEATs: Audio Pre-Training with Acoustic Tokenizers; ICML"); [code](). - **MOMENT** - Masked time-series pretraining supports classification, forecasting, imputation, and anomaly detection. [Goswami et al. 2024]( "MOMENT: A Family of Open Time-series Foundation Models; ICML"); [code](). - **MantisV2** - Uses synthetic pretraining, layer selection and token aggregation for frozen time-series features. UCR ablations favor intermediate layers after more updates, but the final layer with 100K samples and 100 epochs; fine-tuning favors all layers. [Feofanov et al. 2026]( "MantisV2: Closing the Zero-Shot Gap in Time Series Classification with Synthetic Data and Test-Time Strategies; arXiv"); [code](). ### Cross-modal representations - **CLIP features** - Paired image and text encoders produce embeddings for cross-modal comparison; see [Multi-View Learning](https://mlpapers.org/multiview-learning/) for CLIP's objective and implementation. - **CLAP** - Contrastively trains audio and text encoders on paired recordings and descriptions. Their shared space supports audio retrieval and zero-shot classification through text prompts. [Elizalde et al. 2023]( "CLAP: Learning Audio Concepts From Natural Language Supervision; ICASSP"). - **SigLIP 2** - Combines sigmoid image-text learning with self-distillation and masked prediction for multilingual alignment and dense visual features. [Tschannen et al. 2025]( "SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features; arXiv"); [code](). - **Qwen3-VL-Embedding** - Shared vectors for text, images, document images, and video. The accompanying reranker scores pairs rather than producing independently indexable vectors. [Li et al. 2026]( "Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking; arXiv"); [code](). ## Representation compression - **Matryoshka representation learning** - Trains nested vector prefixes at multiple dimensions. Compression is built into representation training; arbitrary truncation of ordinary embeddings is not equivalent. [Kusupati et al. 2022]( "Matryoshka Representation Learning; NeurIPS"); [code](). - **TLDR** - Compresses features using neighbor pairs and the Barlow Twins objective. On ROxford retrieval, the reported 128-dimensional codes gain 4.1 mAP points over whitened PCA with the tested GeM features and fitting data. [Kalantidis et al. 2022]( "TLDR: Twin Learning for Dimensionality Reduction; TMLR"); [code](). - **Retrieval dimension and margin** - For unit-vector retrieval supporting every k-subset of n documents, optimal-order margin requires dimension of order k log(n/k). This worst-case relevance family does not determine the dimension needed for a particular corpus. [Bangachev et al. 2026]( "Is Dimensionality a Barrier for Retrieval Models?; NeurIPS"). ## Evaluation and comparative evidence ### Benchmarks and geometry - **Neighborhood preservation** - Trustworthiness penalizes introduced neighbors; continuity penalizes lost neighbors. These measure geometry rather than held-out task performance. [Venna and Kaski 2006]( "Local multidimensional scaling; Neural Networks"); [docs](). - **Intrinsic dimension** - Global and local estimators characterize data geometry; their estimates are not task-optimized embedding dimensions. [Bac et al. 2021]( "Scikit-dimension: a Python package for intrinsic dimension estimation; Entropy"); [docs](). - **Local versus global structure** - On tested 2D layouts, t-SNE/art-SNE preserve neighborhoods better, while PCA/TriMap/PaCMAP/ForceAtlas2 preserve distance ranks better on most datasets. These visualization results do not rank general predictive representations. [Huang et al. 2022]( "Towards a comprehensive evaluation of dimension reduction methods for transcriptomic data visualization; Commun. Biol."). - **MTEB / MMTEB** - Text-embedding evaluation across tasks, domains, and languages. [Muennighoff et al. 2023]( "MTEB: Massive Text Embedding Benchmark; EACL"), [Enevoldsen et al. 2025]( "MMTEB: Massive Multilingual Text Embedding Benchmark; ICLR"); [code](). ## Software - **Python (reduction and dimension estimation)** - scikit-learn, umap-learn, openTSNE, PaCMAP/LocalMAP, PHATE, scikit-dimension. [docs](), [docs](), [docs](), [code](), [code](), [docs](). - **Python (extraction)** - Sentence Transformers, timm, Kymatio, tsfresh; evaluation: MTEB. [docs](), [docs](), [docs](), [docs](), [code](). - **R** - dimRed, dyndimred, intrinsicDimension, Rdimtools; discriminant analysis: MASS, sda. [You 2022]( "Rdimtools: An R package for Dimension Reduction and Intrinsic Dimension Estimation; Software Impacts"); [code](), [code](), [code](), [code](), [code](), [code](). - **JavaScript (extraction)** - Transformers.js runs compatible pretrained models through ONNX Runtime with WASM/WebGPU. [docs](). - **JavaScript (visualization)** - tsne, umap-js, dimred; interactive tool: StatSim Vis. [docs](), [docs](), [docs](), [docs](). - **C++** - Tapkee. [code](). ## Connections - **Sparse autoencoders for interpretability** - Overcomplete sparse features from internal model activations, evaluated for reconstruction and interpretability rather than compactness alone. [Gao et al. 2024]( "Scaling and evaluating sparse autoencoders; arXiv"). ## Related topics - [Feature selection](https://mlpapers.org/feature-selection/) - [Neural networks](https://mlpapers.org/neural-nets/) - [Multiview learning](https://mlpapers.org/multiview-learning/) - [Interpretability](https://mlpapers.org/interpretability/)