---
title: "Natural Language Processing"
aliases: ["NLP", "Language Models", "Text Mining"]
tags:
  - nlp
  - deep-learning
  - transformers
  - language-models
---

# Natural Language Processing

## Statistical models and structured prediction

- **Maximum-entropy language models** - Combines observed feature statistics in a probabilistic model and estimates parameters by maximum likelihood. Develops feature selection and estimation procedures for language-processing tasks. [Berger et al. 1996](<https://aclanthology.org/J96-1002/> "A Maximum Entropy Approach to Natural Language Processing; Computational Linguistics").
- **N-gram smoothing comparisons** - Compares smoothing methods across corpora, training sizes, and n-gram orders using held-out cross-entropy. Shows that model settings and data scale affect which estimator works best. [Chen and Goodman 1996](<https://aclanthology.org/P96-1041/> "An Empirical Study of Smoothing Techniques for Language Modeling; ACL").
- **Conditional random fields** - Models label sequences conditional on observations, allowing dependent labels and rich input features. Avoids the label-bias limitation of locally normalized maximum-entropy Markov models. [Lafferty et al. 2001](<https://daiwk.github.io/assets/Conditional%20random%20fields%20Probabilistic%20models%20for%20segmenting%20and%20labeling%20sequence%20data.pdf> "Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data; ICML").
- **Neural tagging from learned features** - Uses a shared neural approach for part-of-speech tagging, chunking, entity recognition, and semantic roles. Learns representations from largely unlabeled text instead of engineering separate input features for every task. [Collobert et al. 2011](<https://arxiv.org/abs/1103.0398> "Natural Language Processing (almost) from Scratch; JMLR").
- **Naive Bayes and SVM baselines** - Studies unigram/bigram features and NB/SVM variants for sentiment and topic classification. Uses NB log-count ratios as SVM features and shows why a precisely specified baseline matters. [Wang and Manning 2012](<https://aclanthology.org/P12-2018/> "Baselines and Bigrams: Simple, Good Sentiment and Topic Classification; ACL").
- **Neural dependency parsing** - Uses learned dense features to choose transitions in a greedy dependency parser. Replaces large sparse feature-template sets with compact representations of parsing configurations. [Chen and Manning 2014](<https://aclanthology.org/D14-1082/> "A Fast and Accurate Dependency Parser using Neural Networks; EMNLP").

## Tokenization and word representations

- **word2vec** - Learns word vectors through continuous bag-of-words and skip-gram prediction objectives. Produces a reusable vector per vocabulary word; the representation does not change with each sentence context. [Mikolov et al. 2013](<https://arxiv.org/abs/1301.3781> "Efficient Estimation of Word Representations in Vector Space; arXiv").
- **GloVe** - Fits a weighted log-bilinear model to global word co-occurrence statistics. Combines corpus-level counts with vector representations rather than learning only from individual context windows. [Pennington et al. 2014](<https://aclanthology.org/D14-1162/> "GloVe: Global Vectors for Word Representation; EMNLP").
- **Subword translation with BPE** - Represents rare and unseen words as sequences of subword units. Adapts byte-pair encoding to vocabulary construction for translation, reducing dependence on whole-word dictionaries. [Sennrich et al. 2016](<https://arxiv.org/abs/1508.07909> "Neural Machine Translation of Rare Words with Subword Units; ACL").
- **fastText subword vectors** - Represents words by sums of character n-gram vectors within a skip-gram model. Can construct vectors for unseen words from their subword components. [Bojanowski et al. 2017](<https://arxiv.org/abs/1607.04606> "Enriching Word Vectors with Subword Information; Transactions of the Association for Computational Linguistics").
- **fastText classification** - Uses a simple supervised text-classification architecture with word and n-gram features. Provides a computationally inexpensive baseline distinct from the fastText method for unsupervised word vectors. [Joulin et al. 2017](<https://arxiv.org/abs/1607.01759> "Bag of Tricks for Efficient Text Classification; EACL").
- **SentencePiece** - Trains subword tokenization directly from raw sentences without requiring prior word segmentation. Provides tokenization and detokenization for languages with different writing conventions. [Kudo and Richardson 2018](<https://arxiv.org/abs/1808.06226> "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing; EMNLP").

## Contextual pretraining

- **ELMo** - Builds context-dependent word representations from internal states of a pretrained bidirectional language model. Allows downstream tasks to combine signals from different layers. [Peters et al. 2018](<https://arxiv.org/abs/1802.05365> "Deep contextualized word representations; NAACL").
- **BERT** - Pretrains bidirectional Transformer representations with masked-token prediction and an original next-sentence objective. Fine-tunes the encoder for tasks such as classification and extractive question answering. [Devlin et al. 2019](<https://arxiv.org/abs/1810.04805> "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding; NAACL").
    - **RoBERTa** - Reexamines BERT training choices, data size, and optimization. Demonstrates that training conditions can explain apparent differences between pretraining methods. [Liu et al. 2019](<https://arxiv.org/abs/1907.11692> "RoBERTa: A Robustly Optimized BERT Pretraining Approach; arXiv").
- **XLNet** - Trains autoregressive predictions across permutations of the factorization order to learn bidirectional context. Integrates Transformer-XL mechanisms without using BERT's masked-token objective. [Yang et al. 2019](<https://arxiv.org/abs/1906.08237> "XLNet: Generalized Autoregressive Pretraining for Language Understanding; NeurIPS").
- **ELECTRA** - Trains an encoder to distinguish original tokens from plausible generator replacements. Uses a discrimination target across input positions rather than predicting only the identities of masked tokens. [Clark et al. 2020](<https://arxiv.org/abs/2003.10555> "ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators; ICLR").
- **Multi-token prediction** - Trains future-token heads on a shared trunk. At matched 13B parameter counts, four-token training raises HumanEval pass@1 from 14.1% to 15.8% and MBPP from 26.0% to 30.5%. Smaller models can regress. The heads also draft tokens for decoding. [Gloeckle et al. 2024](<https://proceedings.mlr.press/v235/gloeckle24a.html> "Better &amp; Faster Large Language Models via Multi-token Prediction; ICML").
    - **Soft-MTP** - Routes existing cross-entropy gradients so future-token heads learn from the longer-context main head, while the shared output layer keeps hard-label training. At 1.2B, average zero-shot accuracy rises from 56.80% to 57.31% over standard MTP. [Teoh 2026](<https://arxiv.org/abs/2610.38331v1> "Multi-Token Prediction Done Right; arXiv"); [code](<https://github.com/JaydenTeoh/Soft-MTP>).
- **ModernBERT** - Updates bidirectional encoder pretraining with rotary positions and alternating local/global attention; evaluated for classification and both single-vector and late-interaction retrieval. [Warner et al. 2025](<https://arxiv.org/abs/2412.13663> "Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference; ACL").

## Sequence generation and translation

- **Sequence-to-sequence LSTMs** - Encodes an input sequence into a fixed-dimensional vector and decodes a target sequence with another LSTM. Establishes a jointly trained neural architecture for sequence transduction. [Sutskever et al. 2014](<https://arxiv.org/abs/1409.3215> "Sequence to Sequence Learning with Neural Networks; NeurIPS").
- **Neural attention for translation** - Learns soft alignments so each generated word can depend on relevant source positions. Relieves the fixed-vector bottleneck of early encoder-decoder translation models. [Bahdanau et al. 2015](<https://arxiv.org/abs/1409.0473> "Neural Machine Translation by Jointly Learning to Align and Translate; ICLR").
- **Transformer architecture** - See [Neural Networks](https://mlpapers.org/neural-nets/#sequence-models-attention-and-implicit-depth) for self-attention, positional representations and architectural follow-ups.
- **T5** - Expresses classification, question answering, translation, and other tasks in a common text-to-text format. Compares pretraining objectives, architectures, datasets, and transfer procedures within that framework. [Raffel et al. 2020](<https://arxiv.org/abs/1910.10683> "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer; JMLR").
- **BART** - Pretrains an encoder-decoder by reconstructing text corrupted with noise, including span masking and sentence reordering. Connects bidirectional encoding with left-to-right generation. [Lewis et al. 2020](<https://arxiv.org/abs/1910.13461> "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension; ACL").
- **Vision-language generation** - Flamingo and LLaVA condition language generation on images; see [Multi-View Learning](https://mlpapers.org/multiview-learning/) for architectures and visual instruction tuning.

## Multilingual learning

- **XLM-R** - Pretrains a masked language model across 100 languages. Studies cross-lingual transfer and the tradeoff between sharing representations and dividing finite model capacity among languages. [Conneau et al. 2020](<https://arxiv.org/abs/1911.02116> "Unsupervised Cross-lingual Representation Learning at Scale; ACL").
- **mT5** - Extends text-to-text pretraining to multilingual data. Studies cross-lingual task transfer and accidental translation, where generated answers switch away from the requested language. [Xue et al. 2021](<https://arxiv.org/abs/2010.11934> "mT5: A massively multilingual pre-trained text-to-text transformer; NAACL").

## Scaling and in-context learning

- **Language-model scaling laws** - Fits loss against model size, data and compute. In the tested regime, compute-optimal parameter count grows faster than token count, favoring larger models stopped before convergence. [Kaplan et al. 2020](<https://arxiv.org/abs/2001.08361> "Scaling Laws for Neural Language Models; arXiv").
    - **Compute-optimal training** - Chinchilla finds roughly equal parameter/token scaling, near 20 tokens per parameter. Refitting its third procedure corrects premature fit termination and rounded coefficients, widens uncertainty, and agrees with its other two methods. [Hoffmann et al. 2022](<https://arxiv.org/abs/2203.15556> "Training Compute-Optimal Large Language Models; NeurIPS"), [Besiroglu et al. 2024](<https://arxiv.org/abs/2404.10102v2> "Chinchilla Scaling: A replication attempt; arXiv").
- **GPT-3 and few-shot prompting** - Studies task demonstrations supplied in the prompt without task-specific gradient updates. Distinguishes in-context conditioning from fine-tuning. [Brown et al. 2020](<https://arxiv.org/abs/2005.14165> "Language Models are Few-Shot Learners; NeurIPS").
- **Concentrated data repetition** - Repeating a small subset can worsen test loss despite 90% unique training tokens. At the worst repetition level, an 800M model approaches a 340M model's loss. Copying and induction-head measures also degrade. [Hernandez et al. 2022](<https://arxiv.org/abs/2205.10487v1> "Scaling Laws and Interpretability of Learning from Repeated Data; arXiv").
- **Data-constrained scaling** - At fixed compute, up to four epochs of repeated data yield nearly the same loss as fresh data in tested language models. A fitted law discounts repeated tokens, capturing diminishing returns but not the observed loss increases at heavy repetition. [Muennighoff et al. 2023](<https://arxiv.org/abs/2305.16264v5> "Scaling Data-Constrained Language Models; NeurIPS"); [code](<https://github.com/huggingface/datablations>).
    - **Repetition and overfitting** - Adds an overfitting penalty to data-constrained scaling. At three tested data/compute budgets, its prescribed larger models with fewer epochs achieve lower perplexity than Chinchilla and the earlier effective-data law. [Lovelace et al. 2026](<https://arxiv.org/abs/2605.01640v1> "Prescriptive Scaling Laws for Data Constrained Training; arXiv").
    - **Compute-data scaling** - Models the diminishing value of repeated or paraphrased tokens. On Dolma-3, a fit on 14M/30M models predicts 60M-600M losses with log-loss RMSE 0.079. Only learning rate and decay are tuned; larger-model advice is extrapolation. [Qin et al. 2026](<https://arxiv.org/abs/2607.25271v1> "Bridging Compute- and Data-Optimal Pretraining; NeurIPS").
- **Joint MoE scaling** - At 1.1B total parameters, 2- and 4-expert MoEs reach lower loss than a dense model over a tested FLOPs range, using more tokens at matched compute. The dense model is past its compute-optimal token count. Costs use FLOPs, not measured time. [Ludziejewski et al. 2025](<https://proceedings.mlr.press/v267/ludziejewski25a.html> "Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient; ICML").
- **Data filtering and compute** - On 670M-10B-token Common Crawl pools, sufficiently trained dense models outperform tested filtered subsets in language-modeling loss. Filtering still helps at low compute; full-crawl predictions extrapolate beyond the experiments. [Mohri et al. 2026](<https://arxiv.org/abs/2605.19407v1> "A Bitter Lesson for Data Filtering; NeurIPS").
- **Hyperparameter transfer** - See [Optimization](https://mlpapers.org/optimization/#hyperparameter-transfer) for width transfer, token-horizon effects and joint learning-rate/batch-size scaling.

## Instruction and preference learning

- **Instruction tuning / FLAN** - Fine-tunes on multiple tasks expressed as natural-language instructions and evaluates unseen task types. Studies how instruction format, task diversity, and model scale affect transfer. [Wei et al. 2022](<https://arxiv.org/abs/2109.01652> "Finetuned Language Models Are Zero-Shot Learners; ICLR").
- **InstructGPT** - Combines supervised demonstrations with preference rankings and reinforcement learning. Studies following instructions on a specified prompt distribution; preference training does not eliminate factual errors. [Ouyang et al. 2022](<https://arxiv.org/abs/2203.02155> "Training language models to follow instructions with human feedback; NeurIPS").
- **Constitutional AI** - Uses written principles to guide model critiques, revised responses, and AI-generated preference feedback. Combines supervised and reinforcement learning while retaining human choices about the principles. [Bai et al. 2022](<https://arxiv.org/abs/2212.08073> "Constitutional AI: Harmlessness from AI Feedback; arXiv").
- **Direct Preference Optimization** - Reparameterizes a reward-based preference objective as direct policy training with a classification loss. Uses preference pairs without a separate reward-model fitting stage or online sampling during fine-tuning. [Rafailov et al. 2023](<https://arxiv.org/abs/2305.18290> "Direct Preference Optimization: Your Language Model is Secretly a Reward Model; NeurIPS").
- **Unified training** - Fits static text and reward-checked responses with one weighted likelihood loss. Mixing beats staged training at larger estimated FLOP budgets in a 124M puzzle model. In a 0.76B model, it reduces forgetting but lowers single-attempt coding accuracy. [Song et al. 2026](<https://arxiv.org/abs/2610.40564v1> "Training Foundation Models with a Single Algorithm; arXiv"); [code](<https://github.com/yudasong/unified-training>).

## Sentence representations and retrieval

### Sentence and document vectors

- **Sentence-BERT** - Siamese/triplet training makes BERT sentence vectors usable with cosine similarity, avoiding joint query-document encoding. [Reimers and Gurevych 2019](<https://arxiv.org/abs/1908.10084> "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks; EMNLP"); [code](<https://github.com/UKPLab/sentence-transformers>).
- **SimCSE** - Learns sentence embeddings contrastively using dropout views, or supervised entailment positives and contradiction negatives. Studies how these objectives shape embedding alignment and uniformity. [Gao et al. 2021](<https://arxiv.org/abs/2104.08821> "SimCSE: Simple Contrastive Learning of Sentence Embeddings; EMNLP").
- **E5** - Contrastive pretraining on weakly supervised text pairs; evaluated both without labeled-data fine-tuning and after supervised fine-tuning. [Wang et al. 2022](<https://arxiv.org/abs/2212.03533> "Text Embeddings by Weakly-Supervised Contrastive Pre-training; arXiv"); [code](<https://github.com/microsoft/unilm/tree/master/e5>).
- **Qwen3 Embedding** - Adapts language-model backbones for instruction-aware multilingual text embeddings. [Zhang et al. 2025](<https://arxiv.org/abs/2506.05176> "Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models; arXiv"); [code](<https://github.com/QwenLM/Qwen3-Embedding>).

### Passage retrieval and token scoring

- **Dense Passage Retrieval** - Learns separate question and passage encoders for dense-vector retrieval. Trains from question-passage examples to select contexts for open-domain question answering. [Karpukhin et al. 2020](<https://arxiv.org/abs/2004.04906> "Dense Passage Retrieval for Open-Domain Question Answering; EMNLP").
- **ColBERT** - Independently encodes queries and passages into token vectors; sums each query token's maximum similarity to a passage token. Late interaction retains multiple vectors, whereas late chunking pools each chunk. [Khattab and Zaharia 2020](<https://arxiv.org/abs/2004.12832> "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT; SIGIR"); [code](<https://github.com/stanford-futuredata/ColBERT>).
- **SPLADE** - Learns sparse lexical representations with explicit sparsity regularization and saturated term weights. Supports retrieval with inverted indexes and an effectiveness-sparsity tradeoff. [Formal et al. 2021](<https://arxiv.org/abs/2107.05720> "SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking; SIGIR").
- **BGE-M3** - Jointly supports dense vectors, sparse lexical weights, and token multivectors through self-knowledge distillation. [Chen et al. 2024](<https://arxiv.org/abs/2402.03216> "M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation; Findings of ACL"); [code](<https://github.com/FlagOpen/FlagEmbedding>).

### Document context and chunk pooling

- **Late chunking** - Encodes the document before pooling spans into chunk vectors. Across three encoders and four BEIR datasets, mean nDCG@10 rises from 52.2 to 54.0 with 256-token chunks; individual settings include ties and regressions. [Günther et al. 2024](<https://arxiv.org/abs/2409.04701> "Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models; arXiv").
- **pplx-embed-context** - Combines diffusion-pretrained bidirectional encoders, contrastive learning, and late chunking for document-context-aware passage vectors. [Eslami et al. 2026](<https://arxiv.org/abs/2602.11151> "Diffusion-Pretrained Dense and Contextual Embeddings; ACL Industry").

### Retrieval and generation

- **Retrieval-Augmented Generation** - Combines a pretrained sequence generator with a learned retriever and an external passage index. Compares retrieval shared across a response with retrieval varying by token; retrieved context does not guarantee factuality. [Lewis et al. 2020](<https://arxiv.org/abs/2005.11401> "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks; NeurIPS").
- **RAGTruth** - Annotates unsupported or contradictory spans in nearly 18,000 retrieval-augmented responses. Demonstrates that providing retrieved evidence does not ensure faithful generation; retrieval quality and support for generated claims need separate evaluation. [Niu et al. 2024](<https://aclanthology.org/2024.acl-long.585/> "RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models; ACL").
- **GraphRAG** - Builds entity graphs and community summaries for corpus-wide questions. On two corpora, LLM judges rate its answers more comprehensive than vector RAG's. Gains shrink against summaries made directly from source chunks without a graph. [Edge et al. 2024](<https://arxiv.org/abs/2404.16130v2> "From Local to Global: A Graph RAG Approach to Query-Focused Summarization; arXiv"); [code](<https://github.com/microsoft/graphrag>).
- **HippoRAG** - Uses personalized PageRank over an LLM-extracted graph to retrieve passages across multiple hops. With ColBERTv2, top-five passage recall on 2Wiki rises from 68.2% to 89.1%. On HotpotQA it falls from 79.3% to 77.7%. [Gutiérrez et al. 2024](<https://proceedings.neurips.cc/paper_files/paper/2024/hash/6ddc001d07ca4f319af96a3024f6dbd1-Abstract-Conference.html> "HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models; NeurIPS"); [code](<https://github.com/OSU-NLP-Group/HippoRAG>).
    - **HippoRAG 2** - Adds passage nodes and LLM-filtered query-to-triple matching. With a Llama-3.3-70B reader, 2Wiki answer F1 rises from 61.5 with NV-Embed-v2 retrieval to 71.0. On HotpotQA it changes from 75.3 to 75.5. [Gutiérrez et al. 2025](<https://proceedings.mlr.press/v267/gutierrez25a.html> "From RAG to Memory: Non-Parametric Continual Learning for Large Language Models; ICML"); [code](<https://github.com/OSU-NLP-Group/HippoRAG>).
- **Zep / Graphiti** - Tracks fact validity intervals and superseded edges. On LongMemEval, GPT-4o accuracy rises from 60.2% with full history to 71.2% with retrieved memory, but falls from 94.6% to 80.4% on questions about the assistant's statements. [Rasmussen et al. 2025](<https://arxiv.org/abs/2501.13956v1> "Zep: A Temporal Knowledge Graph Architecture for Agent Memory; arXiv"); [code](<https://github.com/getzep/graphiti>).
- **Direct corpus interaction** - Lets an agent search raw files with terminal tools. On 100 BrowseComp-Plus queries with GPT-5.4 nano, accuracy rises from 45% with embedding retrieval to 73%, while agent cost roughly doubles and gold-document recall falls. [Li et al. 2026](<https://arxiv.org/abs/2605.05242v1> "Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction; NeurIPS").
- **RAG vs. GraphRAG** - With Llama-3.1-8B, local GraphRAG improves HotpotQA F1 (60.04 to 61.66) but trails passage retrieval on Natural Questions (63.01 vs 64.78). Extracted graphs omit about a third of answer entities in both corpora. [Han et al. 2026](<https://arxiv.org/abs/2502.11371v3> "RAG vs. GraphRAG: A Systematic Evaluation and Key Insights; KDD"); [code](<https://github.com/haoyuhan1/RAGvsGraphRAG>).

## Long-context models

- **Transformer-XL** - Combines segment-level recurrence with a positional encoding scheme to extend context across segments. Reuses prior representations while addressing fragmentation from fixed-length training windows. [Dai et al. 2019](<https://arxiv.org/abs/1901.02860> "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context; ACL").
- **Longformer** - Combines local sliding-window attention with task-specific global attention. Reduces attention cost for long documents when window size and the number of global positions remain bounded. [Beltagy et al. 2020](<https://arxiv.org/abs/2004.05150> "Longformer: The Long-Document Transformer; arXiv").
- **Lost in the Middle** - Finds position-dependent failures in long-context question answering and key-value retrieval. In the studied models, relevant information near the middle can be harder to use than information near either end. [Liu et al. 2024](<https://arxiv.org/abs/2307.03172> "Lost in the Middle: How Language Models Use Long Contexts; Transactions of the Association for Computational Linguistics").

## Decoding and reasoning procedures

- **Nucleus sampling** - Samples from a dynamically selected high-probability portion of the next-token distribution. Studies how decoding changes repetition and diversity even when the underlying language model is unchanged. [Holtzman et al. 2020](<https://arxiv.org/abs/1904.09751> "The Curious Case of Neural Text Degeneration; ICLR").
- **PICARD constrained decoding** - Rejects tokens that violate incremental parsing constraints during generation. Improves validity and text-to-SQL performance on Spider and CoSQL; a syntactically valid query can still answer the wrong question. [Scholak et al. 2021](<https://aclanthology.org/2021.emnlp-main.779/> "PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models; EMNLP").
- **Chain-of-thought prompting** - Supplies worked reasoning examples in the prompt before asking for a solution. Studies arithmetic, commonsense, and symbolic tasks; generated explanations are not themselves a correctness check. [Wei et al. 2022](<https://arxiv.org/abs/2201.11903> "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models; NeurIPS").
    - **Self-consistency** - Samples multiple reasoning paths and aggregates their answers instead of choosing one greedy path. Improves the studied reasoning benchmarks at the cost of additional generation. [Wang et al. 2023](<https://arxiv.org/abs/2203.11171> "Self-Consistency Improves Chain of Thought Reasoning in Language Models; ICLR").
- **ReAct** - Interleaves generated reasoning with actions that query external information or interact with an environment. Studies how observations can update task plans instead of relying only on an uninterrupted text completion. [Yao et al. 2023](<https://arxiv.org/abs/2210.03629> "ReAct: Synergizing Reasoning and Acting in Language Models; ICLR").
- **Speculative decoding** - A smaller model drafts tokens that the target model verifies in parallel. A sampling correction preserves the target distribution; speed depends on draft cost and acceptance. See [multi-token pretraining](#contextual-pretraining) for in-model draft heads. [Leviathan et al. 2023](<https://arxiv.org/abs/2211.17192> "Fast Inference from Transformers via Speculative Decoding; ICML").
- **Process supervision** - Trains reward models from human judgments of individual solution steps rather than only final answers; introduces PRM800K for studying process supervision in mathematical reasoning. [Lightman et al. 2023](<https://arxiv.org/abs/2305.20050> "Let&#x27;s Verify Step by Step; arXiv").
- **Reflexion** - Stores verbal diagnoses of failed attempts as memory for retries, without updating weights. On 50 hard HumanEval tasks translated to Rust, GPT-4 with tests and reflection reaches 68% success, versus 60% without reflection and 52% without tests. [Shinn et al. 2023](<https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference.html> "Reflexion: language agents with verbal reinforcement learning; NeurIPS"); [code](<https://github.com/noahshinn/reflexion>).
- **Self-Refine** - One model critiques and revises its output using few-shot prompts, without weight updates. On code optimization, ChatGPT optimizes 27.5% of programs with specific feedback versus 24.8% with revision alone. [Madaan et al. 2023](<https://proceedings.neurips.cc/paper_files/paper/2023/hash/91edff07232fb1b55a505a9e9f6c0ff3-Abstract-Conference.html> "Self-Refine: Iterative Refinement with Self-Feedback; NeurIPS"); [code](<https://github.com/madaan/self-refine>).
- **Self-correction and stopping** - On 100 closed-book HotpotQA questions, GPT-4 scores 49% initially and 43% after two critique-and-revision rounds. Using known answer labels to stop revising instead gives 59%. [Huang et al. 2024](<https://proceedings.iclr.cc/paper_files/paper/2024/hash/8b4add8b0aa8749d80a34ca5d941c355-Abstract-Conference.html> "Large Language Models Cannot Self-Correct Reasoning Yet; ICLR").
- **Reasoning budgets** - At matched token or call budgets, self-consistency often beats tested debate and self-evaluated Reflexion on math and QA. On Game24, Tree-of-Thoughts beats self-consistency with GPT-4, but not GPT-3.5. [Wang et al. 2024](<https://aclanthology.org/2024.emnlp-main.1112/> "Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies; EMNLP").
- **SCoRe** - Trains revisions with two-stage reinforcement learning and correctness rewards. After MBPP training, Gemini 1.0 Pro's HumanEval accuracy rises from 52.4% on its first attempt to 64.6% after one revision, without test feedback at inference. [Kumar et al. 2025](<https://proceedings.iclr.cc/paper_files/paper/2025/hash/871ac99fdc5282d0301934d23945ebaa-Abstract-Conference.html> "Training Language Models to Self-Correct via Reinforcement Learning; ICLR").
- **Recurrent-depth language models** - Repeat shared layers to refine hidden states before producing tokens. See [recurrent depth and adaptive computation](https://mlpapers.org/neural-nets/#recurrent-depth-and-adaptive-computation) for architectures and evidence.

## Optimizing language-model programs

- **DSPy** - Represents LM pipelines as typed modules and bootstraps demonstrations from successful execution traces. A compiler selects module parameters against a supplied metric; the program structure and evaluator remain design choices. [Khattab et al. 2024](<https://proceedings.iclr.cc/paper_files/paper/2024/file/f1cf02ce09757f57c3b93c0db83181e0-Paper-Conference.pdf> "DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines; ICLR").
- **TextGrad** - Propagates LLM-written critiques through a computation graph to revise prompts, code or other variables. Its textual gradients are feedback, not numerical derivatives. [Yuksekgonul et al. 2025](<https://arxiv.org/abs/2406.07496v1> "TextGrad: Automatic &quot;Differentiation&quot; via Text; Nature").

## Text-generation metrics

- **BLEU** - Compares generated translations with references using modified n-gram precision and a brevity penalty. A corpus-level overlap metric; it does not directly establish factual or semantic equivalence. [Papineni et al. 2002](<https://aclanthology.org/P02-1040/> "Bleu: a Method for Automatic Evaluation of Machine Translation; ACL").
    - **SacreBLEU and reproducibility** - Shows that tokenization and normalization choices can make BLEU scores incomparable. Provides standardized reference processing and reporting to make evaluations easier to reproduce. [Post 2018](<https://arxiv.org/abs/1804.08771> "A Call for Clarity in Reporting BLEU Scores; WMT").
- **ROUGE** - Evaluates summaries using overlap with reference summaries, including n-grams and sequence-based measures. Different ROUGE variants capture different forms of overlap. [Lin 2004](<https://aclanthology.org/W04-1013/> "ROUGE: A Package for Automatic Evaluation of Summaries; Text Summarization Branches Out").
- **chrF** - Uses character n-gram precision and recall for translation evaluation. Avoids dependence on a particular word tokenizer and exposes a parameter controlling the recall-precision balance. [Popović 2015](<https://aclanthology.org/W15-3049/> "chrF: character n-gram F-score for automatic MT evaluation; WMT").
- **BERTScore** - Matches candidate and reference tokens using contextual embedding similarity rather than exact lexical equality. Measures representation-level agreement; it is not a factuality verifier. [Zhang et al. 2020](<https://arxiv.org/abs/1904.09675> "BERTScore: Evaluating Text Generation with BERT; ICLR").

## Benchmarks and behavioral evaluation

- **SQuAD** - Tests reading comprehension with questions whose answers are spans of the supplied Wikipedia passage. Measures extractive question answering rather than unrestricted open-domain knowledge. [Rajpurkar et al. 2016](<https://arxiv.org/abs/1606.05250> "SQuAD: 100,000+ Questions for Machine Comprehension of Text; EMNLP").
- **MultiNLI** - Provides entailment, contradiction, and neutral sentence pairs across genres of written and spoken English. Includes evaluation of transfer across genres. [Williams et al. 2018](<https://arxiv.org/abs/1704.05426> "A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference; NAACL").
- **GLUE** - Combines language-understanding tasks with a diagnostic suite for linguistic behavior. Encourages shared representations across tasks rather than optimizing a single dataset. [Wang et al. 2019](<https://arxiv.org/abs/1804.07461> "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding; ICLR").
    - **SuperGLUE** - Introduces a harder collection of language-understanding tasks following progress on GLUE. Supplies a toolkit and shared evaluation; aggregate scores still combine distinct abilities. [Wang et al. 2019](<https://arxiv.org/abs/1905.00537> "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems; NeurIPS").
- **CheckList** - Organizes behavioral tests by linguistic capability and test type. Uses generated cases to reveal failures that aggregate held-out accuracy can conceal. [Ribeiro et al. 2020](<https://arxiv.org/abs/2005.04118> "Beyond Accuracy: Behavioral Testing of NLP models with CheckList; ACL").
- **MMLU** - Tests knowledge and problem solving across 57 academic and professional subjects. Supports per-subject analysis; performance on these questions is not a general measure of real-world competence. [Hendrycks et al. 2021](<https://arxiv.org/abs/2009.03300> "Measuring Massive Multitask Language Understanding; ICLR").
- **BIG-bench** - Collects diverse tasks for examining language-model capabilities and scaling. Studies differences across tasks, model sizes, and evaluation metrics instead of treating aggregate improvement as uniform progress. [Srivastava et al. 2022](<https://arxiv.org/abs/2206.04615> "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models; TMLR").
- **TruthfulQA** - Tests whether generated answers reproduce common misconceptions. Targets a particular source of falsehood; passing the benchmark does not establish truthfulness on arbitrary questions. [Lin et al. 2022](<https://arxiv.org/abs/2109.07958> "TruthfulQA: Measuring How Models Mimic Human Falsehoods; ACL").
- **HELM** - Evaluates language models across shared scenarios and multiple criteria, including accuracy, calibration, robustness, bias, toxicity, and efficiency. Makes coverage gaps and metric tradeoffs explicit. [Liang et al. 2023](<https://arxiv.org/abs/2211.09110> "Holistic Evaluation of Language Models; TMLR").
- **Memory consolidation failures** - Repeatedly distilling ground-truth solutions makes previously solved ARC-AGI tasks fail for GPT-5.4. Raw-episode stores remain competitive with consolidators in WebShop, ALFWorld and AppWorld; compression can discard useful evidence. [Zhang et al. 2026](<https://arxiv.org/abs/2605.12978v2> "Useful Memories Become Faulty When Continuously Updated by LLMs; NeurIPS").
- **Raw-history memory** - On MemoryAgentBench with GPT-4o-mini, tested additions of fact extraction, summary augmentation or graph expansion to dense retrieval with reranking lower accuracy. Raw-RAG trails two structured systems on PersonaMem-v2; these are offline evaluations. [Bao et al. 2026](<https://arxiv.org/abs/2610.30755v1> "Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought; arXiv"); [code](<https://github.com/Litmeb/RAW-RAG>).
- **Thinking and refusal** - In four moderate-size open reasoning models, tested safety defenses often trade less harmful compliance for more benign refusals. Thinking-prefix and defense interventions concern refusal behavior, not every form of safety or frontier-scale reasoning. [Ri et al. 2026](<https://arxiv.org/abs/2606.25013v1> "Do Thinking Tokens Help with Safety?; NeurIPS").

## Software

- **Transformers** - Provides a library of pretrained Transformer architectures and task interfaces. The paper describes reusable implementations for NLP research and applications, rather than a single model release. [Wolf et al. 2020](<https://arxiv.org/abs/1910.03771> "HuggingFace&#x27;s Transformers: State-of-the-art Natural Language Processing; EMNLP").
- **Stanza** - Provides a multilingual NLP pipeline for tokenization, morphology, syntax, and named entities. Packages neural processing components for text analysis across languages. [Qi et al. 2020](<https://arxiv.org/abs/2003.07082> "Stanza: A Python Natural Language Processing Toolkit for Many Human Languages; ACL").

## Related topics

- [Neural Networks](https://mlpapers.org/neural-nets/)
- [Transfer Learning](https://mlpapers.org/transfer-learning/)
- [Generative Models](https://mlpapers.org/generative-models/)
- [Computer Vision](https://mlpapers.org/computer-vision/)
- [Multi-View Learning](https://mlpapers.org/multiview-learning/)
- [Neurosymbolic learning and reasoning](https://mlpapers.org/neurosymbolic-learning/)