https://www.eleuther.ai/papers Papers and preprints from EleutherAI http://www.rssboard.org/rss-specification python-feedgen en Sun, 23 Aug 2026 01:30:07 +0000 https://arxiv.org/abs/2601.04301 As frontier AI systems are pretrained on web-scale data, test set contamination has become a critical concern for accurately assessing their capabilities. While research has thoroughly investigated the impact of test set contamination on discriminative evaluations like multiple-choice question-answering, comparatively little research has studied the impact of test set contamination on generative evaluations. In this work, we quantitatively assess the effect of test set contamination on generative evaluations through the language model lifecycle. We pretrain language models on mixtures of web data and the MATH benchmark, sweeping model sizes and number of test set replicas contaminating the pretraining corpus; performance improves with contamination and model size. Using scaling laws, we make a surprising discovery: including even a single test set replica enables models to achieve lower loss than the irreducible error of training on the uncontaminated corpus. We then study further training: overtraining with fresh data reduces the effects of contamination, whereas supervised finetuning on the training set can either increase or decrease performance on test data, depending on the amount of pretraining contamination. Finally, at inference, we identify factors that modulate memorization: high sampling temperatures mitigate contamination effects, and longer solutions are exponentially more difficult to memorize than shorter ones, presenting a contrast with discriminative evaluations, where solutions are only a few tokens in length. By characterizing how generation and memorization interact, we highlight a new layer of complexity for trustworthy evaluation of AI systems. https://arxiv.org/abs/2601.04301 Mon, 16 Feb 2026 00:00:00 +0000 https://arxiv.org/abs/2508.06601 Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs https://arxiv.org/abs/2508.06601 Mon, 25 Aug 2025 00:00:00 +0000 https://arxiv.org/abs/2407.06483 Composable Interventions for Language Models https://arxiv.org/abs/2407.06483 Wed, 09 Jul 2025 00:00:00 +0000 https://arxiv.org/abs/2507.06378 Evaluating Morphological Alignment of Tokenizers in 70 Languages https://arxiv.org/abs/2507.06378 Tue, 08 Jul 2025 00:00:00 +0000 https://arxiv.org/abs/2506.23869 Scaling Self-Supervised Representation Learning for Symbolic Piano Performance https://arxiv.org/abs/2506.23869 Mon, 30 Jun 2025 00:00:00 +0000