# Code for "Efficient Benchmarking is Just Feature Selection and Multiple Regression" (Neurips 2026) Paper: https://arxiv.org/abs/2605.25773 Tutorial: https://github.com/sambowyer/mrmr_eval ### Repo layout: - `benchpred/`: efficient benchmarking methods (code adapted from https://github.com/socialfoundations/benchmark-prediction, whose license is provided in benchpred/LICENSE.txt) - `mrmr.py`: contains the mRMR methods - other files have been adapted to allow for e.g. ridge/kernel ridge regression - `py_irt/models/`: contains several extra continuous IRT model implementations in Pyro - `consolidated_results/`: consolidated results for the experiments (we have aggregated the results from the individual runs for each experiment to avoid ~200GB of disk space for the raw results -- see `consolidate_results.py` for more details) - `main.py`: main script for running the experiments - `data/`: model-dataset scores for binary (openllm + helm), continuous (continuous_cat_scores), and pass@k (pass_at_k_code_v3/open) datasets - `mi_cache/`: cached mutual information values for binary datasets (mainly for Appendix F results) - `viz/`: visualization code for the results (including all the figures in the paper) - `notebooks/`: jupyter notebooks for generating the figures in the paper - `plots/`: pdf figures corresponding to the notebooks - `data_loader.py` and `data_utils.py`: data loading and preprocessing code - `consolidate_results.py`: utilities for cleaning and consolidating the results - `full_tables/`: latex tables for the full results (including those omitted from the paper due to space constraints) - `method_runner.py`: script for running groups of experiments (e.g. all the experiments for a given dataset, method, etc.) - `main.py`: script for calling `method_runner.py` on a given set of experiments - At the moment, we've added in options for mrmr, irt, anchor points, and some other baselines, but this is easily flexible - `sweep_binary.sh`, `sweep_continuous.sh`, `sweep_pass_at_k.sh`: scripts for calling `main.py` on a given set of experiments on the binary, continuous, and pass@k datasets. - `sweep_binary_test.sh`: test script for running the experiments on the binary dataset with a small number of runs and coreset sizes ## Installation Create a uv environment and install the requirements, ```bash uv venv uv sync ``` ## Usage To run the paper's main text experiments, you can use the `sweep_binary.sh`, `sweep_continuous.sh`, `sweep_pass_at_k.sh` scripts (or the `sweep_binary_test.sh` script for a quicker mRMR-only test run). To run extra experiments, you can use the `main.py` script to specify datasets, methods, coreset sizes, number of model trains, and number of runs. (For method specification via names, see `benchpred/__init__.py`, e.g. `k3mrmr6_MIQ_y` for the mRMR method with mutual-information hyperparameter k=6, polynomial kernel order=6 for kernel ridge regression, and MIQ objective; or `Beta_gpirt5` for the continuous Beta IRT model with dimension=5; or `kanchor_points_weighted+` for the anchor points method with d=2 polynomial kernel ridge regression (note that these use `benchpred/backfill.py` and require the original `anchor_points_weighted` method to have already been run).) ## Visualisations The figures in the paper are generated by the `viz/notebooks/` notebooks, each of which corresponds to a certain type of plot in the paper: - `01_pareto_rmse_tau.ipynb`: Plots figure 1 in the paper's main text (`viz/plots/01_pareto_rmse_tau/pareto_rmse_tau_binary_vs_passk_k1_only_5pct_m15.pdf`). - `02_combined_grids.ipynb`: Plots figures for appendix F (13, 14, 15) - `03_combined_grids.ipynb`: Plots figures 2 and 4 in the paper's main text (`viz/plots/03_combined_grids/combined_grid_binary_2row_filtered_with_lasso_metabench.pdf` and `viz/plots/04_combined_grids/combined_grid_continuous_and_passk_k1_two_row.pdf`) and other appendix plots. - `04_difficulty_stability_timing.ipynb`: Plots figure 3 in the paper's main text (`viz/plots/04_difficulty_stability_timing/mmlu_difficulty_plus_stability_timing.pdf`) - `05_true_vs_pred.ipynb`: Plots figure 5 in the paper's main text (`viz/plots/05_true_vs_pred/true_vs_pred_arc_challenge.pdf`) and other appendix plots. - `06_ablations.ipynb`: Plots most of the appendix plots. - `07_winner_maps.ipynb`: Plots figured for appendix B.2 (6, 7). ## Tables The tales of full results can be found in the `full_tables/` directory, and are generated by the `viz/notebooks/03_combined_grids.ipynb` notebook via latex in `viz/plots/03_combined_grids/tables/`. # Citation If you use this repository in your work, please cite the accompanying paper: ```bibtex @inproceedings{bowyer2026efficientbenchmarkingjustfeature, title = {Efficient Benchmarking Is Just Feature Selection and Multiple Regression}, author = {Bowyer, Sam and Locatelli, Acyr and Cao, Kris}, booktitle = {Advances in Neural Information Processing Systems}, volume = {39}, year = {2026}, note = {Evaluations and Datasets Track}, eprint = {2605.25773}, archivePrefix = {arXiv}, primaryClass = {stat.ML}, url = {https://arxiv.org/abs/2605.25773}, } ```