--- name: sigir-reproducibility description: Use when strengthening the reproducibility of a SIGIR paper or preparing a SIGIR Reproducibility track submission — pinning the retrieval pipeline, seeds and variance for neural rankers, documenting collection versions and index settings, and structuring a reproduction study of published IR results with honest divergence analysis. --- # SIGIR Reproducibility Reproducibility is unusually load-bearing at SIGIR for two reasons. First, the venue runs a dedicated **Reproducibility track** (its own track in 2026, split out of the former combined Resource & Reproducibility track — budget and dates 待核实 on the current page), so reproduction studies are publishable first-class work. Second, the field's own literature documents how often reported IR gains fail to replicate under matched tuning — reviewers of *regular* papers therefore read reproducibility signals as a proxy for whether the gains are real. ## Why IR results drift: the usual suspects | Drift source | Typical symptom | Pin it by | |---|---|---| | Collection version | "MS MARCO" numbers off by points | Exact version/split ids, `ir_datasets` identifiers, checksums | | Index-time analysis | BM25 baseline differs across papers | Scripted index build; record stemmer, stopwords, k1/b | | Doc processing for neural models | Recall@k shifts | Max length, stride, title concatenation recorded as config | | Truncated vs judged pools | Inflated dense-retrieval scores | State pooling; report judged@k alongside nDCG | | Seeds and nondeterminism | ±0.005 nDCG run-to-run | Multiple seeds; report mean ± sd, not the best run | | Eval tool discrepancies | MAP differs at 4th decimal | One canonical scorer (`trec_eval`/`ir_measures`) with flags recorded | | Hyperparameter asymmetry | Baselines lose by under-tuning | Equal tuning budget, documented per system | A reproducibility-strong SIGIR paper closes each row with an artifact, not a promise: the config file *is* the documentation. ## Minimum reporting block for any empirical SIGIR paper Put this in the paper (it fits in ~0.3 page and pre-empts three review objections): - Collections with versions; train/dev/test usage; any filtering. - Index/build settings for every system, baselines included. - Tuning protocol: search space, budget, selection metric, dev split — symmetric across systems. - Seeds: how many, and whether tables show mean, sd, and the significance test. - Compute: hardware, wall-clock for index+train+retrieve, and query latency setup. - Pointer to the repository with run files (see `sigir-artifact-evaluation`). ```yaml # config-as-artifact: one file per reported system, committed to the repo system: ours-dense-v2 collection: msmarco-passage/dev/small # ir_datasets id index: {tokenizer: bert-base-uncased, max_len: 256, stride: 128} train: {seeds: [13, 42, 71], batch: 64, lr: 2e-5, epochs: 3} eval: {tool: ir_measures, metrics: [nDCG@10, RR@10, R@1000], qrels: official} significance: {test: paired-t, correction: bonferroni, alpha: 0.05} ``` ## Writing a Reproducibility track paper A reproduction study is not a re-run; it is an *investigation*. The track rewards: 1. **Target selection with stakes** — results the community builds on (a widely cited ranker, a standard baseline configuration, a claimed efficiency win). 2. **Faithful reimplementation first**: reproduce with the original artifacts where they exist; document every forced deviation and why. 3. **Divergence analysis as the contribution**: when numbers differ, isolate the cause (collection version? tuning? eval tool?) with controlled toggles — the drift table above is your experimental design. 4. **Generalization probes**: does the original conclusion survive new collections, matched tuning, or current baselines? "Holds, but the margin halves under equal tuning" is a publishable, community-serving finding. 5. **Respectful register**: the genre convention is scientific, not gotcha — criticize configurations, not authors, and give original authors' artifacts credit where due. Anti-patterns the track's reviewers flag: reproducing only the headline number while skipping ablations; declaring "failure to reproduce" without exhausting configuration space; and shipping a reproduction whose *own* pipeline is unpinned (the irony reject). ## Decay planning: the three-year audience A SIGIR paper's reproducibility has two audiences with different failure modes: the reviewer this spring (needs the 10-minute runnable path) and the researcher in three years (meets link rot, dataset takedowns, deprecated toolkit APIs, and vanished model checkpoints). Plan for the second audience explicitly: - Cite collections by **stable identifiers** (`ir_datasets` ids, TREC track names, dataset DOIs), never by lab-server URLs. - Freeze an **archival copy** of your own artifacts (Zenodo/institutional DOI) at camera-ready; GitHub is a working mirror, not an archive. - Record toolkit **versions in the paper text**, not only in the lockfile — the paper outlives the repository more often than authors expect. - If a resource you depend on has restrictive terms (query logs, commercial APIs), state what a future reproducer *can* do without it: which tables survive, which die. - Leave the qrels/run checksums in the repo README; three years later they are the only way to prove a re-download matches the evaluated state. ## Reproducibility as review defense for regular papers - Close comparisons without variance are the most common SIGIR experimental criticism; three seeds with mean ± sd is cheap insurance for neural systems. - An honest "we could not reproduce baseline X's published number; we report our best faithful configuration (details in repo)" reads as strength, not weakness — the community knows the drift problem intimately. - Never copy baseline numbers across collections or eval setups from other papers' tables without saying so; mixed-provenance tables are a known reject trigger. ## Quick self-audit before submission - [ ] Every number in every table regenerates from a shipped run file plus one documented command. - [ ] Every system row has a config file; diffs between systems are visible as config diffs, not prose. - [ ] The seeds behind each neural row are enumerable, and the shipped run is identified (which seed, or the per-topic mean). - [ ] A colleague outside the project reproduced Table 1 from the repo README without asking questions (the strongest cheap test available). - [ ] The paper's reporting block and the repo's configs agree — reviewers diff them when suspicious. ## Output format ```text [Mode] hardening a regular paper / Reproducibility track study [Drift audit] rows closed with artifacts: /7 (collection/index/processing/pool/seed/tool/tuning) [Reporting block] present in paper y/n; missing items [For repro studies] target + stakes / faithfulness log / divergence causes isolated [Variance] seeds , mean±sd shown y/n, test named y/n [Biggest residual risk] ```