--- name: uai-experiments description: Use when designing or auditing UAI experiments where inference quality is the endpoint, covering calibration and coverage measurement, posterior diagnostics, structure-recovery metrics for causal and graphical models, seeded repeated runs, baselines from both sampling and optimization families, and mapping each claim to its evidence. --- # UAI Experiments Use this while the empirical design is still changeable. At UAI the object under test is usually an inference procedure — a posterior, a graph, an interval, a decision policy — so the experimental question is rarely "is accuracy higher?" and usually "is the uncertainty right, and at what cost?". Reviewers score whether claims are backed up convincingly; design the study so each claim has a designated exhibit. ## Metrics follow the probabilistic object | Claimed object | Primary metrics | Supporting diagnostics | |---|---|---| | Posterior approximation | Wasserstein/KL to gold-standard posterior on tractable cases | R-hat, ESS, trace plots; ELBO with restarts | | Predictive uncertainty | NLL, CRPS, empirical coverage vs nominal | Reliability diagrams; ECE with stated binning | | Conformal / interval methods | Coverage at each α, interval width | Conditional coverage slices, not just marginal | | Causal structure | SHD, SID, edge precision/recall vs ground truth | Performance vs sample size; sensitivity to faithfulness violations | | Treatment effects | Bias/RMSE on ATE/CATE with known ground truth | Overlap diagnostics; propensity calibration | | Decision policies | Regret, expected utility under the stated prior | Robustness under prior misspecification | The recurring UAI failure is a proxy mismatch: claiming better uncertainty while measuring only accuracy, or claiming a better posterior while reporting only downstream prediction. Pick the metric that measures the claimed object directly. ## The synthetic-to-real ladder Because ground-truth posteriors and ground-truth graphs exist only where you construct them, strong UAI papers climb a ladder: 1. **Exact-truth regime**: small models where exact inference or the true SCM is available — the only place "closer to the truth" is literally checkable. 2. **Stress regime**: the same generators with an assumption deliberately broken (unfaithful distributions, heavy-tailed noise, hidden confounding) — this is where your limitations section gets its numbers. 3. **Real-data regime**: demonstrates relevance, evaluated by proxies (held-out NLL, stability, downstream decisions) with the proxy status stated plainly. A paper living only on rung 3 cannot back an inference-quality claim; one living only on rung 1 will be asked why anyone should care. Budget experiments across all three. ## Stochasticity is part of the result - Repeat over seeds and report dispersion (mean ± sd, or paired comparisons when methods share data draws); single-run tables at an uncertainty venue are self-refuting. - Distinguish the two randomness sources — data generation and algorithm internals — and vary them separately when the claim depends on one of them. - Fix the comparison protocol before running: same data splits, same compute budget or an explicit cost axis, same tuning effort for baselines as for the proposed method. ```python # Paired, seeded comparison harness: every method sees identical data draws import numpy as np def run_grid(methods: dict, make_data, seeds=range(10)): rows = [] for s in seeds: data = make_data(rng=np.random.default_rng(s)) # shared draw per seed for name, fit in methods.items(): post = fit(data, seed=s) rows.append({"seed": s, "method": name, "coverage@90": post.coverage(0.90), "nll": post.nll(data.test), "ess_min": post.min_ess()}) return rows # aggregate as mean ± sd; report per-seed table in the appendix ``` ## Baseline selection that survives this reviewer pool - Include the cheap classical baseline (exact inference where feasible, a well-tuned Laplace approximation, the PC algorithm, plain split conformal): UAI reviewers use these as sanity anchors, and their absence reads as fear. - Include the strongest recent neighbor from UAI/AISTATS/NeurIPS, tuned in good faith with its search grid disclosed. - When your method has a compute knob (chains, particles, restarts), plot the quality-versus-cost curve rather than a single operating point chosen post hoc. ## Compute fairness, concretely - Match tuning budgets: if the proposed method saw 50 configurations, baselines see 50, drawn from grids their authors would endorse. - Match convergence criteria: comparing your converged sampler against a baseline's fixed-iteration run measures patience, not methods. - When budgets cannot match (an exact baseline that only scales to n=100), report it at its feasible scale and mark the cell honestly rather than dropping the method. - State who tuned what: "baseline hyperparameters from the original paper" and "tuned by us on the same validation split" are different evidentiary claims. ## Reporting block, standardized Give every experiment family the same reporting block in the appendix, so reviewers can audit uniformly and you can spot your own gaps: ```text EXPERIMENT — backs claim: data: methods: randomness: compute: metrics: result: caveats: ``` The `caveats` line is not decoration. At this venue an experiment section that admits where the method loses reads as calibrated; one that never loses reads as curated. ## Ablations for mechanisms, not rituals Design each ablation to isolate the component your theory says matters: remove the coupling, swap the score function, freeze the calibration step. An ablation grid nobody can interpret is appendix filler; a single ablation matching a theorem's prediction is evidence. ## Output format ```text [Claim → exhibit map] [Ladder coverage] exact-truth / stress / real — which rungs are missing [Uncertainty of results] seeds, dispersion, pairing — adequate? [Baseline audit] classical anchor present? strongest neighbor tuned fairly? [Proxy mismatches] ```