--- name: braintrust-design-eval-experiment description: >- Design or audit controlled eval experiments for model, prompt, retrieval, tool, guardrail, or agent-architecture changes. Use before data collection to state directional and minimum-effect hypotheses, name independent, dependent, and control variables including the serving environment and tool surface, choose paired designs, set repetitions and allocation, distinguish exploratory from confirmatory comparisons, and pre-specify stopping, exclusion, multiplicity, and analysis rules. Do not use primarily to analyze results already collected. --- # Design the experiment Contract: `references/interaction-contract.md`. Calibration, templates, provenance: `references/experiment-manifest.md`. ## Trigger - A planned comparison: prompt A vs. B, M0 → M1, retrieval on/off, a new guardrail. - "Write a hypothesis." / "Create a pre-analysis plan." / "Name the stopping rule." - A sweep about to run with no plan for how the winner gets confirmed. ## Do 1. Convert the change into **one tradeoff-aware hypothesis**: directional, minimum effect, naming what may not regress, plus population, proxies, and scorers. This **is** the release gate stated in advance. 2. Name independent, dependent, and control variables (`reference.md`). Any control that matters must be a schema field, or it cannot be held constant or sliced by. 3. Pin what hides inside apparently atomic variables: the **serving path** behind a model string, the **tool manifest** behind "the agent." If more than one thing changes per arm, the design cannot attribute the effect to any single factor — isolate first. 4. Choose **pairing** (same items across arms — the default, far more sensitive) and K = 3–5 runs for anything gating. 5. **Verify the treatment is implementable in every arm.** Publish the per-arm implementation table, and exclude arms that cannot receive it from the treatment-effect claim rather than recording them as "no benefit." 6. Pre-specify stopping, exclusions, exploratory vs. confirmatory, multiplicity, and estimators — the elicitation regime dictates the estimator. ## Avoid - Do not choose the analysis after seeing outcomes, or relabel a sweep as confirmatory. - Do not fix a defect mid-comparison. Restart under the corrected config, or finish as designed and record the defect — a mid-run correction leaves no trace in the results table. - Do not run unbounded parallelism; it is the most productive source of fake failures. - Do not analyze results here. ## Check - One hypothesis with direction, minimum effect, and guardrail bounds. - All three variable classes enumerated; every control exists as a field. - Serving environment and tool manifest pinned per arm. - Pairing and K stated; stopping rules fixed; exploratory vs. confirmatory labeled in advance. - Per-arm treatment implementation table drafted; manifest reproducible. ## Risk - Flexible stopping, undocumented arms, and uncontrolled changes make an apparent improvement indistinguishable from selection noise. - A treatment with no uniform implementation quietly becomes several treatments. - Metered ceilings fail in the worst available way — partway through one arm, leaving a partial result that resembles data. - Confounds enumerated on a whiteboard stay there unless they become fields. ## Braintrust **Write H1 into the experiment description** so the pre-registered claim travels with the results — the cheapest integrity control available. Mark the confirmatory run explicitly in its **name and description**. This is the stage that has to get the shared mechanics right, because they are decided here and cannot be repaired afterwards. From `references/platform-mechanics.md`: **§3 pinning**, since pairing must be chosen before running and cannot be applied retroactively; **§4 metadata** for every independent variable, serving path and tool manifest included, and every control named in step 2; **§5 naming**, or the arms never line up; **§6 separate sweep experiments**, losers kept, so the denominator survives to the report; **§7 hygiene** for concurrency and smoke-test deletion. Each of those is cheap now and impossible later. That asymmetry is the whole argument for designing the experiment before running it.