---
name: eval-gates
description: Use when the question is whether a change made an AI system or test suite worse, or whether an eval result can be trusted — "did my prompt change make it worse", "my evals are flaky", "can I bump the threshold to get CI green", "the score came back NaN", "set up evals before we ship". ALWAYS use when a gate is red and someone wants to raise/lower a threshold, widen a noise band, downgrade a metric, refresh the baseline, or otherwise make CI pass, and when a check "passed" suspiciously. Covers fail-closed release gates, thresholds, baselines, noise bands, metric registries, BLOCK fixtures, and the failure modes naive gates miss.
---

# Fail-closed eval gates

Use this skill whenever the user is designing, reviewing, or debugging an
eval release gate — with this plugin's tooling or anyone else's. The doctrine
transfers; the file names don't matter.

## The core rule

**A gate that has never been seen to BLOCK is decoration.** Every hard gate
needs a demonstrated BLOCK on a fixture before anyone trusts it. Every
number needs a reproducible producer. Anything else is eval theater.

## Registry design

One registry file is the single source of truth: name, direction
(`higher_better`/`lower_better`), threshold, noise_band, blocking tier
(`hard`/`soft`/`monitor_only`). When helping a user pick metrics:

- Hard tier is for invariants: schema compliance, safety/injection pass
  rate, PII leakage, the one quality metric the product dies without.
  Everything else starts soft or monitor_only and EARNS promotion with a
  validated measurement (an LLM judge becomes a gate only after ≥85%
  agreement with human labels — before that it is monitor_only, always).
- `lower_better` metrics (latency, cost): the breach is ABOVE the threshold.
  This is the most common gate bug and the most common fixture bug.
- Thresholds change only in reviewed commits with written justification.
  "Never edit a threshold to turn a red gate green" — if asked to, refuse
  and name the alternative: fix the regression, or justify the new threshold
  on its own evidence in its own PR.

## The failure modes naive gates miss (check ALL of these in any gate review)

1. **NaN passes silently.** `NaN < threshold` and `NaN > threshold` are both
   False, so a broken scorer PASSes both breach and regression checks. Any
   non-finite or non-numeric score must BLOCK — even on monitor_only rows,
   because the instrument itself is broken.
2. **Missing metric = silent skip.** A metric absent from the candidate must
   BLOCK, or deleting a metric becomes the easiest way to pass.
3. **Stale/demo baseline.** Candidate and baseline must prove they measured
   the same thing: carry dataset + registry hashes in a manifest and BLOCK
   on mismatch before judging any metric.
4. **Registry typos.** `lower_beter` silently disables a gate if the loader
   defaults unknown values. Schema-lint the registry (enums, required keys,
   no unknown keys) and refuse to run on any error.
5. **Zero-width bands on noisy metrics.** Regression bands must be
   max(declared band, measured 2-sigma from N≥3 baseline runs). A guessed
   band produces either false blocks or false confidence — both discredit
   the gate.
6. **Baseline refreshed while red.** Rebuilding the baseline to make a
   failing candidate pass is threshold-editing in disguise.

## BLOCK fixtures (prove the gate fires)

For every hard row, one fixture candidate that breaches ONLY that row —
every other metric healthy — plus one fully healthy fixture:

- assert the breaching fixture exits 1 and blocks EXACTLY that metric with
  zero WARNs elsewhere (a fixture that trips two rows proves nothing about
  either);
- assert the healthy fixture exits 0 with zero WARNs;
- add a coverage test: set(hard rows in registry) == set(block fixtures), so
  a new hard row forces a new fixture;
- exercise the CLI exit codes end-to-end, not just the library function —
  CI consumes the exit code.

## Honest limits (state them, never paper over them)

- `registry_hash` is VERIFIED: the gate recomputes sha256 of the registry
  file and blocks on mismatch. `dataset_hash` is an ATTESTATION: the gate
  compares it between candidate and baseline but cannot know your dataset —
  drift detection, not forgery-proof. Say so in any claims you make.
- The gate judges the scores it is given; it cannot detect an eval that
  measures the wrong thing. Judge validation and dataset review are separate
  work the registry only points at.

## Wiring

The gate runs in CI on every push including changes to the gate's own code
and tests — a gate whose own changes don't trigger it will rot. Order:
registry lint → unit tests → produce candidate → gate. All release-blocking.