---
name: zoto-update-evals
description: "Diff-aware eval updater. Detects drift between covered targets and generated eval cases for skills and central plugin eval files (commands/agents/hooks), classifies changes as critical or non-critical, and preserves user-authored cases verbatim. Apply-mode regeneration re-invokes the LLM analyser per drifted primitive and dispatches per-framework static stamping plus the single unified LLM regeneration (`regenerateLlm`) (CI defaults to cached analyser payloads unless `--with-analyser`). The `_meta.generated === true` (case) and `// _meta.generated\\: true` (file) contracts are enforced at runtime AND compile time. Supports targeted mode, rediscovery mode, `--no-analyser` / CI cache reuse, and CI `--check` (exit 2 on critical drift)."
---

# Update Evals

Surgical diff engine that detects when covered AI primitives have changed and presents each proposed eval update for user confirmation before writing. Behavioural drift is managed consciously — nothing is rewritten silently. User-authored cases are never mutated and user-authored test files are never overwritten — both contracts are enforced at runtime and at compile time, gated by the canonical helpers in `plugins/zoto-eval-system/engine/_user-case-guards.ts` (`isGeneratedCase`, `isGeneratedFile`).

## Configuration

Reads `.zoto/eval-system/config.yml` and `.zoto/eval-system/manifest.yml`. Both must exist; if the manifest is missing, the skill aborts with `"Run /z-eval-create first."`.

The rediscovery path uses `manifest.discovery_config` for **skillsRoots**, **discoveryTargets**, optional **ignore** globs, and the rest of the stored discovery snapshot — not live `config.yml` discovery fields during **full-catalog** rediscovery (`--check` or rediscovery without `--target`). **`--target`** runs use the live `config.yml` for discovery enumeration. (`eval-analyse.ts` / `eval-stamp.ts` do not re-apply ignore filters when analysing or stamping a target.)

The active static framework is read via `plugins/zoto-eval-system/engine/manifest-snapshot.ts#readManifestSnapshot()` (subtask 02). The `static.framework` field selects the static stamper (pytest / vitest / jest). The LLM side has no strategy axis: regeneration always re-stamps the co-located `<kind>/evals/<name>.json` for the target via `regenerateLlm()`.

## Coverage paths

Drift detection considers eval files at:

- Skills: `…/skills/<name>/evals/evals.json` (under configured `skillsRoots`).
- Commands: `plugins/<plugin>/evals/commands/<name>.json` **and** `.cursor/evals/commands/<name>.json`.
- Agents: `plugins/<plugin>/evals/agents/<name>.json` **and** `.cursor/evals/agents/<name>.json`.
- Hooks: `plugins/<plugin>/evals/hooks/<plugin>.json` **and** `.cursor/evals/hooks/hooks.json` (canonical target id **`hook:cursor-workspace`**).

The **preserve user-authored cases** contract applies uniformly: no `_meta`, or `_meta.generated === false` → never overwritten.

## Modes

| Invocation | Interactive? | Writes? |
|------------|--------------|---------|
| `/z-eval-update --check` | no | no — exits 0 or 2 |
| `/z-eval-update` (no args) | no | no — rediscovery dry run |
| `/z-eval-update --apply` | command-owned `askQuestion` per change + resume | yes |
| `/z-eval-update --target <glob>` | no | no — targeted dry run |
| `/z-eval-update --target <glob> --apply` | command-owned `askQuestion` per change + resume | yes |
| `/z-eval-update --no-analyser [--apply]` | as above | yes/no — bypasses LLM, reuses cached payloads |
| Env `CI=true` without `--with-analyser` | same interaction/write semantics as invocation without `--with-analyser` | skips LLM; reuses payloads like `--no-analyser` |
| `/z-eval-update --with-analyser …` | applies to whichever mode you combine | forces fresh `runAnalyser` despite `CI=true` |

## When to Use

- Immediately after `/z-eval-create` (should no-op).
- Whenever a covered target (skill body, agent body, command, hook definition) is edited.
- In CI — the `--check` variant is designed to gate builds.

## Workflow (`--apply`)

### Step 1: Load manifest, config, and snapshot

Read `.zoto/eval-system/config.yml` and `.zoto/eval-system/manifest.yml`. Record `previous_ref = manifest.git_ref`. Compute `current_ref = git rev-parse HEAD`. Call `readManifestSnapshot()` for the active `static.framework`.

### Step 2: Rediscover

Run discovery using **`manifest.discovery_config`** for full-catalog modes (`--check`, rediscovery dry/apply with no `--target`). For **`--target …`** invocations, use the current `.zoto/eval-system/config.yml` discovery fields so the narrowed scope matches the live scaffold. Produce a fresh list of targets with `content_hash` values and `public_surface` data.

### Step 3: Classify manifest deltas

- `added` — target present now, absent in manifest.
- `removed` — target present in manifest, absent now.
- `modified` — content hash differs.
- `unchanged` — same hash.

Classify each delta against `config.update.criticalChangeRules`:

| Kind | Critical when |
|------|--------------|
| added | `addedTargetWithoutCoverage` is enabled |
| removed | covered by at least one active generated case AND `removedTargetWithActiveCases` enabled |
| modified (frontmatter name/description) | `skillFrontmatterChange` enabled |
| modified (public surface) | `publicSurfaceChange` enabled |
| modified (comment / whitespace only) | non-critical |

### Step 4: Per-primitive analyser refresh

For each `added` or `modified` target:

- **Default**: call `runAnalyser({ target, invalidate: true })` from ``pnpm run eval:analyse`` — refreshes the per-primitive `_meta.primitive_analysis` payload from the LLM.
- **`--no-analyser`** (or **`CI=true` without `--with-analyser`**): load the cached payload from `.zoto/eval-system/cache/analyser/<source_hash>.json` instead. When `process.env.CI === "true"` in that cached-analysis mode, stderr writes `[CI WARNING] skipping fresh primitive analysis in CI; reusing payloads from .zoto/eval-system/cache/analyser/`. **`--with-analyser`** opts into LLM refreshes even on CI hosts. Optional escalation: `update.failOnNoAnalyserInCI: true` aborts with exit code 5 right after that stderr banner.

### Step 5: Dispatch regeneration

The dispatcher reads the manifest snapshot and routes per-primitive payloads to the framework-specific helpers:

| `static.framework` | helper |
|---|---|
| `pytest` | `regeneratePytest()` → `stampPytestPerPrimitive()` |
| `vitest` | `regenerateVitest()` → `stampVitestPerPrimitive()` |
| `jest` | `regenerateJest()` → `stampJestPerPrimitive()` |

The LLM side has no per-strategy dispatch — every drifted target invokes the single unified helper:

| Backend | helper |
|---|---|
| LLM (every target) | `regenerateLlm()` → re-stamps the co-located `<kind>/evals/<name>.json`. The harness reads each case's `requiresInteraction` flag and branches at runtime between the scripted-answer path and the single-prompt path; `runner` cases dispatch to sibling `.test.ts` files. |

**Hard-coded file-level guard** (every `*.json` LLM eval / `*.test.js` / `*.test.py` overwrite): JSON eval helpers check the top-level `_meta.generated` envelope; static test helpers call `isGeneratedFile(path)` from `plugins/zoto-eval-system/engine/_user-case-guards.ts`. A file lacking the generated marker is **user-authored** — the helper skips the write, records the path under `files_preserved`, and emits a `manual_merge_required` note. User-authored `*.test.ts` runner files and scenario files are never overwritten.

**Hard-coded case-level guard** (every `evals.json` row mutation for the central catalogue JSON consumed by static stampers): the helper parses the file with `json-source-map`, walks the cases array, and per row calls `isGeneratedCase(c)`. User-authored cases (no `_meta`, or `_meta.generated === false`) are preserved byte-identically — only their containing file's generated rows are surgically replaced via per-pointer byte splicing.

### Step 6: Write

For accepted changes:
- Per-framework static helpers write per-primitive test files (only when guard passes).
- `regenerateLlm()` writes the co-located `<kind>/evals/<name>.json` (only when bytes differ).
- Refresh `.zoto/eval-system/manifest.yml` with new `git_ref`, `updated_at`, `generated_by: zoto-update-evals`, and the rediscovered `targets[]` (each carries a fresh `content_hash`).
- APPEND the new manifest snapshot to `.zoto/eval-system/manifest.history.yml` (append-only — never compacted).

### Step 7: Summary

Print a structured JSON summary: `{ mode, regenerated_targets, files_written, files_preserved_user_authored, user_cases_preserved, reports[] }`. Each `report` lists framework, `files_written`, `files_preserved` (user-authored skips), `cases_replaced`, `cases_added`, `cases_removed`, `user_cases_preserved`, and any `notes[]`.

## Workflow (`--target <glob>`)

Given `<glob>`:

1. Resolve against discovered targets — match either `target.path` or `target.id` against the glob (dot-aware).
2. Filter `current` and only compute deltas for the resolved subset.
3. Apply-mode decisions arrive via command resume — not via skill `askQuestion`.

## Workflow (`--check`)

Non-interactive. The check:

1. Runs the analyser payload parity gate first (`pnpm run eval:update --check` — `engine/update.ts` invokes `check-analyser-payload-parity.ts` from the resolved plugin in lean mode, or from `.zoto/eval-system/scripts/` when ejected). Drift surfaces under `parity_drift` in the report.
2. Computes deltas as above.
3. Emits a single JSON summary line plus per-critical-delta JSON-line stderr entries.
4. Exits `0` on clean drift + parity, else `config.update.checkExitCodeOnCriticalDrift` (default `2`).

This mode is wired into subtask 12's orchestrator drift hook; keep `--check` non-interactive at all times.

## Surgical-diff invariants

Enforced by the unit suite at `plugins/zoto-eval-system/tests/eval-update-guards.test.ts`:

- A user-authored case in a mixed `evals.json` is byte-identical (canonical JSON) before and after `--apply`.
- A generated `*.json` LLM eval lacking the `_meta.generated` envelope is byte-identical before and after `--apply`.
- A generated `*.json` LLM eval with the envelope is regenerated.
- Cached-analysis CI runs (`CI=true`, without `--with-analyser`, or explicit `--no-analyser`) emit the `[CI WARNING] skipping fresh primitive analysis` stderr line.

## What NOT to Do

- Do not touch user-authored cases. Ever.
- Do not overwrite a generated `*.json` LLM eval lacking the `_meta.generated` envelope, or a user-authored `*.test.ts` runner/scenario file — log `manual_merge_required` instead.
- Do not use the current `config.yml` discovery fields for **full-catalog** rediscovery — always enumerate from `manifest.discovery_config` (**including snapshot `skillsRoots`, `discoveryTargets`, and `ignore`**). Edits to live `config.yml` alone do not change rediscovery unless you re-run `/z-eval-create` or refresh the manifest snapshot. Targeted `--target` flows use live `config.yml`.
- Do not rewrite whole `evals.json` files when a surgical replacement suffices — always go through `surgicallyReplaceGeneratedCases()` for mixed files.
- Do **not** call `askQuestion` from inside the skill — the command owns interactive prompts.
- If a palette-driven or resumed apply flow reaches an undocumented branch, return structured `needs_user_input` (validate against `plugins/zoto-eval-system/templates/schema/needs-user-input.schema.json`) for the command instead of calling `askQuestion` from the skill.