---
name: claude-code-evals
description: "MANDATORY load before writing or editing any eval scenario (scenario.yaml), check (checks.yaml), or evals.yaml — wherever the eval suite lives. Evaluate Claude Code configurations (skills, CLAUDE.md, hooks, MCP servers, settings, sub-agents) using the scuttlerun/pincenez/craboodle eval pipeline, and choose between it and the official `claude plugin eval` runner. TRIGGER when: choosing how to eval a plugin; authoring or modifying scenario.yaml or checks.yaml files; designing checks; interpreting eval results; benchmarking configurations; testing CLAUDE.md effectiveness; verifying hooks work; regression testing configs; building or extending an eval suite. DO NOT skip because 'I'll consult check-design.md directly' or 'one quick check edit' — load first, the schema and anti-pattern rules are cheap to surface."
---

# Claude Code Evals

Evaluate Claude Code configurations with behavioral evidence.

## When to Use

- Verifying a skill, CLAUDE.md instruction, hook, or MCP server works as intended
- Regression testing after changes to any config component
- Iterating on scenarios and check quality
- Building confidence before sharing or publishing a configuration
- Deciding whether a configuration change actually improved behavior

## The Eval Mindset

Evaluation replaces intuition with behavioral evidence. Instead of reading your configuration and guessing whether it works, you run Claude against realistic tasks and measure outcomes.

**The key question**: "Would Claude behave the same way without this configuration?" If yes, your configuration may not be adding value — or your checks may not be targeting what the configuration adds.

**When to evaluate:**
- After any behavioral change to a configuration
- Before sharing or publishing configs for others to use
- When debugging inconsistent behavior — evals reveal whether the problem is the config or the task

**When NOT to evaluate:**
- Trivial formatting-only changes (whitespace, comments, reordering)
- Configs with no behavioral expectations (e.g., editor settings)
- During initial exploratory drafting — write first, evaluate when the config stabilizes

**Evaluation is probabilistic.** LLM behavior is non-deterministic. Evals run multiple repetitions and report pass *rates*, not binary pass/fail. A pass rate of 0.8 means the configuration works reliably, not that it works every time. Lint is non-deterministic too: each check is a single un-seeded LLM judgment, so re-running `craboodle lint` may add or drop individual findings (see `pincenez lint --help`). Treat lint findings as advisory — judge each on its merits rather than expecting an identical issue count on every run.

## Choosing a Runner

Claude Code ships its own eval runner, `claude plugin eval` ([docs](https://code.claude.com/docs/en/plugin-evals)). Prefer it whenever it can express the case; use craboodle for what it can't. Use `claude plugin eval` when every condition in the left column holds; use craboodle when any condition in the right column does.

| `claude plugin eval` | craboodle |
|---|---|
| The configuration under test ships in a plugin | The configuration is CLAUDE.md, settings, or hooks outside a plugin — official runs load only the plugin, never user or project config |
| The case is one prompt with no replies (resuming a recorded transcript via `context.history_file` still counts) | The case needs a simulated user who answers `AskUserQuestion` or reacts mid-session — scuttlerun's synthetic user has no official equivalent |
| Free deterministic graders fit (`regex`, `tool_used`, `tool_order`, `file_exists`), optionally with an `llm` rubric | Checks should be linted for anti-patterns before paying for runs (`pincenez lint`) |

The official runner also brings a built-in no-plugin baseline (Δ), MCP server mocks, a cost ceiling, and CI exit codes.

**One exception: skillcraft's own eval steps.** When skillcraft's Create step 4 or its Behavioral Edit Testing gate is what calls for evals, use craboodle even for a skill inside a plugin — that gate snapshots `evals/*/scenario.yaml` suites with `craboodle run`. Every other request to eval a plugin, including one that targets a single skill in it, follows the table above.

**Official case layout**: `evals/<case>/prompt.md` (frontmatter such as `max_turns`, `allowed_tools`; body is the prompt) plus one grader per file in `evals/<case>/graders/*.md` (frontmatter sets `type:`, that type's options, and an optional `weight:`; an `llm` grader's body is its PASS/FAIL rubric). `claude plugin eval init` runs an authoring interview and writes cases; `claude plugin eval init --bare <name>` writes a blank one. Run with `claude plugin eval .` from the plugin root. To assert a skill fired, use a `tool_used` grader with `tool: Skill` and an `input_match` regex on the skill name.

**Both runners use `evals/` by default.** When a plugin already has a craboodle suite there, put official cases in a separate directory and pass `--eval-dir <dir>` to both `init` and the run.

The rest of this skill covers craboodle.

## The Three Tools

```
craboodle (orchestrator)
  ├── scuttlerun (session driver) ── runs Claude, produces transcript
  └── pincenez (grader) ────────── grades transcript, produces verdict
```

- **scuttlerun** runs a headless Claude session with a synthetic user (LLM oracle), producing a full transcript. It handles multi-turn interactions and project scaffolding.
- **pincenez** grades one transcript against a checks file. Each check gets an independent LLM call — binary pass/fail with evidence. No cross-contamination between verdicts. The CLI has two modes: `pincenez checks.yaml output` grades (the unmarked default action, invoked positionally by craboodle), and `pincenez lint checks.yaml` lints checks for quality anti-patterns before grading.
- **craboodle** discovers scenarios, runs each through scuttlerun N times, grades each run with pincenez, averages results, and streams YAML output.

Run `<tool> --help` for CLI flags, YAML schemas, and field references.

## Quick Start

Evaluate whether a CLAUDE.md instruction changes behavior:

**1. Scaffold the eval root:**
```bash
craboodle init my-evals/
```

This creates `my-evals/evals.yaml` — a single, fully-commented config holding both pipeline knobs (top level) and a scuttlerun base config (under `scenarios.base`). The eval root can be a skill directory (next to `SKILL.md`), a plugin root (next to `.claude-plugin/plugin.json`), or any generic directory.

**2. Edit `my-evals/evals.yaml`:**
```yaml
version: "1"
min_pass_rate: 0.8

scenarios:
  base:
    model: claude-sonnet-4-6
    additional_tools:
      - TaskCreate
      - TaskUpdate
    project:
      claude_md: |
        Always write tests before production code. Use test-driven development.
```

> **`additional_tools:` vs `tools:`** — `additional_tools:` **extends** scuttlerun's defaults: entries are appended after the defaults, with duplicates removed on a first-wins basis so original ordering is preserved. `tools:` **replaces** the defaults entirely. Use `additional_tools:` when you want defaults plus extras (e.g. `TaskCreate`, `TaskUpdate`, `Agent`); use `tools:` when you need an exact set (e.g. dropping `Skill` if no skills are under test, or `AskUserQuestion` if no multi-turn interaction). Run `scuttlerun --help` for the default tools list. An exact `tools:` set must still include every tool the config-under-test mandates — never trim below that to bound cost (see `references/config-type-patterns.md` § Skills for the failure mode).

**3. Review lint rules** (before writing checks, learn what lint looks for):
```bash
pincenez lint --help
```

**4. Write `my-evals/evals/tdd-instruction/scenario.yaml`** (per-scenario scuttlerun config):
```yaml
prompt: |
  Write a function called isPrime that checks if a number is prime.
  Save it to prime.js.
```

**5. Write `my-evals/evals/tdd-instruction/checks.yaml`** (pincenez config):
```yaml
checks:
  - test-before-code:
      check: "A test file was written before or alongside the production code"
      note: "Look for a test file created via the Write tool"
  - function-exists:
      check: "The function isPrime exists in prime.js"
  - tests-validate:
      check: "At least one test case validates prime number behavior"
```

**6. Lint checks** (catches anti-patterns before spending money):
```bash
craboodle lint my-evals/
```

**7. Run:**
```bash
craboodle run my-evals/
```

**8. Interpret results** — see `references/results-interpretation.md`.

## What Can Be Evaluated

Any Claude Code configuration that changes agent behavior. Each `scenario.yaml` contains scuttlerun fields that define the configuration under test:

| Config Type | Inject Via | What to Test |
|-------------|-----------|--------------|
| Skills | `project.skills` | Behavioral change with skill loaded |
| CLAUDE.md | `project.claude_md` | Instruction compliance, stickiness under pressure |
| Hooks/Settings | `project.settings` | Hook fires, affects behavior |
| MCP Servers | `sdk.mcp_servers` | Tool discovery and usage |
| Sub-agents | `tools: [Agent]` + `sdk.agents` | Delegation and coordination |
| Workflow-delegating skills | `project.skills` + `Workflow` tool | Skill delegates to a background workflow, waits, reads the artifact, presents without fabricating |
| Model/Effort | `model`, `effort` | Quality at different cost points |
| Plugins | `project.plugins` | Plugin components compose; skills + hooks + agents + MCP load and interact as one unit |
| Bundled Combos | Multiple fields | Combined config stack works as a unit |

See `references/config-type-patterns.md` for scenario examples and design guidance for each type. For a guided bootstrap of plugin-level eval coverage, see plugincraft's `workflows/bootstrap-evals.md`.

## Suite Lifecycle

An eval suite evolves alongside the configuration it tests:

- **Growing the suite** — Add scenarios when you add new behavior, discover edge cases, or find untested dimensions
- **Improving checks** — Iterate on check quality; fix always-passes anti-patterns, tighten vague checks, add missing `note:` fields
- **Running after changes** — Run your suite after config changes to catch regressions. Review any scenarios with degraded pass rates
- **Retiring scenarios** — Remove stale scenarios that no longer test meaningful behavior (e.g., config changed, scenario tests removed functionality)

See `references/scenario-design.md` for scenario structure and suite evolution guidance.

## Reference Files

| File | Purpose |
|------|---------|
| `references/check-design.md` | Check patterns, anti-patterns, quality criteria |
| `references/scenario-design.md` | Scenario structure, suite evolution |
| `references/config-type-patterns.md` | Per-config-type eval guidance with examples |
| `references/results-interpretation.md` | Reading results, decision framework, iteration |
| `references/config-precedence.md` | Full config precedence chain across all three tools |

## Dependencies

- **craboodle** — eval orchestrator (`craboodle --help`)
- **scuttlerun** — session driver (`scuttlerun --help`)
- **pincenez** — checks grader (`pincenez --help`)
- **skillcraft** (sibling skill in this plugin) — for skill-specific eval patterns (pressure testing by skill type, trigger testing, worked examples per skill category)