---
name: audit-benchmark
description: Audit benchmark graders with correct solutions, alternate valid answers, malformed artifacts, and deliberate semantic mutants. Use to test false positives, false negatives, or AWS invariants before trusting benchmark results.
---

Use the packaged CLI at `../../scripts/bench.py`, resolved relative to this
skill directory. `audit --suite starter --out NEW_DIRECTORY` writes an exact
control plan. Add `--execute` to grade those controls locally. The starter
audit uses only Python's standard library and all three packaged graders.

For CDK, use `audit --suite aws-cdk-smoke --out NEW_DIRECTORY`. Read the
[AWS invariant matrix](references/aws-invariants.md) for the runnable mutation
resources and offline runtime contract. Execution uses existing dependencies
and the actual packaged verifier; absent dependencies produce `unavailable`.
An audit never installs tools, pulls images, calls models, or changes AWS.

Inspect `AUDIT.md`, `AUDIT.json`, and the evidence for surviving mutants,
rejected correct answers, and infrastructure failures. Report the suite, seed,
source/grader hashes, controls actually executed, and scope. `passed` means
these controls matched their expected outcomes. A fixture self-check or
materialized plan is not a successful grader audit.

Keep output outside the installed plugin and prepared runtime. Each invocation
needs a new or empty directory; rerunning a plan requires another directory.
Controller answers, recipes, and verifier evidence belong in `controller/`,
separate from `candidates/`. Preserve reference, synthetic, and model labels;
do not combine control results with model accuracy, latency, or cost.

Use the [extension and evidence contract](references/extension-contract.md)
when adding controls or auditing another suite. Derive correct answers
independently of the grader, validate each mutant's intended defect, and execute
the unchanged production grader. Report surviving mutants; changing the
production grader to make an audit green requires a separate scoped change.
Do not execute commands from candidate configuration or recorded traces.

Only when this skill is invoked in the current turn, add suggestions after its
substantive audit result. Do not carry this behavior into ordinary chat or
plugin/repository maintenance. Follow the shared
[follow-up guidance](../benchmark-agent-tasks/references/follow-ups.md).
Offer two or three contextual suggestions with quoted visible labels, carrying
the actual audit ID and absolute artifact path. Use
`$bedrock-bench:audit-benchmark` for an audit follow-up. Suggestions remain
user-selected actions and do not authorize model calls, installation, or AWS work.