--- name: coding-guidance-benchmark description: >- A/B test of guidance placed in front of a coding agent: does a skill, rules file, hook, MCP server configuration, or any mix of them change the security of the code the agent writes? Each arm is an opaque folder copied into the agent's workspace before it starts; the baseline has nothing installed. A real coding-agent harness (Claude Code, Codex CLI, Gemini CLI or OpenCode, run through Inspect AI's inspect-swe) builds the same seeded BaxBench tasks under every arm, and BaxBench's functional tests and exploits score the result. Reports the correct-and-secure rate per arm with intervals, the paired difference from baseline, functional and secure rates separately, failed builds, the tokens and time the guidance cost, and whether the agent touched the guidance at all. Use it when someone asks whether a piece of coding guidance works. It ships no guidance of its own. For a guardrail or content filter use guardrail-benchmark. license: MIT compatibility: "Requires Docker, git, and two Python environments: one with inspect_ai 0.3.289 and inspect-swe 0.3.0 plus the SDK of the model provider (runs the agent), and Python 3.12 with pipenv (runs BaxBench's scoring). The keyless smoke test needs only Python 3.11+." --- # coding-guidance-benchmark — does this guidance make the agent's code more secure? You are running this skill **on behalf of a user** who has some guidance for a coding agent and wants to know whether it changes the security of the code the agent writes. Drive **Plan → Run → Analyze**. Use `AskUserQuestion` for choices. The arms are the one thing that varies. Everything else is held fixed: the harness and its version, the model, the tasks, the prompt, the scorer. This skill measures; it does not say whether a result is good enough, and it ships no guidance, rules, hooks or templates. The only arm in this folder is a placebo. Execution tier: **T1**. The agent works in a Docker container with no network. See Notes for what scoring does. | Held fixed | Varied | |---|---| | Harness and pinned version, model, time limit | The arm: a folder copied in before the agent starts | | The seeded draw of BaxBench tasks, the same for every arm | `baseline` = nothing installed (always run) | | BaxBench's prompt with **no** safety prompt | | | BaxBench's functional tests and exploits | | ``` SKILL.md run_benchmark.py report.py new_experiment.sh arms/placebo/ the only arm shipped: irrelevant content fixtures/ fake tasks and fixed code for the keyless checks (not BaxBench content) lib/ arms, tasks, baxbench (+ bridge), harnesses, transcript, stats, evalkit references/arm-format.md how to lay out an arm folder references/baxbench-integration.md what is taken from BaxBench and how results are read ``` --- ## PLAN ### 1. Preflight - `docker info` works. - **Agent environment** (runs Inspect and the harness): ``` uv venv .venv-agent uv pip install --python .venv-agent/bin/python inspect_ai==0.3.289 inspect-swe==0.3.0 anthropic ``` Replace `anthropic` with the SDK of the provider that serves the model. Inspect names the missing package if one is needed. The provider's API key stays on this machine; it is not passed into the agent's container. - **BaxBench environment** (runs scoring): Python 3.12 and `pipenv`. The runner clones BaxBench at its pinned commit on first use; then run `pipenv sync` once inside the clone (`--baxbench-dir`, default `~/.cache/coding-guidance-benchmark/baxbench-`). - The harness binary is downloaded on this machine and copied into each container, so this machine needs network access; the containers do not. ### 2. Ask the user - **Arms.** A directory whose subfolders are the arms, one folder per piece of guidance. The user supplies them; read `references/arm-format.md` with them and check each folder is laid out the way their harness will find it. Add `arms/placebo` beside their arms: it shows how far two arms differ when the arm cannot matter. - **Harness and version**, e.g. `claude_code@2.1.311`. A version number is required so the result can be reproduced. `codex_cli`, `gemini_cli` and `opencode` are accepted too. - **Model**, as an Inspect model string (`anthropic/`, `openai/`, ...). - **Frameworks** (optional): restrict the draw, e.g. `--frameworks Python-Flask,JavaScript-express`, when the guidance is about one stack. - **Depth and repeats** (next step). ### 3. Depth, repeats and cost (`AskUserQuestion`) `--depth`: `smoke` (25 tasks; proves the wiring only: a per-arm rate near 50% has an interval of about ±19 points), `standard` (100 tasks, about ±10 points), `full` (all 392, or all tasks of the chosen frameworks), or a number. `--repeats N` runs each task N times per arm; agents are not deterministic, so use at least 2 when the user wants to know how stable a result is. Show the cost first: ``` python run_benchmark.py --arms --harness claude_code@ \ --model --depth standard --repeats 2 --plan ``` The plan lists agent runs and app builds per arm and for the whole run. It does **not** estimate tokens: token cost depends on the harness, the model and how long the agent works on a task. Run `smoke` first and read the Cost table before committing to a deeper run. A deeper run with the same `--seed` contains every task of a shallower one. ### 4. Open an experiment folder ``` bash new_experiment.sh coding-guidance-benchmark "