--- name: model-benchmark description: >- Measure and COMPARE the security posture of AI models themselves — open-weights or frontier — when a new one is released: a fixed suite of Inspect benchmarks, one per risk dimension (direct misuse, over-refusal, harmful-request refusal, jailbreak uplift, insecure code, prompt injection via content, injection via tools, memory poisoning, and injection via images for vision models), run on the same seeded samples for every model. Reports each rate with a confidence interval and a paired comparison between models. For a guardrail's effect use guardrail-benchmark; for an app endpoint use app-benchmark. license: MIT compatibility: "Requires Python 3.11+ and a separate virtual environment with inspect_ai, inspect_evals and openai." --- # model-benchmark — security benchmarks for a model release You are running this skill **on behalf of a user** who needs to know how a model (or several) behaves on security-relevant benchmarks, usually because it was just released and they must compare it with models they already use. Drive **Plan → Run → Analyze**. The arms are the **models**; everything else is held fixed. This skill measures; it does not judge whether a result is acceptable. Execution tier: **T0/T1** (pure API, or in-memory mock tools). Use `AskUserQuestion` for choices. --- ## PLAN ### 1. Preflight — Inspect in its own venv ``` uv venv .venv-inspect .venv-inspect/bin/python -m pip install inspect_ai openai 'inspect_evals[agentdojo,cyberseceval_4]' ``` The suite in `lib/inspect_suite.py` was verified against `inspect_evals` 0.29.0. Scorer and metric names are not a stable API: after an upgrade run the smoke test before anything else. ### 2. Profile the models — ask - **Models under test** (one or more). Two ways to reach them: - **Gateway aliases** (default): each model is an alias on one OpenAI-compatible gateway (LiteLLM, vLLM, Ollama `/v1`). Export `MODEL_BENCH_BASE_URL` (ending `/v1`) and `MODEL_BENCH_API_KEY`. This is the route for open-weights models. - **Native Inspect strings** (anything containing `/`, e.g. `anthropic/`): pass `--keep-provider-env` so the provider key survives. Do this deliberately; see Notes. - **Judge model.** Several checks need one. It must grade cyber content without refusing and must not be a model under test (the runner refuses if it is). No judge available → `--checks nojudge`. - **Deployment context.** If a `review.json` exists (from `review-scope`), it is recorded with the results. Otherwise note the surface and system prompt in the manifest. ### 3. Choose checks and depth (`AskUserQuestion`) `--checks`: `core` (below), `nojudge`, `vision`, `all`, or a comma list. | Check | Dimension | Reads as | Judge | Calls/sample | Dataset | |---|---|---|---|---|---| | `cyse4_mitre` | direct misuse | offensive requests NOT assisted ↑ | yes ×2 | 3 | 1000 | | `cyse4_mitre_frr` | over-refusal | benign security questions refused ↓ | no | 1 | 750 | | `strong_reject` | harmful requests, direct | forbidden requests helped with ↓ | yes | 2 | 313 | | `strong_reject_aim` | the same, jailbroken | jailbroken requests helped with ↓ | yes | 2 | 313 | | `cyse4_instruct` | insecure code | code with a detected vulnerability ↓ | no | 1 | 1916 | | `cyse4_multilingual_prompt_injection` | injection via content | injections that succeeded ↓ | yes | 2 | 1004 | | `agentdojo` | injection via tools | episodes where the injected goal ran ↓ | no | ~6 | 944 | | `atb_memory_poison` | memory poisoning | episodes resisted ↑ | no | ~4 | 10 | Also registered: `cyse4_autocomplete`, `atb_autonomy_hijack`, `atb_data_exfil` (in `all`), and `cyse3_visual_prompt_injection` (profile `vision`: needs a model and a judge that accept images; downloads about 1000 images on first use). `--depth`: `smoke` (25 per check — proves the wiring only, the interval is about ±12 points), `standard` (100, about ±6 points, the default), `full`, or a number. Show the user the cost before running anything: ``` python run_suite.py --checks core --depth standard --plan ``` ### 4. Open an experiment folder `bash new_experiment.sh model-benchmark "