--- name: app-benchmark description: >- Fast static security benchmark for an LLM app: runs a bundled, license-clean adversarial corpus (prompt injection, harmful, data leakage, over-refusal + M2S multi-turn) through promptfoo eval and reports F1 / recall / FPR / ASR by technique. Use to benchmark defenses, compare models/versions, or A/B/C a param-based guardrail. For live multi-turn attacks use app-redteam; for a guardrail API alone use control-isolate. license: MIT compatibility: "Requires python>=3.11, node>=18." --- # app-benchmark — static security benchmark (promptfoo) You are running this skill **on behalf of a user** who wants to benchmark their LLM application against a standard adversarial corpus. Drive the whole journey: **Plan → Run → Analyze**. Use `AskUserQuestion` for choices, gather endpoint details conversationally, write the config files yourself, then run and analyze. Narrate briefly; stop and ask on anything ambiguous rather than guessing. --- ## PLAN ### 1. Preflight From this skill's directory, run `bash install_dependencies.sh` (installs promptfoo locally, sets air-gap env). Confirm `python` and `node` exist. ### 2. Profile the target — ask, then write the config Gather these from the user (one or two at a time; they may paste a working `curl`): - **Endpoint URL** and **HTTP method** (usually POST). - **Auth** — header name + scheme (e.g. `Authorization: Bearer …`, `x-api-key: …`). - **Request body shape** — how the user prompt is embedded. Common shapes: `{messages:[{role:user,content:"…"}]}`, `{input:"…"}`, `{prompt:"…"}`, `{query:"…"}`. Get a real example. - **Response shape** — where the reply text lives (the shared transform already handles OpenAI/Anthropic/`response`/`output`/`text`/`messages[]`; only note it if it's unusual). - Whether the app exposes a **param to toggle a guardrail** (for the optional A/B/C). Then **do the setup for them**: 1. `cp env.example .env` and fill `TARGET_URL`, `TARGET_API_KEY`, and the judge vars (next step). `.env` is gitignored. 2. **Edit `promptfooconfig.yaml`** `providers[0].config` so `body` and `headers` match the app's real schema (replace the default `messages` body if needed; keep `{{prompt}}` where the user input goes). Leave `transformResponse` as-is. 3. Validate: `npx promptfoo validate -c promptfooconfig.yaml`. ### 3. Choose the judge Every case is graded by an LLM judge (promptfoo `llm-rubric`) via an OpenAI-compatible provider. Ask for the judge gateway (`JUDGE_BASE_URL` = gateway base `…/v1`, `JUDGE_API_KEY`, `JUDGE_MODEL`). **REQUIRED gate — do not skip:** before running, compare the judge to the target. If `JUDGE_MODEL` is the same model as the target (or `JUDGE_BASE_URL` points at the same model as the target), STOP and tell the user the eval would be self-graded and invalid; have them pick a different judge model before proceeding. A small capable model from a different family (e.g. gpt-4o-mini) is a good default. ### 4. Choose how much / which of the corpus to run (`AskUserQuestion`) The **full corpus (~2900 cases) is bundled** in `corpus/` — no build step. It's tagged by `category` (cyber / prompt_injection / data_leakage / content_safety). Pick: - **Depth:** `smoke` (`--filter-sample 30` — proves the wiring; the interval on recall is about ±15 points), `standard` (`--filter-sample 150`), or `full` (omit the flag). Each case is one call to the target and one to the judge: tell the user the totals before running. - **Scope:** the whole corpus, or a **category slice** for a focused audience — e.g. a cyber team: `--filter-metadata category=cyber` (also `prompt_injection`, `data_leakage`). - Combine them (`--filter-metadata category=cyber --filter-sample 60`). Ask the user which axes they care about; cyber/security teams usually want cyber + injection + data_leakage, app teams the full spread. ### 5. (Optional) A/B/C a control If the app toggles a guardrail by a body param, duplicate `providers[0]` with a second `label` and set that param in its `body` — promptfoo runs every case against both and the report compares them per case. ### 6. Open an experiment folder (captures this run) If a `review.json` exists (from `review-scope`), the deployment context is recorded with the results; otherwise note the surface and system prompt in the manifest. Propose a short legible label (target + scope, e.g. `acme-prod-cyber-smoke`), confirm it with the user, then: ``` bash new_experiment.sh app-benchmark "