--- name: decision-model-benchmark description: >- Compare decision models (TypeSafe Jev, OpenAI Decisions, Microsoft-Decision-1, Gemini Decision, Clef, any System One compatible server) used as a security gate, and measure how easily a decision can be flipped. Decision models take a state plus typed questions with a closed answer set (yes/no, choice, score) and return probabilities, no text. One adapter speaks the three wire formats; every arm gets the same seeded labelled cases and adversarial variants (injection, padding, option reorder, misleading option rename, paraphrase, encoding, extra JSON fields). Reports fail-open and fail-closed rates, flip rate per attack axis beside the identical-re-run noise floor, targeted attack success, consistency, calibration, refusal and error rates, latency and input tokens, with 95% intervals and a paired comparison between arms. Use when asked whether a decision model can gate tool calls, screen for prompt injection, route alerts or enforce a content policy. For chat models use model-benchmark. license: MIT compatibility: "Python 3.10+, standard library only (no SDKs, no extra packages). Real runs need HTTPS access to each endpoint under test and its API key in an environment variable. The keyless smoke test needs neither." --- # decision-model-benchmark — how often a decision model is wrong, and how easily it flips You are running this skill **on behalf of a user** who wants to put a decision model in front of something (an agent's tool calls, retrieved documents, an alert queue, a chat assistant) and needs to compare candidates. Drive **Plan → Run → Analyze**. A decision model takes an input (`state`) and typed questions with a closed answer set, and returns probabilities. The policy travels in the request, as each question's instructions and option criteria. That makes two things attacker-reachable: the state, and any option text that is built from data. This skill **measures**. It sets no threshold and gives no verdict; the user's organisation decides what is acceptable. **Execution tier: T0** (pure API): the runner sends HTTPS requests to the endpoints named and runs nothing else. Files: `run_bench.py` (runner), `report.py` (summary), `adapter.py` (wire formats), `variants.py` (corpus loader and variant generator), `metrics.py`, `mock_server.py`, `corpus/` (seed corpus), `references/` (detail), `tests/`. --- ## PLAN ### 1. Ask what is being decided - **Which decision is the model making**, and what does a wrong answer cost in each direction? Fail-open (the gate lets through what it should stop) and fail-closed (it stops what it should let through) are reported separately; the user needs to know which one hurts. - **Who can write to the state, and is any option text built from data?** If option labels or criteria can be influenced from outside, the `option_rename` and `option_order` results apply directly. - **What happens to an unanswered request** in their integration (treated as allow, as block, or sent to a person)? Refusals and errors are reported as their own class for this reason. ### 2. Choose the arms An arm is one model behind one endpoint. A run varies the arm and holds the cases, variants, seed and depth fixed. The first arm is the baseline for paired comparison unless `report.py --baseline` names another. ``` --arm name=