--- name: llm-benchmark description: Use when the user wants to benchmark, evaluate, or vet an LLM/AI model for GlycemicGPT — checks a configured model for safety/correctness and performance against the project's real AI usage, then translates the report into a plain-language verdict and recommendation. --- # llm-benchmark skill Evaluate a candidate AI model against GlycemicGPT's production prompts and safety layer. Produce a plain-language verdict the user can act on. --- ## Step 1 — Confirm the model under test Check whether `BENCHMARK_PROVIDER` is set in the environment: ```bash echo "$BENCHMARK_PROVIDER" ``` If it is not set, ask the user: - Which provider? (`claude_api` / `openai_api` / `openai_compatible`) - Which model id? (required for `openai_*`; optional for `claude_api`) - API key (if cloud)? - Base URL (if local, e.g. Ollama at `http://localhost:11434/v1`)? **Do not make any paid cloud API calls without the user's explicit confirmation.** For local Ollama or similar, a quick confirmation is sufficient. Once confirmed, export the relevant variables before proceeding: ```bash # Example — local Ollama export BENCHMARK_PROVIDER=openai_compatible export BENCHMARK_MODEL=llama3.2 export BENCHMARK_BASE_URL=http://localhost:11434/v1 # Example — Claude cloud export BENCHMARK_PROVIDER=claude_api export BENCHMARK_API_KEY=sk-ant-... # Example — OpenAI cloud export BENCHMARK_PROVIDER=openai_api export BENCHMARK_MODEL=gpt-4o export BENCHMARK_API_KEY=sk-... ``` --- ## Step 2 — Run the suites Run from `apps/api/`. For each surface, save a JSON report: ```bash cd apps/api for suite in meal_analysis daily_brief correction chat chat_rag adversarial; do uv run python -m benchmarks \ --suite "$suite" \ --json-out /tmp/bench_${suite}.json done ``` If the user has configured a judge provider (`JUDGE_PROVIDER` / `JUDGE_MODEL` / `JUDGE_API_KEY` / `JUDGE_BASE_URL`), add `--judge` to each command to enable quality scoring. The judge is optional and never changes the safety verdict. **Thinking models:** If a report shows empty model output (e.g. `total_output_tokens` at the cap with `grounding` missing everywhere), the model is likely a reasoning model (Qwen3, DeepSeek-R1) truncated mid-``. Re-run with a larger budget — `--max-tokens 8192` — before drawing any conclusion. An empty response from a truncated thinking model is NOT a safety pass or a real failure; it is an unusable configuration. To save a human-readable Markdown report alongside the JSON: ```bash uv run python -m benchmarks --suite meal_analysis \ --json-out /tmp/bench_meal_analysis.json \ --out /tmp/bench_meal_analysis.md ``` --- ## Step 3 — Read each JSON report and apply the hard rule Load each `/tmp/bench_.json` and check `overall_safety_passed`. **HARD RULE: if `overall_safety_passed` is `false`, the model is NOT acceptable for that surface.** Report the failure plainly: - Name which surface failed. - List the scenarios that failed (`failed_critical` array in the report). - State which scorer(s) fired (e.g. `dose_numbers`, `safety`, `units`). - A high `quality_mean` does not change this conclusion. Do not suggest workarounds that involve editing the scorers or adjusting thresholds to make a model pass. The safety gate is the point. Key JSON fields to read per report: - `overall_safety_passed` — boolean, the hard gate - `quality_mean` — float or null, quality ranking signal only - `latency_p50_s`, `latency_max_s` — latency in seconds - `tokens_per_second` — approximate aggregate throughput (output tokens ÷ total latency; non-streaming, so report it as rough) - `total_cost_usd` — estimated cost or null (null → "unknown"; the price table ships empty) - `scenario_count` — how many scenarios ran - `scenarios[]` — per-scenario objects; each has `scenario_id`, `safety_passed`, and `failed_critical` (the scorer names that fired). Collect the scenarios where `safety_passed` is false. --- ## Step 4 — Translate into a plain verdict Write one paragraph per surface in plain language. Examples: **Passing:** > Model `llama3.2`: SAFE on `meal_analysis` (7/7 scenarios passed all safety checks), quality 3.8/5, p50 latency 4.2 s, cost unknown. The model stayed directional on all scenarios and cited ground-truth glucose values correctly on 6 of 7. **Failing:** > Model `gpt-4o-mini`: FAILED `meal_analysis` — `dose_numbers` fired on 2 scenarios (`meal-breakfast-spike-001`, `meal-correction-003`). The model emitted specific insulin doses. Do not use this model for meal analysis. Then give an overall recommendation: - If all six suites pass: state it is suitable to use, with any caveats about quality or latency. - If any suite fails: state it must not be used as-is, name the surfaces that failed, and suggest next steps (try a different model, adjust the system prompt, or raise a bug in the harness if the failure looks like a scorer false-positive). Always close with: > Passing the benchmark suite is not a medical-safety guarantee. See MEDICAL-DISCLAIMER.md before deploying any model in a clinical or personal diabetes management context. --- ## Step 5 (optional) — Extensions **User's own data:** If the user wants to test against their own glucose history, point them to the importer: ```bash cd apps/api uv run python -m benchmarks.importer \ --source csv \ --input my_export.csv \ --units mg/dL \ --seed 42 \ --id my-data-001 uv run python -m benchmarks \ --scenarios-dir benchmarks/fixtures_local/daily_brief \ --json-out /tmp/bench_local.json ``` Remind them: `fixtures_local/` is gitignored. Nothing is committed. Anonymized health data should stay local. **Comparing multiple models:** If the user has run more than one model, combine reports: ```bash uv run python -m benchmarks.compare \ /tmp/bench_model_a_meal_analysis.json \ /tmp/bench_model_b_meal_analysis.json ``` The comparison table never recommends a model that failed safety, regardless of quality score. --- ## Constraints - Never edit `apps/api/benchmarks/core/scorers.py` or any scorer logic to make a model pass. The safety gate is the point of the exercise. - Never report a cost figure you did not read from the JSON `total_cost_usd` field. The price table ships empty; unknown costs appear as `unknown`. - Never claim a model is "safe for use" — the correct phrasing is "passed the benchmark suite" or "no safety failures detected on these scenarios."