--- name: benchmark-runner description: Auto-discover all skills with evals in RConsortium/pharma-skills, benchmark each with vs. without skill using matched isolated sessions, and post scored results to the linked GitHub issue. Use whenever someone says "run benchmarks", "compare skill performance", "eval the skills", or wants to measure whether a skill improves output quality. --- # Skill Benchmark Runner Benchmark every evaluation case in the `_automation/evals/` directory of the `RConsortium/pharma-skills` repository. Each routine invocation is **one of two short phases (~20 min each)**. The routine inspects GitHub issue comments on startup to decide which phase to execute — no configuration needed, no commits required, no repo write access required from the human user. Repository: `RConsortium/pharma-skills` (https://github.com/RConsortium/pharma-skills) --- ## Model Selection — Series Label and Sub-Agent Model `{CURRENT_MODEL_NAME}` throughout this skill is the **benchmark series label**, fixed to the literal string `Claude Routine`. Use it verbatim everywhere it appears: `get_next_eval.py --model`, `record_run_result.py --model`, the marker JSON (`"model"`), and the report metadata. The scripts normalise this label (lowercase, strip spaces/punctuation) for deduplication, so all runs group into one consistent series. Sub-agent model: - Agent A and Agent B are launched **without an explicit `--model` flag** (`claude -p ...`), so each sub-agent inherits this session's default model — the same model orchestrating the routine. This keeps the measurement clean (bare model ± skill, identical model on both sides) without ever writing a concrete model ID into a repository artifact. Never hardcode a concrete model ID anywhere in this skill or its outputs. - The prompt files constructed in Step 2 / Step 6 and the `CLAUDE_CODE_MAX_OUTPUT_TOKENS` setting are the **only** information passed from the parent session to a sub-agent. Do not forward any other parent context: no conversation history, no additional environment variables, no eval assertions, no scoring prompt, no blinded map. This prevents the orchestrator's context from leaking into either candidate. --- ## Routine Setup (one-time) Create a single routine at [claude.ai/code/routines](https://claude.ai/code/routines): | Field | Value | |---|---| | **Prompt** | `Read _automation/benchmark-runner/SKILL.md and execute.` | | **Repository** | `RConsortium/pharma-skills` | | **Schedule** | `0 1,6 * * *` (1 AM and 6 AM UTC — 5 h gap, matches rolling usage window) | That is all. The skill determines its own phase on every invocation. --- ## GitHub Access — Use Whichever Method Works Throughout this skill you will read issue comments, post issue comments, and create release assets. **Use whatever method is available in your environment** — pick the one that works without prompting: | Method | Best when | Notes | |---|---|---| | `mcp__github__*` MCP tools | Running inside Claude Code with the GitHub MCP server | No token required; preferred when available | | `gh` CLI (`gh issue view`, `gh release upload`, etc.) | Running locally with `gh` authenticated | Concise, supports all operations | | REST API via `curl` | Anywhere with `GH_TOKEN` / `GITHUB_TOKEN` set | Universal fallback; use for release-asset upload (no MCP equivalent) | | Provider-specific GitHub tools (Codex, Gemini, etc.) | Running under another agent CLI | Use whatever the host provides | Reason about which method to use; do not enforce a rigid order. If one fails, try another. Always confirm the operation succeeded (e.g., the comment URL came back, the asset was uploaded) before continuing. For release-asset upload there is currently no MCP tool — use `gh release upload` or `curl` POST to the upload URL. --- ## Phase Detection — Run Before Any Other Step Scan all benchmark eval issues to find any that are waiting for Phase 2 (Agent B + scoring) **for the model you are running**: 1. List all eval files: `ls _automation/evals/*.json` — extract each `id` field (e.g. `github-issue-27` → issue **#27**). 2. For each issue number, fetch comments using whichever GitHub access method is available (see above). Scan each comment body for a ` ``` --- ## Phase 1 — Agent A Run (With Skill) Runs when no Phase 2 candidate is found. Executes Agent A, archives its output, and posts a partial comment that holds state for Phase 2. ### Step 0 — R Environment Pre-flight **Always run first. Idempotent — safe to re-run.** ```bash bash _automation/benchmark-runner/scripts/setup_r_env.sh ``` Exits non-zero on failure — stop and report the error. Do not proceed. > R packages installed: `jsonlite`, `digest`, `gsDesign`, `gsDesign2`, `lrstat`, `graphicalMCP`, `eventPred`, `ggplot2` ### Step 1 — Discover Next Eval ```bash python3 _automation/benchmark-runner/scripts/get_next_eval.py --model {CURRENT_MODEL_NAME} ``` - `STATUS: UP_TO_DATE` → all evals complete for this model+SHA. Exit. - JSON output → parse to a temp file; extract `_skill_name`, `_skill_sha`, `_skill_content`, `_bundled_resources`, `_prompt_a`, `_blinded_scoring_map`, and the issue number from `id`. Optional flags: ```bash --runner-id {YOUR_NAME} # stable per-person ordering --priority-issue github-issue-{N} # force a specific eval ``` ### Step 2 — Run Agent A (With Skill) Create the working directory: ```bash mkdir -p /tmp/benchmark_{id}/agent_A/output_A ``` **Stage bundled resource files to disk** (progressive disclosure — files read on demand, not embedded in the prompt): ```python import os, json agent_a_dir = "/tmp/benchmark_{id}/agent_A" for rel_path, content in eval_case["_bundled_resources"].items(): if rel_path == "SKILL.md": continue dest = os.path.join(agent_a_dir, rel_path) os.makedirs(os.path.dirname(dest), exist_ok=True) with open(dest, "w", encoding="utf-8") as f: f.write(content) ``` **Write `prompt_A.txt`** — `_skill_content` (SKILL.md) followed by `_prompt_a` only. No bundled resource content in the prompt: ```python prompt_a = eval_case["_skill_content"] + "\n\n" + eval_case["_prompt_a"] with open(os.path.join(agent_a_dir, "prompt_A.txt"), "w", encoding="utf-8") as f: f.write(prompt_a) ``` **Launch Agent A:** ```bash export CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000 cd /tmp/benchmark_{id}/agent_A && \ cat prompt_A.txt | claude -p \ --allowedTools "Bash,Read,Write,Edit,Glob" \ --output-format json > agent_A_run.json 2>&1 ``` > **Note:** `export` is required — a prefix (`VAR=val cat ... | claude`) only sets the variable for `cat`, not for the `claude` process receiving the pipe. `--output-format json` emits a single JSON object when the agent finishes — resilient to long-running agents and session timeouts. **When Agent A returns**, extract token count: ```python import json d = json.load(open("/tmp/benchmark_{id}/agent_A/agent_A_run.json")) u = d.get("usage", {}) tokens_a = u.get("input_tokens", 0) + u.get("cache_creation_input_tokens", 0) + u.get("output_tokens", 0) is_error_a = d.get("is_error", False) ``` Record in `runs.json`: ```bash python3 _automation/benchmark-runner/scripts/record_run_result.py \ --eval-id {id} --model {CURRENT_MODEL_NAME} \ --status partial_a --tokens-a {tokens_a} ``` ### Step 3 — Archive Agent A Output Create the zip: ```bash cd /tmp/benchmark_{id} && zip -r benchmark_agent_a_{eval_id}.zip \ agent_A/output_A/ agent_A/agent_A_run.json ``` Upload to the `benchmark-results` GitHub release as a named asset. **The release must already exist** (create it once if needed). Use whichever method works in your environment — examples below; pick what works: - **`gh` CLI** (simplest if available): ```bash gh release view benchmark-results --repo RConsortium/pharma-skills \ || gh release create benchmark-results --repo RConsortium/pharma-skills \ --prerelease --title "Automated Benchmark Results" --notes "Rolling release." gh release upload benchmark-results /tmp/benchmark_{id}/benchmark_agent_a_{eval_id}.zip \ --repo RConsortium/pharma-skills --clobber ``` - **REST API via `curl`** (when only `GH_TOKEN` is available): ```bash # Get-or-create release, then POST to its upload_url with the zip as data-binary. # See https://docs.github.com/en/rest/releases for the exact endpoints. ``` - **MCP** does not currently expose a release-asset upload tool — use `gh` or `curl` for the upload step. Comment posting and reading can still use MCP. Construct the asset download URL (used in the partial comment state): ``` https://github.com/RConsortium/pharma-skills/releases/download/benchmark-results/benchmark_agent_a_{eval_id}.zip ``` **If no upload method works** (no `gh`, no token), skip the upload and set `agent_a_asset_url: null` in the partial state. Phase 2 will detect the null URL and re-run Agent A for that eval — wasteful but correct. ### Step 4 — Post Partial Comment Write the partial comment body to `/tmp/partial_comment_{eval_id}.md`: ```markdown ## Automated Benchmark Results — `{_skill_name}` 🟡 In Progress ### Run Metadata | Field | Value | |---|---| | **Eval ID** | `{id}` | | **Run date** | {YYYY-MM-DD HH:MM UTC} | | **Model** | `{CURRENT_MODEL_NAME}` | | **Skill version** | `{_skill_sha[:7]}` | | **Phase** | 1 of 2 complete — Agent A (with skill) finished | Agent A has completed. Agent B (without skill) will run in the next scheduled window (~5 h). Results will be updated here automatically. ``` Post it using whichever GitHub access method is available (see "GitHub Access" above). The partial comment `id` returned by the API is not needed for Phase 2 (Phase 2 discovers it by scanning), but log it for debugging. **Phase 1 is complete.** Print this summary to the user before exiting: ``` ✓ Phase 1 complete — Agent A finished for {eval_id} ({model}) • Output archived: {asset_url} • Partial comment: {comment_url} • Tokens used: {tokens_a:,} NEXT STEP — Phase 2 (Agent B + scoring): • If running as a scheduled routine: nothing to do. The next scheduled invocation (≥5 h from now, after the rolling usage window resets) will detect this partial state automatically and run Phase 2. • If running manually: re-invoke this skill any time. It will detect the BENCHMARK_PARTIAL marker on issue #{N} and run Phase 2 to completion. ``` Then exit cleanly. --- ## Phase 2 — Agent B Run + Scoring Runs when a `BENCHMARK_PARTIAL` state is found in a GitHub issue comment. Loads Agent A's output, runs Agent B, scores both, posts the full result. ### Step 5 — Load Partial State Parse the `BENCHMARK_PARTIAL` JSON from the comment body found during Phase Detection: ```python import re, json marker_re = re.compile(r'', re.DOTALL) m = marker_re.search(comment_body) state = json.loads(m.group(1)) # state keys: eval_id, model, skill_sha, issue_number, blinded_map, # agent_a_asset_url, run_date, tokens_a ``` Also reload the full eval case (for assertions, scoring prompt, prompt_b): ```bash # Send stdout (the eval-case JSON) to the file and stderr (warnings such as # the >100 KB bundle notice) to a separate log. Do NOT use `2>&1` here — it # would merge warning lines into the JSON file and break the json.load below. python3 _automation/benchmark-runner/scripts/get_next_eval.py \ --model {state["model"]} \ --priority-issue {state["eval_id"]} \ > /tmp/eval_case_{id}.json 2>/tmp/eval_case_{id}.log ``` Restore Agent A's output. If `agent_a_asset_url` is set, download and unzip it. Use whichever method works: ```bash mkdir -p /tmp/benchmark_{id}/agent_A/output_A # Option A — gh CLI: gh release download benchmark-results --repo RConsortium/pharma-skills \ --pattern "benchmark_agent_a_{eval_id}.zip" --dir /tmp/benchmark_{id}/ # Option B — curl (release assets are public for public repos; token only needed for private): curl -L "{agent_a_asset_url}" -o /tmp/benchmark_{id}/benchmark_agent_a_{eval_id}.zip # Then unzip: cd /tmp/benchmark_{id} && unzip -q benchmark_agent_a_{eval_id}.zip ``` If `agent_a_asset_url` is `null` (Phase 1 could not upload), re-run Agent A from scratch using the same procedure as Phase 1 Step 2 before continuing. ### Step 6 — Run Agent B (Without Skill) ```bash mkdir -p /tmp/benchmark_{id}/agent_B/output_B ``` Write `prompt_B.txt` — contains only `_prompt_b`. No skill content, no resource files: ```python with open("/tmp/benchmark_{id}/agent_B/prompt_B.txt", "w") as f: f.write(eval_case["_prompt_b"]) ``` Launch Agent B: ```bash export CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000 cd /tmp/benchmark_{id}/agent_B && \ cat prompt_B.txt | claude -p \ --allowedTools "Bash,Read,Write,Edit,Glob" \ --output-format json > agent_B_run.json 2>&1 ``` Extract token count and record: ```python d = json.load(open("/tmp/benchmark_{id}/agent_B/agent_B_run.json")) u = d.get("usage", {}) tokens_b = u.get("input_tokens", 0) + u.get("cache_creation_input_tokens", 0) + u.get("output_tokens", 0) is_error_b = d.get("is_error", False) ``` ```bash python3 _automation/benchmark-runner/scripts/record_run_result.py \ --eval-id {state["eval_id"]} --model {state["model"]} \ --status completed --tokens-b {tokens_b} ``` ### Step 7 — Score Blinded Outputs Copy outputs per `state["blinded_map"]` to `/tmp/benchmark_{id}/scoring/`: ```bash mkdir -p /tmp/benchmark_{id}/scoring/candidate_1 /tmp/benchmark_{id}/scoring/candidate_2 # blinded_map: {"candidate_1": "output_B", "candidate_2": "output_A"} (or reversed) cp -r /tmp/benchmark_{id}/agent_{X}/output_{X}/. /tmp/benchmark_{id}/scoring/candidate_1/ cp -r /tmp/benchmark_{id}/agent_{Y}/output_{Y}/. /tmp/benchmark_{id}/scoring/candidate_2/ ``` For each candidate, evaluate every assertion in the eval case: - **Pass** — clearly met - **Partial** — partially met - **Fail** — not met Score = `(passes + 0.5 × partials) / total_assertions` Then unblind using `state["blinded_map"]` to map candidate scores back to "With Skill" and "Without Skill". ### Step 8 — Format Full Report Write `/tmp/benchmark_comment_{skill}_{eval_id}.md`: ```markdown ## Automated Benchmark Results — `{_skill_name}` ### Run Metadata | Field | Value | |---|---| | **Eval ID** | `{id}` | | **Run date** | {YYYY-MM-DD HH:MM UTC} | | **Model** | `{model}` | | **Skill version** | `{skill_sha[:7]}` | | **Triggered by** | Scheduled | ### Scorecard | Metric | With Skill | Without Skill | |---|---|---| | **Score** | {score_A} ({pct_A}%) | {score_B} ({pct_B}%) | | **Assertions** | {pass_A} Pass · {partial_A} Partial · {fail_A} Fail | {pass_B} Pass · {partial_B} Partial · {fail_B} Fail | | **Skills loaded** | 1 | 0 | | **Execution time** | {time_A} min | {time_B} min | | **Token usage** | {tokens_a} | {tokens_b} | | **{Key Metric 1}** | {value_A1} | {value_B1} | | **{Key Metric 2}** | {value_A2} | {value_B2} | ### Key Observations - {2-4 bullet points comparing both agents} ### Verdict {1-2 sentence overall verdict} --- ## Technical Details & Artifacts
View Assertion Breakdown, Code Artifacts, and Logs ### Assertion Breakdown | Assertion | With Skill | Without Skill | |---|---|---| | {assertion_text_1} | {Pass/Partial/Fail} | {Pass/Partial/Fail} | ### Debugging Information #### Agent A (With Skill) - **Total Turns:** {num_turns from agent_A_run.json} - **Errors/Retries:** {is_error value, or "None"} #### Agent B (Without Skill) - **Total Turns:** {num_turns from agent_B_run.json} - **Errors/Retries:** {is_error value, or "None"} ### Detailed Artifacts **Agent A Output:** [Download Agent A Archive]({agent_a_asset_url}) #### Agent A (With Skill) {Key output files — .R, .json, text summaries} #### Agent B (Without Skill) {Key output files}
--- *Posted automatically by `benchmark-runner` · Repo: https://github.com/RConsortium/pharma-skills* ``` Note the `` marker at the bottom — this tells future Phase Detection scans that Phase 2 is done for this eval+model+sha. ### Step 9 — Post Full Results Comment Post as a **new comment** using whichever GitHub access method is available (see "GitHub Access" above). The new comment carries the `BENCHMARK_COMPLETE` marker; the partial comment can stay in place — future Phase Detection scans will see the COMPLETE marker on a later comment and skip the partial. If you prefer to also edit the partial comment to mark it as superseded (cleaner timeline), use whatever update method works in your environment (`gh api` PATCH, REST PATCH with `GH_TOKEN`, etc.). Optional — not required for correctness. **Phase 2 is complete.** Print this summary to the user before exiting: ``` ✓ Phase 2 complete — full benchmark posted for {eval_id} ({model}) • Score: With Skill {pct_A}% · Without Skill {pct_B}% • Comment: {comment_url} • Tokens — A: {tokens_a:,} · B: {tokens_b:,} ``` Then exit cleanly. --- ## Execution Flow ``` EVERY ROUTINE INVOCATION: Phase Detection │ ├─ BENCHMARK_PARTIAL found (no BENCHMARK_COMPLETE for same eval+model) ──► Phase 2 │ Step 5: load state from comment + restore Agent A output │ Step 6: run Agent B (without skill) │ Step 7: score blinded │ Step 8: format full report │ Step 9: post full results comment (with BENCHMARK_COMPLETE marker) │ EXIT │ └─ No partial found ──► Phase 1 Step 0: R pre-flight Step 1: get_next_eval.py → if UP_TO_DATE, EXIT Step 2: run Agent A (with skill) Step 3: archive + upload Agent A output Step 4: post partial comment (with BENCHMARK_PARTIAL marker + state JSON) EXIT ``` --- ## Notes on Model Name `{CURRENT_MODEL_NAME}` is the fixed series label `Claude Routine` — see "Model Selection — Series Label and Sub-Agent Model" at the top of this skill. Sub-agents inherit the host's default model rather than being pinned with `--model`, so no concrete model ID is recorded in issue comments or release assets. The dedup logic normalises the label, so every run posted under `Claude Routine` groups into one series. ## Notes on Distributed Selection When several people run the same model, set distinct `--runner-id` values. The dispatcher hashes runner-id + model + UTC minute + eval-id + skill-SHA to spread different runners across different pending evals. Runners starting in the same minute may collide; the GitHub issue-comment deduplication (checking for `BENCHMARK_COMPLETE` markers) prevents redundant Phase 1 runs. ## Notes on Rate Limits If Agent A or Agent B hits a usage rate limit mid-run (`is_error: true`, result contains "You've hit your limit"): - **Agent A rate-limited in Phase 1**: record `status: error_a_rate_limited` in `runs.json`, do NOT post a partial comment, exit. The next Phase 1 invocation will retry. - **Agent B rate-limited in Phase 2**: record `status: error_b_rate_limited`, do NOT post a full results comment. Leave the `BENCHMARK_PARTIAL` comment in place so the next Phase 2 invocation retries Agent B. Include a note in the partial comment body edit if possible. ## Success Criteria - One phase executed per invocation (~20 min each) - State persists entirely in GitHub issue comments — no commits, no repo write access needed from the human user - Blinded scoring: `_blinded_scoring_map` is never visible to the scorer - Deduplication: `BENCHMARK_COMPLETE` marker prevents re-running finished evals - Results posted on the correct GitHub issue with full assertion breakdown