---
name: run-openrouter-experiment
description: Design and run reproducible OpenRouter model experiments from a natural-language request, including model and parameter matrices, repeated prompts, budget-aware execution, response checks, cost and latency capture, and JSONL, CSV, and Markdown artifacts. Use for prompt tests, model comparisons, research experiments, ablations, structured-output checks, or repeatable evaluations that do not require the full Ori harness.
---

# Run OpenRouter Experiment

Turn the user's request into an experiment spec that follows `references/spec.md`, then use `scripts/run_experiment.py`. Prefer this lightweight runner for prompt and model comparisons; use `spawn-ori-eval` when the subject is a complete agent or coding harness.

## Workflow

1. Use live OpenRouter MCP model, endpoint, benchmark, and pricing data to resolve model names and validate requested parameters. Never guess current slugs or prices.
2. Build the smallest experiment matrix that answers the question. Preserve the user's prompt exactly unless they request variants.
3. Require an explicit USD `max_budget_usd`. For a hard server-enforced ceiling, recommend a dedicated limited API key; client-side tracking can overshoot by one in-flight request.
4. Show the models, prompts, repetitions, parameters, checks, expected request count, output location, and maximum budget. Obtain explicit confirmation before a live run.
5. Write the JSON spec outside the skill directory, then run:

```bash
python3 scripts/run_experiment.py experiment.json --dry-run
python3 scripts/run_experiment.py experiment.json --confirm-budget 2.00
```

6. Summarize `report.md`, identify failures or missing cost metadata, and link the artifact directory. Do not hide unsuccessful runs.

## Output contract

The runner creates one new directory containing:

- `experiment.json`: normalized, secret-free configuration and provenance.
- `results.jsonl`: one complete record per attempted request.
- `summary.csv`: per-model counts, success rate, check pass rate, mean latency, and total reported cost.
- `report.md`: human-readable comparison and budget status.

Never place API keys, management keys, OAuth tokens, authorization headers, or unrelated account data in a spec or artifact. The runner rejects common secret fields and reads `OPENROUTER_API_KEY` only from the environment.

Keep execution sequential so budget decisions happen between requests. Set `reserve_per_request_usd` when the user supplies a defensible per-call upper bound. Stop before the next call when the reported cumulative cost or reservation would reach the budget. Stop fail-closed if OpenRouter does not provide usable cost metadata.

Do not enable retries by default. If the user approves retries, set `retry_attempts`; retry only `429` or `503`, honor `Retry-After`, and record every final failure.

Run `python3 scripts/run_experiment.py --self-check` after changing the runner.