The judge scores each output. Default "same as target" always works; pick a distinct model to reduce self-preference bias.
Use this for a model not in the list, or one your key has special access to.
Run each case N times to surface run-to-run variance. 1 is fastest; 3–5 reveals how noisy a score is (costs N× more).
Judge samples folds N judge calls to the median, cutting single-call judge noise (costs N× judge calls). Concurrency runs that many cases in parallel — faster on big suites, but watch provider rate limits.
What are we testing?
Grab the system prompt from whatever console you're on, or paste it.
litmus runs on your own key (OpenAI, Anthropic, or Google), stored only in this browser. Add one to begin.
Don't have a prompt yet?
Go from an idea to a tested system prompt — litmus writes it, you run it.
Prompt analysis
The judges that score outputs
litmus found these dimensions and wrote a rubric for each. Edit any, or add your own.
Eval cases · remove any you don't want
Test whether the model calls the right tool with valid arguments. Define the tools available to it, then add a case that asserts the expected call.
Test a multi-step agent: give it a goal and mock tools (with scripted results — you can inject a failure to test recovery). litmus runs the model in a loop, feeding tool results back, and scores whether it reaches the goal. Tools are mocked — nothing real is executed.
Generate an image per case (the case text is the prompt) and check it deterministically:
size, format, and count are gates; requested elements are matched against a vision description.
Requires an OpenAI image model (e.g. gpt-image-1) as the target.
Running · on your key
Results
◷ Scores vary run-to-run — the target model is non-deterministic. A different score doesn't always mean a different prompt; re-run a version to see its spread.
From the failing cases
Every pass is kept
MCP server
Point litmus at a live MCP server to inspect what it exposes, call its tools, and run an adversarial security scan.