Getting started
litmus runs your system prompt against the model you actually ship on, scores every output with an LLM judge, and keeps each attempt as a version you can compare. Everything runs locally in your browser with your own API keys.
The progress rail at the top of the panel tracks where you are
Paste the system prompt you want to test, or click ⤓ Grab from this tab to pull one from the page. Pick the target model — the model you actually ship on.
litmus scores your prompt on language, intent, format, and tone for the chosen model, and lists concrete rewrite suggestions. This is read-only feedback — nothing is changed yet.
litmus finds the quality dimensions your prompt's output should be judged on and generates a rigorous LLM-as-judge rubric for each. You can edit a rubric, regenerate one, add your own dimension, or check coverage.
litmus generates a set of test cases — typical, edge, and adversarial inputs that match your prompt's real input contract. Remove any you don't want, regenerate the set, or add more (see tips below). The estimated cost updates live.
Each case is sent to your target model, then the judge scores the output against the rubric. Speed (time-to-first-byte, tokens/sec) is measured as it runs.
An overall score, pass/fail counts, a speed strip, and a per-case table. Failing cases sort to the top. Click a case row to expand its full rationale from the judge.
Results — failing cases first, click a row to expand its full rationale
From the failing cases, litmus proposes ranked, concrete edits to your prompt. Click Apply fixes & review → to apply every suggestion to your system prompt via an LLM rewrite — you land back on Capture with the revised prompt, where you review it and re-run when ready. It does not re-run automatically.
Every run is saved as a version. Compare the baseline against the latest by dimension, and export the whole history as Markdown or JSON.
If your model calls tools/functions, you don't need the rubric flow. On the Capture step, under "What are you testing?", choose Tool & agent instead of Output quality. The button changes to Set up tool & agent tests → and takes you straight to the Cases step — skipping Analyze and Build eval prompt, since tool and agent tests are scored deterministically, with no LLM judge.
Checks that, for a given message, the model calls the right tool with valid arguments and avoids the ones it shouldn't.
name, optional description, and a JSON-Schema parameters object. litmus validates it live.✦ Generate tool tests to have litmus propose cases from your catalog, or add one manually: a user message, the expected tool, any forbidden tools, and optional required argument values.Tests a model that uses tools across several turns to finish a task. In the 🤖 Agent scenarios panel, paste a scenario as JSON: a goal, the tools it may call (each with scripted results — you can inject a failure to test recovery), a maxSteps cap, and optional successContains keywords the final answer must include. litmus runs the model in a loop — tool call → mocked result → continue — and scores whether it reached the goal using only the tools you defined.
Tools are mocked. litmus never executes a real tool; it returns the responses you scripted, so runs are deterministic and side-effect-free. Both tool tests and agent scenarios work with OpenAI, Anthropic, and Google targets (litmus normalizes each provider's tool-call format). Richer per-dimension trajectory scoring (efficiency, recovery) is planned; today's agent verdict is goal-reached plus correct tool selection.
If you're building an MCP (Model Context Protocol) server, litmus can connect to it directly and exercise it end to end — no system prompt or rubric needed. On the Capture step, under "What are you testing?", pick the MCP server chip. You'll enter the server's URL, connect, and then inspect and test what it exposes.
Enter the server's endpoint and connect. litmus first asks your browser for host permission for that server's origin — this is granted per origin and stored locally, so you approve each new server once. An auth header is optional: add one (for example a bearer token) if your server requires it, or leave it blank for open servers. If the server requires OAuth, leave it blank and use the sign-in prompt litmus shows on a 401.
Instead of typing the fields, open "Paste MCP config JSON" and paste the mcpServers block from your existing client config (Claude Desktop, Cursor, VS Code). litmus fills in the name, URL, transport, and auth header — you review them and press connect; nothing is dialed or saved until you do. If the config lists several servers you get a picker.
litmus runs in the browser, so it cannot spawn local processes: a stdio entry only imports when it bridges to an http(s) endpoint, as with npx mcp-remote https://host/mcp — the URL in the args is what gets used. Entries that can't be imported are listed with the reason instead of being dropped silently. A token written as an environment placeholder (${TOKEN}) is not imported — paste the real value into the auth field yourself.
If the server answers the handshake with 401, litmus doesn't just report the error — it offers to authenticate. Press Sign in with OAuth and litmus discovers the server's OAuth metadata, registers itself as a client, and opens the server's own sign-in page in a window Chrome controls. Your password never passes through litmus; only the token that comes back does.
What happens under the hood: authorization code + PKCE (S256), with the token bound to that specific server (resource), scopes taken from what the server advertises. The access token and its refresh token are stored in chrome.storage.local and never leave your browser. When the access token expires mid-session litmus refreshes it and retries the call; if the grant has been revoked you're asked to sign in again. Sign out deletes the stored tokens, as do Forget saved server and Delete all keys.
A pasted Authorization header always overrides the OAuth grant, so you can test with a specific token whenever you need to.
Once connected, litmus lists the server's tools, resources, and prompts so you can see exactly what it advertises, with each tool's input schema.
Runs a structured pass over the server — checking that it advertises and responds correctly over the protocol, that tool schemas are well-formed, and that listed capabilities behave as declared.
You don't have to invent test cases by hand. Once connected, litmus can read the server's discovered tools — names, descriptions and input schemas — and write tests for them two ways. Both show you what they produced before anything runs.
Generate agent scenarios asks your model for realistic goals a user would pursue through these tools, each with a success check (substrings a correct answer must contain). You review them, then add them as eval cases. Running them exercises the whole loop: your target model picks the tools, calls them for real on the live server, and gets scored on whether it reached the goal.
Generate direct tool probes skips the model at execution time. It builds concrete argument sets from each tool's schema — mostly valid, plus the occasional deliberately-invalid one the server should reject — calls the tools directly, and grades each reply (no error, non-empty, contains what it should). Cheaper and faster than a full agent run, and it isolates the server rather than the model. Probes are prompted to stay read-only, and running them asks for confirmation first.
Generated scenarios also record which tools they expect (expectedTools), and at least one of them must actually be called — so a case can't pass by reaching plausible-looking text through some unrelated tool. It's "at least one", not "all", because a capable agent may reach the goal in fewer calls than predicted and that shouldn't count against it.
Two things to know about reading agent-scenario results. First, a case bound to a server fails if the agent never called a tool — models often already know the answer to a question about a public repository, and a case satisfied from memory would report your server as healthy without ever contacting it. The rationale says so explicitly when this happens. Set "requiresToolUse": false in the scenario if you deliberately want to test answering without tools. Second, many MCP tools are themselves LLM-backed and word their replies differently on every call, so a success check can pass or fail on wording alone; if a case flakes, raise samples in settings rather than weakening the check.
Both need a provider key, since generation is a model call. Anything the model invents that the server doesn't actually expose is discarded, and probe arguments are re-checked against the real schema before they're offered to you.
You choose which tools get probed. The scan calls each selected tool many times with hostile inputs, which is harmless against a read but a series of real mutations against something like create_room. litmus classifies every discovered tool — using the server's own readOnlyHint/destructiveHint annotations where present, otherwise the tool's name and description — and anything that looks like it writes starts unchecked. You tick it deliberately or not at all, and the confirmation names the write tools you included. Labels are marked as declared by the server or inferred by litmus, because a name-based guess is a guess.
An adversarial pass that sends crafted, hostile payloads to the live server — probing for things like prompt-injection openings, argument-injection, and missing input validation — and reports findings by severity.
How to read the results: possible leak (high) means the reply contained a genuinely sensitive marker, such as /etc/passwd content. accepted-invalid means the server ran a call that violated its own schema. echoed input is informational — the server quoted your payload back, which most servers do in validation messages; it only matters if that text is later rendered or re-interpreted somewhere. A server that reports refusals in the result text rather than via the error flag is credited as having rejected the input.
Before probing, litmus works out realistic baseline values for each tool's other parameters, so the payload-carrying field is the only thing the server can object to. Without this, a placeholder like "sample" fails the server's own domain validation (a repository name, an id, a path) and the probe never reaches the logic under test — the scan then reports a clean sweep of "rejected" while having tested nothing but the input validator. Baselines are reused from any probes you already generated, otherwise litmus generates them (one model call, no server calls). If no key is available it falls back to placeholders and says so in the report's notes, so an all-clear is never ambiguous.
+10 cases to append ten fresh cases without losing the ones you have. litmus is told to avoid duplicating existing cases.Back button, so you can revisit earlier steps without losing your place.The first generation produces 12 cases, spread across typical, edge, and adversarial inputs. Use +10 cases to add more, or ↻ Regenerate to replace the set.
Keys are stored only in your browser's local extension storage. Your prompt and cases are sent only to the model provider you choose (OpenAI, Anthropic, or Google) to run the test. litmus has no backend and collects no analytics. See the Privacy Policy.
A model judging its own output tends to be lenient with itself. Picking a different judge model reduces this self-preference bias and gives more trustworthy scores.
It's an estimate of the total API spend for a run, based on case count and average token sizes. If it exceeds your spend cap (set in Settings), the Run button is disabled until you raise the cap or trim cases.
Deterministically, not by the LLM judge. A tool test passes only if the model called the expected tool, the arguments parsed as JSON and matched the tool's schema (required fields, types), any required values matched, and no forbidden tool was called. Because it's deterministic, a tool test's pass/fail doesn't drift run-to-run the way judged scores can.
Yes. Unlike Agent scenarios, where tools are mocked, MCP mode connects to your live server and every tool call runs for real, with real side effects. The security scan goes further and sends adversarial payloads to that live server. Only test servers you own or control, and prefer a staging instance. Connecting requires host permission for the server's origin (granted per origin, stored in this browser); an auth header is optional.