mcp-eval-gate
A CI regression gate for MCP servers. It calls your server's tools, compares the results to a
saved baseline, and fails the build when the output gets worse, even if the schema did not change.
Read the write-up: [MCP server testing in CI: the bug no schema check can see](https://medium.com/@karachiwalaumer2612/mcp-server-testing-in-ci-the-bug-no-schema-check-can-see-39cfce248043).
## Why
An MCP server's tool names and input schemas can stay identical while what a tool returns
quietly gets worse: a refactor breaks a code path, a dependency bump changes behavior.
Nothing crashes, the agent using it just gets worse answers. The same goes for the contract a
server declares: a schema dialect some clients reject, a tool a client silently drops, an
error message that stopped explaining what went wrong.
Other tools cover parts of this. The official Inspector's CLI runs scripted single-run
assertions. `mcp-server-diff` compares declared schemas and says it does not test output
correctness. A few other early tools record golden outputs and diff them too, for example
[vexyo](https://github.com/vexyohq/vexyo) and
[cisco-open/mcptoolkit-test](https://github.com/cisco-open/mcptoolkit-test).
`mcp-eval-gate` is another take on golden-output regression, with the server's declared
contract checked next to it.
## How it works
1. `init --command ""` lists the server's tools and writes a
`golden_set.yaml`. Tools the server marks read-only and that need no arguments become cases.
Every other tool is a commented stub, since a case calls the tool for real.
2. `run --update-baseline` calls every tool and records what came back, once, while
everything is known to be good. The baseline keeps the raw output, the structured content,
whether it was an error, and the server's contract.
3. `run` calls the same tools again and compares against the baseline. It prints a table and a
diff, and exits `1` if anything failed or changed, so any CI job can gate on it.
## Install
Requires Python 3.12+.
```bash
pip install mcp-eval-gate
# only if you use match_type: judge
pip install "mcp-eval-gate[judge]"
# or run it without installing
uvx mcp-eval-gate --help
```
## Quickstart
```bash
mcp-eval-gate init --command "node dist/index.js" # writes golden_set.yaml from the live tool list
mcp-eval-gate run --update-baseline # first run: record the baseline
mcp-eval-gate run # later runs: gate on changes
```
`init` with no server writes a sample that runs as is against the official MCP reference
server (needs Node.js). `--update-baseline` refuses to record while any case fails, since a
baseline should hold known-good output. Commit `golden_set.yaml` and `baseline.json`.
## Golden set
```yaml
server:
command: node
args: ["dist/index.js"]
# or, for an HTTP server instead of stdio:
# url: http://localhost:3000/mcp
cases:
# compare against whatever was recorded in the baseline
- id: list-projects
tool_name: list_projects
match_type: snapshot
# substring match
- id: get-weather-nyc
tool_name: get_weather
tool_args:
city: "New York"
match_type: contains
expected_output: "New York"
# exact match
- id: cancel-subscription
tool_name: cancel_subscription
tool_args:
immediate: true
match_type: exact
expected_output: "cancelled"
# the tool is supposed to fail, and say why
- id: unknown-user
tool_name: get_user
tool_args:
id: "does-not-exist"
match_type: contains
expected_output: "not found"
expect_error: true
# scored by an LLM against a rubric (needs ANTHROPIC_API_KEY)
- id: retention-policy-answer
tool_name: search_docs
tool_args:
query: "data retention policy"
match_type: judge
judge_criteria: "Answer must state data is retained for 90 days"
min_judge_score: 0.8
```
[`examples/golden_set.yaml`](https://github.com/Umer-2612/mcp-eval-gate/blob/main/examples/golden_set.yaml) is a copy you can edit.
### Case fields
| Field | Required | Default | Meaning |
|---|---|---|---|
| `id` | yes | | Unique name for the case |
| `tool_name` | yes | | The MCP tool to call |
| `tool_args` | no | `{}` | Arguments passed to the tool |
| `match_type` | no | `contains` | `snapshot`, `exact`, `contains`, or `judge` |
| `expected_output` | for `exact` and `contains` | | The text to compare against |
| `judge_criteria` | for `judge` | | The rubric the LLM scores against |
| `min_judge_score` | no | `0.8` | Lowest judge score that passes |
| `expect_error` | no | `false` | The case passes only if the tool returns an error, and the error still has to match |
| `normalize` | no | | Normalizers for this case, on top of the file level ones |
| `timeout_seconds` | no | `30` | A case that takes longer fails on its own, so one hung tool does not stall the run |
A tool call that returns an error fails its case, unless the case sets `expect_error`.
`exact` compares character for character, so a trailing newline is a difference and the
failure shows it.
### Normalizers
Output that changes on every run, like timestamps and generated ids, can't be compared as is.
List normalizers at the top of the golden set to apply them to every case, or under a case to
apply them to that one. They run on both sides of a comparison, so the baseline keeps the raw
output.
```yaml
normalize:
- trim # strip surrounding whitespace
- tmp_paths # /tmp/tmpab12/x becomes /x
- regex: '\d{4}-\d{2}-\d{2}T[\d:.]+Z' # replace anything that matches
replace: ""
- ignore_keys: [id, created_at] # drop these keys at any depth of JSON output
```
### Exit codes
| Code | Meaning |
|---|---|
| `0` | Every case passed and nothing regressed against the baseline |
| `1` | A case failed or regressed, `--update-baseline` was refused because a case failed, `compare` found a difference, or `lint` found a problem |
| `2` | The golden set or baseline is invalid, or the server could not be started or reached |
## Contract
Each recorded baseline also holds the server's contract: protocol version, server info,
capabilities, and every tool as listed by `tools/list`. On a later run the gate diffs the
current contract against it and prints what changed, for example a tool's `inputSchema`, a
renamed server, a different tool order.
Contract changes are warnings by default. `run --strict-contract` fails the build on them.
To silence a path that is expected to move, such as a version string, list it in the golden set:
```yaml
contract_ignore: [serverInfo.version]
```
The same run also lints the contract and each call's result for things that make clients drop
or reject tools: a `$schema` other than 2020-12, a root-level `allOf` or `if` in an
`inputSchema`, an `inputSchema` that is not an object, and structured content that is missing
or does not match the tool's `outputSchema`.
`lint` does only the contract check, and needs no golden set or baseline:
```bash
mcp-eval-gate lint --command "uvx some-mcp-server"
mcp-eval-gate lint --url http://localhost:3000/mcp
```
## Compare two servers
`compare` runs your golden set against two servers and reports where their behavior differs.
Server A is the `server` block of the golden set, server B is passed on the command line.
Every case is treated as a snapshot, A's output is the reference, so cases need no expected
values. A case that can't run on A counts as a failure. Use it for an old and a new SDK version, or a release and its candidate.
```bash
mcp-eval-gate compare --config golden_set.yaml --against "node dist-next/index.js"
mcp-eval-gate compare --config golden_set.yaml --against-url http://localhost:4000/mcp
```
It also diffs the two contracts, and exits `1` if anything differs.
## GitHub Action
```yaml
- uses: actions/checkout@v4
- uses: Umer-2612/mcp-eval-gate@main
with:
config: golden_set.yaml
baseline: baseline.json
```
It installs the package, runs the gate, and writes the results table to the job summary.
`strict-contract: "true"` also fails on contract changes, and `package` takes a pinned
requirement such as `mcp-eval-gate==0.4.0`. This repository's CI runs the action against a
real server, once expecting a pass and once expecting a failure.
## Run as an MCP tool
The package also installs `mcp-eval-gate-mcp`, an MCP server over stdio with one tool,
`run_eval_gate(config_path, baseline_path, update_baseline, strict_contract)`. Add it to an MCP
client's config:
```json
{
"mcpServers": {
"mcp-eval-gate": { "command": "mcp-eval-gate-mcp" }
}
}
```
Without installing, use `uvx`:
```json
{
"mcpServers": {
"mcp-eval-gate": { "command": "uvx", "args": ["--from", "mcp-eval-gate", "mcp-eval-gate-mcp"] }
}
}
```
The tool declares its annotations: not read-only (`update_baseline` rewrites a file), it reaches
whatever server the golden set names, and running it twice gives the same result.
## Validation
[`validation/`](https://github.com/Umer-2612/mcp-eval-gate/tree/main/validation) has runs against real servers, each with the actual command
output committed and steps to reproduce it. No paid API calls.
- A one-line regression planted in the official MCP reference server, caught with exit
code `1` and a real diff.
- [A real bug](https://github.com/Umer-2612/mcp-eval-gate/tree/main/validation/utf8-boundary) the official filesystem server shipped (garbled
text when a multi-byte character straddled a read boundary), caught by running the commit
before its upstream fix against a baseline from the fixed commit. The
[write-up](https://medium.com/@karachiwalaumer2612/mcp-server-testing-in-ci-the-bug-no-schema-check-can-see-39cfce248043)
walks through it.
- [The same three tools on mcp 1.30.0 and mcp 2.2.0](https://github.com/Umer-2612/mcp-eval-gate/tree/main/validation/sdk-1x-vs-2x).
`compare` found that 2.x hides the exception message on a failing tool (the text goes from
`Error executing tool boom: kaboom...` to `Error executing tool boom`), and that the server
version string is reported differently. Everything else matched. Upgrading the SDK is
a one-line change that a schema check never sees.
These are small. They show the gate works end to end on real code, not how often this class
of bug occurs.
## Limitations
- Output that varies between runs needs a normalizer, and you write it. There is no automatic
detection of timestamps or ids.
- `snapshot` compares the text and the structured content a tool returns. `exact` and
`contains` compare the text only. There is no check of resources or prompts.
- The contract check reads what the server declares. It can't see how a particular client
reacts to it beyond the lint rules above.
- `match_type: judge` has only been tested against a stub client, not the live Anthropic API.
## Feedback
If you maintain an MCP server, I'd like to know what you would want to test and what got in
the way. Open an [issue](https://github.com/Umer-2612/mcp-eval-gate/issues/new/choose) with the
feedback form.
## Development
```bash
uv sync --extra judge --dev
uv run pytest --cov=src --cov-report=term-missing
uv run ruff check src tests
```
## License
MIT