mcpcanary
The canary for your MCP server.
Probe it. Drill it with YAML tests. Lint its config.
Catch failures in CI — before your users do.
---
You built an MCP server. You tested it by clicking around the Inspector — once.
Then you changed a tool description, shipped it, and found out from users that
the server hangs on startup.
**The Inspector is a great GUI and a terrible regression suite.** It can't run
in CI, it can't assert anything, and it won't tell you that your `npx` entry
is missing `-y` or that two tools have identical descriptions so the model
picks between them at random.
`mcpcanary` is the missing command line:
```bash
npx mcpcanary probe -- npx -y @modelcontextprotocol/server-everything # does it even start?
npx mcpcanary init -- node ./build/server.js # scaffold a drill file
npx mcpcanary run drills/ # do the tools still work?
npx mcpcanary lint --probe # will clients choke on my config?
```
## Why you want this
- **Probe replaces "let me open the Inspector and see".** One command answers: does it start, what protocol version, what capabilities, how many tools, are any descriptions empty? Exit code tells CI the truth.
- **Drills are YAML, not code.** A tool call plus assertions in six lines. No test framework to wire up, no client library to learn.
- **Lint catches the failure modes nobody warns you about**: `npx` without `-y` hanging every headless client, `mcpservers` instead of `mcpServers` silently ignored, hardcoded API keys, remote servers over plain `http://`, prompt-injection phrasing in third-party tool descriptions, duplicate tool descriptions…
- **Built for CI first**: stable exit codes, `--json` output, GitHub Actions annotations (`::error`) out of the box.
## Install
```bash
npm install -D mcpcanary # as a dev dependency
# or just run it
npx mcpcanary --help
```
Node ≥ 20.
## Quick start
```bash
# 0. Let init write your first drill file from a live probe:
mcpcanary init -- node ./build/server.js
```
```text
created mcp.drill.yaml with 6 drill(s) scaffolded from your server
next: review the TODO expectations, then run:
mcpcanary run mcp.drill.yaml
```
`init` lists your real tool names, scaffolds arguments from each tool's input
schema, and leaves clearly-marked TODO expectations for you to tighten.
(If you skip this step, the YAML below is everything there is.)
```bash
# 1. Does my server start and what does it expose?
mcpcanary probe -- node ./build/server.js
```
```text
tiny-fixture @ 1.2.3 — connected in 104ms
capabilities: tools, resources
tools (2)
· add — Adds two numbers and returns the sum.
· echo — Echoes back whatever it receives.
```
```bash
# 2. Write a drill file…
```
```yaml
# mcp.drill.yaml
server:
command: node
args: [./build/server.js]
timeoutMs: 10000
drills:
- name: adds numbers
tool: add
arguments: { a: 2, b: 3 }
expect:
textEquals: "5"
latencyMsAtMost: 2000
- name: errors surface properly
tool: divide
arguments: { a: 1, b: 0 }
expect:
isError: true
textContains: zero
- name: structured JSON is intact
tool: stats
expect:
jsonPath: $.total
jsonEquals: 42
- name: the toolbox is complete
listTools:
contains: [add, divide, stats]
```
```bash
# 3. Run it (add it to `npm test`)
mcpcanary run mcp.drill.yaml
```
```text
mcp.drill.yaml — my-server@1.0.0
✓ adds numbers 3ms
✓ errors surface properly 1ms
✓ structured JSON is intact 2ms
✓ the toolbox is complete
ok: 1 file, 4 drills, 4 passed, 118ms
```
When something breaks you get the assertion, the actual value, **and the
server's last stderr lines** — which is usually the whole debugging session:
```text
✗ bad math
· text equals "3" — got "2"
ok: …
```
```text
✗ unreachable
server failed to start or respond: connect ECONNREFUSED
stderr tail:
| boom: missing OPENAI_API_KEY
```
## What lint checks
`mcpcanary lint` scans `.mcp.json`, `.cursor/mcp.json`, `.vscode/mcp.json`,
Claude Desktop / Claude Code configs and `server.json` manifests it finds.
| Rule | Severity | Catches |
|------|----------|---------|
| C001 | error | `mcpservers` / `MCP_SERVERS` key typos that clients silently ignore |
| C002 | info | config files with no server entries |
| C003–C004 | error | malformed entries; neither `command` nor `url` |
| C005 | error | **`npx`/`bunx`/`uvx` without `-y`** — startup hangs at the install prompt in every headless client |
| C006 | warning | relative paths that only work on your machine |
| C007–C008 | warn/error | Windows `cmd /c` pitfalls; malformed `env` |
| C009 | error | hardcoded API keys (`sk-…`, AWS, GitHub, Slack) in config files |
| C010–C011 | error/warning | invalid URLs; remote servers over plain `http://` |
| M001–M007 | error/warning | registry manifest problems: missing fields, non-reverse-DNS names, invalid npm identifiers |
| D001–D008 | error/warning | live tool audits (`--probe`): missing descriptions, name-only descriptions, **duplicate descriptions across tools**, invalid tool names, oversized context-burning descriptions, prompt-injection phrasing |
Add it to review CI and stop bad configs at the PR:
```bash
mcpcanary lint --probe
```
## Drills reference
A drill file has an optional `server:` block (stdio via `command`/`args`/`env`/`cwd`,
or HTTP via `url`/`headers`), an optional default `timeoutMs`, and a `drills:` list.
| Drill type | Fields | Asserts on |
|------------|--------|------------|
| tool call | `tool`, `arguments`, `expect` | result content |
| listing | `listTools` / `listResources` / `listPrompts` | advertised names |
Expectations for tool calls:
| Key | Meaning |
|-----|---------|
| `textContains` | string or array of strings, all must appear |
| `textEquals` | exact (trimmed) text match |
| `textMatches` | regular expression source |
| `isError` | whether the tool result is an error |
| `jsonPath` (+ `jsonEquals` / `jsonContains` / `jsonExists`) | JSONPath into `structuredContent`, falling back to JSON embedded in text output |
| `latencyMsAtMost` | wall-clock budget for the call |
Listing expectations: `contains`, `excludes`, `countExactly`, `countAtLeast`, `countAtMost`.
Exit codes: `0` everything passed · `1` failures/errors · `2` usage problem (nothing to run, unreadable file).
## GitHub Actions
Annotations are emitted automatically when `GITHUB_ACTIONS=true`; failures show
up inline on the PR:
```yaml
name: mcp-drills
on: [push, pull_request]
jobs:
drill:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
- run: npm ci && npm run build
- run: npx mcpcanary run drills/
- run: npx mcpcanary lint --probe
```
## How it compares
| | Inspector (official) | GUI test platforms | **mcpcanary** |
|---|---|---|---|
| Runs in CI | ✗ (interactive GUI) | partial | ✓ |
| Declarative assertions | ✗ | some | ✓ YAML |
| Config linting (`npx -y`, secrets, typos) | ✗ | ✗ | ✓ |
| Tool-description audit | manual | manual | ✓ automated |
| Setup | open app | account/API keys | one npm package, zero config |
| Output for bots | ✗ | ✗ | `--json` + GH annotations |
mcpcanary doesn't replace the Inspector for interactive exploration — it replaces
*re-opening the Inspector every time you change something*.
## Roadmap
See [ROADMAP.md](ROADMAP.md) for the full plan. Highlights:
- **v0.2** — `init` scaffolding ✅, standalone GitHub Action, `--watch`, session reuse
- **v0.3** — snapshot testing, SSE transport, schema linting depth, token-cost report
- **v0.4** — registry health sweep, pass/fail badges, preset drill packs
PRs welcome — see [CONTRIBUTING.md](CONTRIBUTING.md).
## License
[MIT](LICENSE)