# mcp-queue-doctor

An [MCP](https://modelcontextprotocol.io) server that **diagnoses** Postgres job
queues — [pg-boss](https://github.com/timgit/pg-boss) and
[graphile-worker](https://github.com/graphile/worker). Retry storms, stuck
workers, missed schedules, expiry overruns: what is broken, why, and the safest
way to recover.

```
❌ "3 jobs in enrichment/corpus-fill are in state 'failed'."

✅ "enrichment/corpus-fill failed 140 times over 3m, peaking at 50 failures in a
   single minute. 91% share one error, which looks like an upstream rate limit.
   This is one fault reproduced many times, not many separate faults — so the fix
   belongs at the source, and retrying the jobs individually will reproduce it.

   Recovery, safest first:
     1. Stop enqueuing to this queue — every new job feeds the same failure.
     2. Confirm when the upstream quota resets; treat that as the time to resume.
     3. Add a cooldown gate after N consecutive 429s.
     ⚠ Do NOT bulk-retry yet — the upstream is still limited.

   Evidence: 140 failures, 91% 'HTTP 429 Too Many Requests (daily quota
   exceeded)', peak 50/min, busiest minutes [...], 12 other errors [...]"
```

The second answer is the product. Every finding carries the evidence it was
drawn from, so you — or an agent — can check the reasoning instead of trusting
it.

## Where the heuristics come from

The rules are extracted from a morning health check that has run daily in
production since April 2026 against a pg-boss instance driving ~30 cron queues.
Every threshold was tuned by a real false positive or a real missed failure, and
each rule below names the incident that motivated it. That provenance is the
point: these are not heuristics invented for a README.

## Install

```bash
npm install -g mcp-queue-doctor
```

```json
{
  "mcpServers": {
    "queue-doctor": {
      "command": "mcp-queue-doctor",
      "env": {
        "QUEUE_DOCTOR_DATABASE_URL": "postgres://readonly:pw@localhost:5432/app"
      }
    }
  }
}
```

Then ask: *"Is anything wrong with my job queue?"*

**Want to see it work first?** [`examples/demo`](examples/demo) spins up a
throwaway Postgres and manufactures seven failures in about a minute. It also
plants a graphile-worker instance in the same database, where four of the seven
rules go deliberately silent — the clearest way to see what capability
declaration actually buys you.

## Connecting it

The server speaks stdio, so every MCP client starts it as a subprocess. The
only thing that varies is where the config lives — and whether that process can
reach your database.

**Claude Desktop** — `~/Library/Application Support/Claude/claude_desktop_config.json`
on macOS, `%APPDATA%\Claude\claude_desktop_config.json` on Windows. Use the
`mcpServers` block above, then restart the app.

Desktop launches its subprocesses from the app bundle, not a login shell, so
`PATH` is minimal and a bare `mcp-queue-doctor` or `npx` often fails to resolve.
Give it an absolute path — `which mcp-queue-doctor` after a global install, or
the absolute path to `npx` with `["-y", "mcp-queue-doctor"]` as its args.

**Claude Code** — one command, no file editing:

```bash
claude mcp add queue-doctor -e QUEUE_DOCTOR_DATABASE_URL=postgres://... -- npx -y mcp-queue-doctor
```

Add `-s project` to write a checked-in `.mcp.json` at the repo root instead of
your personal config, so everyone working in that repo gets the tool.

**Cloud / remote sessions** (Claude Code on the web, and any other headless
runner) — a checked-in `.mcp.json` is the only mechanism that works, because
nobody is there to answer an approval prompt. Reference the connection string
rather than committing it; Claude Code expands `${VAR}` and `${VAR:-default}`
in `.mcp.json`:

```json
{
  "mcpServers": {
    "queue-doctor": {
      "command": "npx",
      "args": ["-y", "mcp-queue-doctor"],
      "env": { "QUEUE_DOCTOR_DATABASE_URL": "${QUEUE_DOCTOR_DATABASE_URL}" }
    }
  }
}
```

Project-scoped servers still need to be trusted before they start. In a
headless session that means setting `enableAllProjectMcpServers: true` in the
repo's `.claude/settings.json`, since the interactive approval never arrives.

**Reachability is the real constraint**, not the config. The server runs
wherever the client runs, and it connects to Postgres directly — there is no
hosted component in between. A cloud sandbox can therefore only diagnose a
database inside that sandbox: the [demo stack](examples/demo), or a dev stack
the session brought up itself. A production queue bound to loopback on your own
host is not reachable from a sandbox at all, and exposing it to make it
reachable is the wrong trade.

Diagnose production from a client on a machine that already has a route to it —
your laptop, over an SSH tunnel:

```bash
ssh -N -L 5432:127.0.0.1:5432 prod-host
```

and point `QUEUE_DOCTOR_DATABASE_URL` at `127.0.0.1:5432`. The tunnel is the
access grant, it lasts exactly as long as the terminal stays open, and the
credentials never leave your machine.

## Tools

| Tool | Answers |
|:---|:---|
| `diagnose` | **Start here.** Runs the whole rule catalog, returns ranked findings with evidence and recovery steps |
| `queue_overview` | Per-queue counts by state, stuck jobs, and each queue's expiry/retention/retry config |
| `failed_jobs` | Failures in a window with error messages, plus a per-queue error-frequency breakdown |
| `stuck_jobs` | Jobs active past a threshold, with age, expiry, and heartbeat staleness |
| `missed_schedules` | Cron queues whose latest firing is older than their expression implies |
| `schedule_status` | Every registered schedule with cron, timezone, last firing, and next expected |
| `job_detail` | One job's full record: state, timings, retries, payload, output |
| `server_info` | Connectivity, detected schema, matched dialect, and reduced capabilities |

Schedule expectations are derived from pg-boss's own `schedule` table by parsing
each cron expression, so **the common case needs no configuration**. The health
check this was extracted from carried a hand-maintained list of expected jobs
that silently stopped covering whatever nobody remembered to add.

## The rule catalog

| Rule | Fires when | Motivating incident |
|:---|:---|:---|
| `retry-storm` | Many failures, densely packed, dominated by one error | A daily API quota tipped over and 875 corpus-fill jobs failed in one night. The count suggested 875 problems; the shape showed one |
| `expiry-overrun` | Failure durations cluster at the queue's expiry | A full-corpus sweep couldn't finish inside a 30-minute expiry once upstream throttling slowed it. It reported as a job failure nightly; the fix was an internal wall-clock budget |
| `stuck-jobs` | Jobs active far too long, or heartbeats stopped | A worker killed without graceful shutdown leaves rows active until maintenance reclaims them |
| `missed-schedule` | Latest firing predates the last expected tick | Distinguishes "never fired" (scheduler never booted) from "stopped firing" |
| `duplicate-registration` | A cron queue enqueued twice for one tick | An instrumentation hook invoked job registration twice per process, so every cron ran double for weeks |
| `retention-window` | Failed-row count disagrees with the windowed count | A health email stayed yellow for days after the bug was fixed, counting rows that failed days earlier |
| `dead-queue` | Registered long ago, unscheduled, holds nothing | A producer that stopped, or a registration dropped in a refactor |

Failures are classified (`rate_limit`, `transient_transport`, `auth`,
`not_found`) because the class changes the advice: the right response to a storm
of 429s is close to the opposite of the right response to connection resets.

## Backends

| Backend | Support | Verified against |
|:---|:---|:---|
| pg-boss v11+ | Full | 11.1.2 (schema 26), 12.27.0 (schema 37) |
| pg-boss v10 | Recognised, refused — see below | 10.4.2 (schema 24) |
| pg-boss v9 and earlier | Recognised, refused | — |
| graphile-worker 0.17 | Partial, capability-declared | 0.17.3 |

Select with `QUEUE_DOCTOR_BACKEND=pgboss` (default) or `graphile`; the schema
default follows the backend.

### Capabilities, not zeros

Backends don't just name columns differently — they model work differently.
graphile-worker **deletes a job when it succeeds**, has no per-job expiry, no
worker heartbeats, and keeps cron expressions in a file rather than the
database. So "how many completed in the last day" has no answer there at any
price.

Every backend therefore declares what it can answer, and rules that depend on
missing data **stay silent** rather than reporting a zero — a zero reads like a
measurement.

| Rule | pg-boss v11+ | graphile-worker |
|:---|:---:|:---:|
| `retry-storm` | ✅ | ✅ |
| `stuck-jobs` | ✅ (with heartbeats) | ✅ (age only) |
| `expiry-overrun` | ✅ | — no expiry exists |
| `missed-schedule` | ✅ | — cron lives in a file |
| `duplicate-registration` | ✅ | — no firing history |
| `retention-window` | ✅ | — nothing is retained |
| `dead-queue` | ✅ | — no queue registry |

`server_info` reports the capability set and spells out each limitation.

## Versioned against pg-boss

pg-boss's tables are not a stable API. Across versions it has renamed every
timestamp column (`createdon` → `created_on`), dropped a whole table (`archive`,
removed in v11), changed a duration from an interval to an integer (`expire_in`
→ `expire_seconds`), partitioned the job table, and added columns
(`heartbeat_on`) that materially change what can be diagnosed.

A tool that hard-codes one shape breaks on the next upgrade — silently, if it is
unlucky. That is exactly how the health check this is extracted from spent weeks
emitting a confident, wrong "missed schedules" warning that was really SQLSTATE
42P01 after `pgboss.archive` disappeared.

So schema knowledge lives in one file, [`src/pgboss/dialect.ts`](src/pgboss/dialect.ts), as data:

- **Every relation and column name is declared in a dialect.** Query builders
  emit identifiers from it, so supporting a new pg-boss layout is an edit to
  that file — no SQL elsewhere mentions a pg-boss table by name.
- **Dialects are matched on observed shape, not on a version number.** pg-boss's
  release→schema-version mapping is not published as a contract, and a guessed
  mapping would reintroduce the very failure this guards against. The version
  integer is read, reported, and used to say *"this is newer than anything we
  have verified"* — but it never decides which SQL runs.
- **Optional columns are feature-detected.** No `heartbeat_on`? Stuck-job
  detection degrades to age-based and says so, instead of failing.
- **Unknown layouts are refused, by name.** A pre-v10 schema is recognised
  specifically and rejected with the reason, because diagnosing it against
  modern queries would silently miss every archived job. A wrong diagnosis is
  worse than a refusal.

`server_info` reports the matched dialect, the schema version, whether that
version has been *verified* against real pg-boss, and any reduced capabilities.

This is not a theoretical concern — it has already caught a real bug. The
dialect originally claimed a **v10** floor, on the belief that v10 removed the
`archive` table. Booting pg-boss 10.4.2 showed the archive table still present
and expiry still an `expire_in` interval, so the dialect was rejecting v10
outright and matching nothing at all for it. The real floor is **v11**, and
v10 now has its own dialect: recognised, and refused by name, because reading
the job table alone on v10 silently misses everything already archived.

CI keeps this honest. The integration suite boots pg-boss 10, 11 and 12 into
separate schemas and asserts that the observed schema version appears in the
dialect's verified list — so a future pg-boss that changes the schema fails
loudly rather than running unverified SQL.

## Read-only, by construction

Every query runs inside a `BEGIN READ ONLY` transaction with a
`statement_timeout` and a row cap, and is always rolled back. Recovery actions
are *recommended*, with exact commands — never executed. A confused agent cannot
purge your queue, because the database itself refuses the write.

Three independent guarantees, because the failure being guarded against is
writing to someone's production queue:

1. `BEGIN READ ONLY` on every transaction
2. `default_transaction_read_only=on` at connection level
3. The docs tell you to connect as a least-privilege role — the only guarantee
   that does not depend on this code being correct

Timeouts bind as parameters via `set_config(..., is_local => true)` rather than
being interpolated into SQL. The schema name — the one identifier that cannot be
a bind parameter — is validated against an identifier grammar and quoted.

## Log correlation (optional)

Queue state says *that* a job failed; application logs usually say *why*. Point
the server at a log backend and findings quote the lines behind a failure.

```bash
QUEUE_DOCTOR_AXIOM_TOKEN=xapt-...      # read-capable PAT
QUEUE_DOCTOR_AXIOM_DATASET=app-prod
QUEUE_DOCTOR_AXIOM_ORG_ID=your-org
QUEUE_DOCTOR_AXIOM_QUEUE_FIELD=job     # field carrying the queue name
```

Deliberately optional, and deliberately unable to break anything: a dead log
backend never turns a working diagnosis into a failed one, and "we did not
look" stays distinguishable from "we looked and found nothing" — otherwise an
absent log line reads as evidence of absence. Half-configured settings are a
startup error rather than a silent downgrade.

## Reaching a database you cannot connect to

Production queues are often the ones you most want diagnosed and least able to
reach: Postgres bound to loopback, no port forwarding, only the application in
front of it exposed. Opening the database to the network so a diagnostic can
connect is a poor trade — the grant is permanent and far wider than the need.

So the server can run its SQL over HTTPS against a read-only SQL endpoint
instead:

```bash
QUEUE_DOCTOR_HTTP_SQL_URL=https://your-app.example/api/admin/sql
QUEUE_DOCTOR_HTTP_SQL_TOKEN=...
```

Set these and no connection string is needed; set both and the HTTP transport
wins, so an ambient `DATABASE_URL` cannot quietly become the target. The
endpoint must accept `{"query": "...", "params": [...]}` and answer with
`{"rows": [...], "truncated": bool}`. Reference implementation:
[showbook's `/api/admin/sql`](https://github.com/ethanasm/showbook/blob/main/apps/web/app/api/admin/sql/route.ts).

The safety properties move to the far end, which is an improvement rather than
a compromise. The endpoint opens its own read-only transaction, enforces its
own timeout and row cap, can rate-limit, can log every query, and can be backed
by a role with narrower grants than the application's own — none of which
depend on this client being correct. What changes for you: the endpoint's
`statement_timeout` and row cap win over `QUEUE_DOCTOR_STATEMENT_TIMEOUT_MS`
and `QUEUE_DOCTOR_MAX_ROWS`, a truncating endpoint is reported as `truncated`
rather than silently short, and one `diagnose` costs roughly a dozen requests
against whatever rate limit is in force.

Bind parameters are required, not optional: a client forced to inline its own
literals to reach a read-only endpoint would be building an injection sink to
get there.

## Configuration

| Variable | Default | Purpose |
|:---|:---|:---|
| `QUEUE_DOCTOR_DATABASE_URL` / `DATABASE_URL` | — | **Required**, unless the HTTP transport is used. Postgres connection string |
| `QUEUE_DOCTOR_HTTP_SQL_URL` | — | Read-only SQL endpoint to query through instead of connecting |
| `QUEUE_DOCTOR_HTTP_SQL_TOKEN` | — | Bearer token for that endpoint |
| `QUEUE_DOCTOR_BACKEND` | `pgboss` | `pgboss` or `graphile` |
| `QUEUE_DOCTOR_SCHEMA` | per backend | Schema the queue was installed into |
| `QUEUE_DOCTOR_STATEMENT_TIMEOUT_MS` | `5000` | Per-query timeout (100–120000) |
| `QUEUE_DOCTOR_MAX_ROWS` | `500` | Row cap per query (1–10000) |
| `QUEUE_DOCTOR_LOG_LEVEL` | `info` | `debug`/`info`/`warn`/`error`/`silent` (stderr) |
| `QUEUE_DOCTOR_THRESHOLDS` | — | JSON object overriding rule thresholds (see below) |

See [`.env.example`](.env.example). Requires Node.js ≥ 20.11.

## Tuning the rules

The thresholds are tuned to the queue these rules were extracted from. That is
a defensible starting point and a poor universal answer: a queue that
legitimately fails fifty times an hour against a flaky upstream does not have a
retry storm, and being told it does every time teaches you to stop reading.

Override any of them with a JSON object — only the keys you set change:

```bash
QUEUE_DOCTOR_THRESHOLDS='{"stormMinFailures":50,"idleQueueSeconds":2592000}'
```

| Key | Default | Governs |
|:---|---:|:---|
| `stormMinFailures` | `20` | Failures before a burst counts as a storm |
| `stormDominantShare` | `0.5` | Share one error must hold to be called dominant |
| `stormPeakPerMinute` | `5` | Failures in a minute that mark a burst, not a trickle |
| `stormCriticalFailures` | `100` | Above this a storm is critical, not a warning |
| `expiryProximity` | `0.95` | Fraction of expiry that looks killed rather than failed |
| `expiryMinJobs` | `3` | Jobs at expiry before it is a pattern |
| `heartbeatMissedMultiplier` | `3` | Missed heartbeats before a worker counts as gone |
| `missedScheduleCriticalSeconds` | `86400` | Lateness beyond which a miss is critical |
| `retentionMismatchMin` | `5` | Extra stale failed rows before flagging retention |
| `duplicateTickMin` | `2` | Ticks with duplicate firings before suspecting double registration |
| `idleQueueSeconds` | `604800` | Age at which an empty queue is worth mentioning |
| `correlatedLogSample` | `5` | Log lines attached to a finding as evidence |

An unknown key is a **startup error**, not a warning — a typo that silently
leaves the default in place is the failure this prevents. `server_info` reports
the effective values and which ones you set, so you can confirm an override
took.

## Publishing to the MCP registry

`server.json` is the registry manifest. Its version and the npm version it
points at are both synced by `npm version` (see `scripts/sync-version.mjs`), and
a test fails if they drift — a registry entry naming a version that is not on
npm sends clients to a 404, which is worse than a stale entry.

Ownership is proved by the `mcpName` field in the **published** `package.json`,
so npm must be published first:

```bash
npm version patch          # syncs src/version.ts and server.json
npm publish                # the registry reads mcpName off this
mcp-publisher login github # device auth as the io.github.<user> namespace owner
mcp-publisher publish
```

## What this is not

- **Not a queue browser.** To page through jobs, `psql` is better.
- **Not a dashboard.** This is agent infrastructure; your MCP client is the UI.
- **Not a Redis queue tool.** Both supported backends are Postgres-native,
  which is what makes the read-only transaction guarantee possible at all.
  BullMQ and Celery would need a different safety story.
- **Not a writer.** It will not retry, cancel, or purge anything.

## Roadmap

- [x] Read-only database layer, schema probe, CI
- [x] The read-only tool surface
- [x] The diagnosis engine
- [x] A `docker compose up` demo with a chaos worker
- [x] Integration tests against real pg-boss 10/11/12 in CI
- [x] Log correlation, so findings can cite application logs
- [x] A second adapter (graphile-worker)
- [ ] Opt-in write tools (`retry_job`, `cancel_job`) behind an explicit flag
- [ ] Configurable rule thresholds

## Development

```bash
npm install
npm run verify     # lint + typecheck + test + build
```

The unit suite drives the database layer through a scripted fake client and the
rules through fixtures reconstructing each motivating incident, so `npm test`
runs with no Postgres, no containers, and no network.

The integration suite boots real pg-boss (v10, v11, v12) and real
graphile-worker against a live Postgres:

```bash
docker run -d -p 55432:5432 -e POSTGRES_USER=qd -e POSTGRES_PASSWORD=qd \
  -e POSTGRES_DB=qd postgres:16-alpine
QUEUE_DOCTOR_TEST_DATABASE_URL=postgres://qd:qd@127.0.0.1:55432/qd \
  npm run test:integration
```

It skips itself when that variable is unset, so a contributor without Postgres
is never blocked. For a hands-on run, use [`examples/demo`](examples/demo).

## License

MIT