# Testing This document is the source of truth for how we test in this repo. Read it before adding a test. AI agents: this overrides any default testing instinct. ## The one rule: don't mock your own code Every other rule here is a consequence of this one. A test earns its place when it is likely to catch a real bug at a price worth paying. Mocking a dependency you own destroys the first half of that trade: you stop testing the contract and start testing a fiction you typed yourself. The test then stays green precisely when the real code breaks — the worst possible outcome, because it costs maintenance while catching nothing. So the enemy is **not** infrastructure. Postgres, NATS, and Better Auth are real dependencies; testing against them is good. The enemy is the *mock of your own contract* — a stubbed `StudioContext`, a fake `storage` adapter, a hand-written `fetch` that returns a response shape you invented. If your test asserts against something you made up, delete it and test the real thing. Concretely: a test that mocks all of `storage` and then asserts "the handler called `storage.claimRunStart`" is asserting that the code calls the function the code calls. It is tautological. It will not catch the broken SQL inside `claimRunStart`, which is the only bug that handler can actually have. ## Categorize by what's under test, and whether its dependencies are real There is no "unit vs. integration vs. e2e" taste call. The category falls out of two mechanical questions: *what is the unit under test*, and *are its dependencies real or mocked*. | Under test | Dependencies | Tier | Catches | | --- | --- | --- | --- | | A pure function | none | **Unit** (`bun test`) | logic bugs | | One layer (e.g. a storage adapter) | **real** (real Postgres) | **Storage-integration** (`bun test`, real DB) | SQL / contract / serialization bugs | | The whole system, through the front door | real, via HTTP/UI | **E2E** (Playwright) | flow / wiring bugs | | One layer | **mocked** | ⛔ the bad zone | nothing real — tautologies | The bottom row is the only forbidden one. If you find yourself mocking the dependencies of a single layer, you are in the bad zone: either drop the mocks and test that layer against its real dependency (move up a row), or test the behavior through the front door (move to e2e). Never ship the mocked middle. ### Unit (`bun test`) — pure logic only - **No mocks. No I/O.** No DB, no network, no filesystem (beyond reading test fixtures). - **What belongs here:** schema validators, encryption, JWT/HMAC, PKCE, parsers, sanitizers, code generators, pure utility functions, migration SQL *string* correctness, pure state-transition functions. - **Where:** co-located `*.test.ts` next to the source file. If you reach for a mock to make a "unit" test compile, you picked the wrong tier. Move up the table. #### Narrow exception: state-machine unit tests with one boundary mock Some modules are pure-ish state machines whose contract is *what error class do we throw and how fast*, not what HTTP status comes out the other end. Forcing those into Playwright loses precision (the proxy translates errors, the assertions weaken to "some 5xx") without buying realism. A unit test may mock **exactly one module function** at the boundary if all of these hold: - The SUT is internal logic (no HTTP route exists that exercises it cleanly). - The "downstream" used in the success path is real, just in-process (e.g. the MCP SDK's bridge transport — a real client and server connected without going over the network). - The assertion is on a JS class, timing, or in-memory state — not on a wire response. Example: `apps/api/src/mcp-clients/lazy-client.test.ts` mocks `./client.clientFromConnection` to inject failures, uses a real MCP server via in-process bridge transport for the success path, and asserts `CircuitOpenError` + fail-fast timing. Migrating it to Playwright would mean: hit `/api/:org/mcp/:connectionId` repeatedly while toggling a test MCP server's failure injection, observing the proxy's HTTP status code (no `CircuitOpenError` class), losing timing precision under workers-in-parallel, and risking cross-test pollution of the module-global circuit map. This is a genuine exception, not a loophole. It is for one boundary mock guarding a class/timing assertion — *not* for "I didn't want to set up a real fixture." Mocking your whole `storage` interface never qualifies. ### Storage-integration (`bun test` against real Postgres) — one layer, real DB This tier exists because some contracts can only be tested against a real SQL engine, and routing them through the full stack would make the test assert the wrong thing. - **Real Postgres. Zero mocks.** Exercise the storage adapter directly: call `SqlThreadStorage.saveMessages(...)`, then assert the rows. - **What belongs here:** `ON CONFLICT` / upsert semantics, `RETURNING`, FK constraints, JSON column (de)serialization, pagination edges, advisory locks, `LISTEN`/`NOTIFY`, CAS update logic, migration *behavior*. Anything whose correctness lives in the SQL the adapter emits. - **No HTTP, no auth, no browser, no NATS.** If the test needs those, it's e2e. This tier is the storage layer and its database, nothing else. - **Where:** co-located `*.integration.test.ts` next to the source. The filename convention auto-routes it to the `storage-integration` CI workflow (real `postgres:16`), separate from the fast unit job. Why not just make these e2e? Because to test "`saveMessages` dedupes on message id" through Playwright you'd boot the server, sign up via Better Auth, find an HTTP route that happens to trigger a duplicate-id upsert, drive it, and then fish in the DB to assert. The assertion drifts three layers from the thing under test, and a route may not even exist for every storage behavior (FK violation, null `parts`, pagination off-by-one). Direct + real Postgres is the *most* honest test of SQL, not a compromise. (It's also faster — no browser — but that's a bonus, not the reason.) This is **not** a license to test route handlers with a mocked DB. Real DB, single layer, no mocks. The moment you mock `storage`, you've left this tier for the bad zone. ### E2E (Playwright) — the whole system, through the front door - **Real Postgres, real NATS, real Better Auth, real HTTP, real browser.** - The only acceptable "mock" is at the *infrastructure edge* — e.g., a flaky test MCP server you bring up as a real process, not a mocked client inside the app. - **What belongs here:** anything that crosses the auth, HTTP-route, MCP, NATS, or UI boundary; any flow whose bugs are wiring bugs (a handler orchestrating real storage + SSE under real concurrency); anything that would otherwise force you to mock your own code to test it. - **Where:** `packages/e2e/tests/*.spec.ts`. ## Writing an e2e spec ```ts import { test, expect } from "../fixtures/test"; test("creates a connection for the authed user", async ({ authedPage }) => { const { page, orgSlug } = authedPage; await page.goto(`/${orgSlug}/settings/connections`); // ... }); ``` - Use `import { test } from "../fixtures/test"` — that's the extended Playwright `test` with shared fixtures. - Default to the **`authedPage`** fixture. It signs up a fresh user via the Better Auth API, sets cookies on the browser context, and gives you `{ page, user, orgSlug }`. Each test gets a unique user + org. - Use **API factories** from `packages/e2e/fixtures/factories.ts` for setup state (create org, create connection, invite member). Don't `INSERT` directly when an API path exists. - Use **`connectDevDb()`** (from `packages/e2e/fixtures/db.ts`) for **assertions** about DB state, or for seeding state that has no API (e.g., `organization_domains` rows). - Use **page objects** from `packages/e2e/pages/` for recurring screens. Promote an inline helper to a page object on the *second* use, not the first. ### Exception: signup and onboarding specs Tests that *are* the signup/onboarding flow (e.g., `org-access-gate.spec.ts`, `auto-domain-join-multi-org.spec.ts`, future signup-flow specs) MUST drive the real UI via the `signUp()` helper in `fixtures/auth.ts`. Don't shortcut the very flow you're testing. Everything else uses `authedPage`. ## Anti-patterns Don't do any of these: - **Mocking your own code to make a test green.** The headline rule. Stubbing `storage`, `StudioContext`, `auth.api.getSession`, `AgentSandboxProvider`, `clientFromConnection`, or `global.fetch` means the test belongs in storage-integration (drop the mocks, use real Postgres) or e2e (go through the front door). (An *inert* collaborator a code path never invokes — e.g. the unused `storage` a pure in-memory state-machine test must hand to a constructor — is not this: nothing the assertion depends on is faked. Keep it plain and unexercised, not a `mock()` with canned return values.) - **Asserting "the handler called the function."** If the assertion is `expect(storage.foo).toHaveBeenCalled()` against a mocked `storage.foo`, the test is tautological. Test the *effect* against the real dependency instead. - **A route/handler test with a mocked DB.** The bad-zone middle. Move to e2e. - **A CLI/client test with a hand-written `fetch` mock** that returns an invented server-response shape. You're asserting against your own fiction; it won't break when the real server changes. Stand the real server up as a fixture and round-trip against it. - **Direct SQL inserts in a Playwright spec** when an API path exists. Go through the same code paths a real client would. (Seeding state that has *no* API is fine via `connectDevDb()`.) - **Shared global state across tests.** No `beforeAll` that seeds rows other tests depend on. - **Hard-coded slugs/emails.** They collide under parallel workers. Use randomized values. - **`test.afterAll` SQL cleanup as the default.** Rely on per-test unique orgs/users so cleanup isn't needed. - **`mock.module` without a way back.** Bun keeps module mocks alive for the whole shard, and a top-level `afterAll` runs at the end of the PROCESS, not the end of the file — so a module mock at file scope leaks into every later file, and there is no hook that can undo it. Three separate failures came from this: a settings stub left two unrelated files failing on `"http://localhost:undefined"`, and it also made the two flags the plan feature gate reads `undefined`, which the gate treats as "no answer" and OPENS — so a test written for those gates would have passed whether or not the gate worked. `apps/api/src/settings/settings-not-mocked.test.ts` is a canary for exactly that. Install real state through the real accessor (`setGlobalSettings(resolveConfig(...).settings)`), or expose a narrow named test hook on the module that owns the state (`__armRefreshBackoffForTesting`), rather than replacing the module. - **Flags `bun test` does not have.** It silently swallows unknown ones — `bun test --definitely-not-a-flag ` exits 0 — so a flag that does nothing looks like a flag that works. Check `bun test --help` **of the pinned CI Bun** (`.github/actions/setup-bun`), not whatever is on your machine: `--parallel` reads as a no-op on an older local Bun and is load-bearing on the pinned one, where it runs each file in its own worker process and implies `--isolate`. Dropping it from the root `test` script cost 168 cross-file failures — leaked happy-dom globals, a readonly `window.localStorage`, leaked `mock.module` stubs — none of which reproduce locally. ## When developing a feature Decision tree: 1. Is the feature **pure logic** (parser, hash, Zod schema, pure transform, pure state transition)? → Co-located **unit** test, no mocks. 2. Is the unit under test **a storage adapter**, and does its correctness live in the SQL it emits? → Co-located **`*.integration.test.ts`**, real Postgres, no mocks. 3. Does the feature **cross auth, an HTTP route, the MCP boundary, NATS or the UI** — or would testing it otherwise force you to mock your own code? → **Playwright** spec. 4. Did you just reach for a mock of something you own to make a test compile? → You're in the bad zone. Go back to 2 or 3. ## Specialized suites (don't write these casually) - **Resilience tests** — `tests/resilience/scenarios/`. Docker Compose + Toxiproxy chaos. For testing behavior under infrastructure failure (DB outage, NATS disconnect, MCP latency). - **Multi-pod tests** — `tests/multi-pod/scenarios/`. 3-pod Studio cluster, shared Postgres + NATS. For testing cross-pod behavior (session rehoming, API key sharing, DBOS replay). These are not part of the default test loop. They run on dedicated CI workflows. Only add a scenario here if the behavior cannot be expressed in a single-pod Playwright spec. ## Running tests ```bash # Unit suite, with isolation and infrastructure suites excluded bun run test bun test apps/api/src/encryption # subset # A single storage-integration file needs a real DATABASE_URL pointing at Postgres # (CI provisions postgres:16; locally point it at your dev DB). bun test apps/api/src/storage/threads.integration.test.ts # E2E bun run --cwd=packages/e2e test:e2e # all e2e bun run --cwd=packages/e2e test:e2e:ui # interactive UI bun run --cwd=packages/e2e test:e2e some-spec # filter by name # Resilience / multi-pod (requires Docker) ./tests/resilience/run.sh ``` CI splits the three tiers by filename convention: - `*.test.ts` → unit job (`.github/workflows/test.yml`), must stay fast. - `*.integration.test.ts` → storage-integration job (`.github/workflows/storage-integration.yml`), real Postgres. - `*.e2e.test.ts` → reserved for the sandbox-daemon workflow. Adding a test is just choosing the right filename — it auto-routes. ## CI expectations - **`unit` job** must stay fast (target < 2 min). If a "unit" test starts needing infra, it's the wrong tier — move it. - **`storage-integration` job** runs each `*.integration.test.ts` against real `postgres:16`. No HTTP/auth/browser here; those belong in e2e. - **`e2e` job** runs parallel against real services. Target < 10 min wall time; shard if it grows. - **Resilience / multi-pod** run on dedicated workflows (`resilience.yml`, multi-pod equivalent). Not required on every PR. ## Examples - Good unit test: [`apps/api/src/encryption/credential-vault.test.ts`](apps/api/src/encryption/credential-vault.test.ts) - Good storage-integration test: [`apps/api/src/storage/threads.integration.test.ts`](apps/api/src/storage/threads.integration.test.ts) - Good e2e spec: [`packages/e2e/tests/connection-create.spec.ts`](packages/e2e/tests/connection-create.spec.ts)