# End-to-end tests in CI

The Playwright suite (`e2e/`) runs on every pull request through
`.github/workflows/e2e.yml`. This document covers the one repository secret it
needs and the **branch-protection step that turns a red run into a blocked
merge** — the workflow reports pass/fail on its own, but only branch protection
makes that check *required*.

## What the workflow does

On every `pull_request` targeting `main` (or `upstream-main`, the fork mirror),
on manual `workflow_dispatch`, and **nightly at 03:47 UTC** (a `schedule` cron,
so drift that lands between PRs — dependency bumps, Supabase CLI changes,
selector-breaking UI tweaks — is caught within a day), the `e2e / playwright`
job:

1. installs the root (Playwright), `backend/`, and `frontend/` dependencies;
2. boots **MinIO** (S3-compatible object storage — several specs upload documents);
3. boots **local Supabase** (Auth + Postgres) via the Supabase CLI and loads the
   current fresh-install shape from `backend/schema.sql`. It intentionally does
   not replay historical migrations on top: doing so can replace current
   functions with older definitions. The separate schema-drift workflow proves
   that the supported upgrade path (its pinned baseline plus later migrations)
   converges with this fresh-install path;
4. writes `backend/.env` and `frontend/.env.local` from the live Supabase values;
5. builds the backend and runs the pinned `sync:workflows` release job, matching
   production ordering so the default and add-on catalog exists before startup;
6. **builds** the web app (`next build`) and serves it with `next start` — a
   production build, not `next dev`, so there is no on-demand compilation (which
   makes first-hit page loads slow enough to time out specs) and no dev
   hydration-error overlay (whose injected DOM pollutes text locators). Starts the
   backend API (`:3001`) and the web server (`:3000`) and waits for both healthy;
7. runs `npx playwright test` and uploads the HTML report + traces as an artifact
   (`playwright-report`) on pass, fail, or timeout.

`e2e/auth.setup.ts` bootstraps the shared test user (`e2e@mike.local`) against
the local Supabase admin API, so no login secret is needed — the credentials
baked into that file are the single source of truth.

A keyless run is expected to end **27 passed / 4 skipped / 0 failed** — the
suite currently has 31 tests, 4 of them LLM-gated (see "Confirm the specs ran"
below). Use the Playwright summary as the source of truth if tests are added or
removed.

## Accessibility scans

`e2e/accessibility.spec.ts` runs an [axe-core](https://github.com/dequelabs/axe-core)
scan (via `@axe-core/playwright`) over the core pages: `/login` (pre-auth),
`/assistant`, `/projects`, and `/tabular-reviews`. The policy is two-tier:
**`critical`-impact violations fail the build**; `serious`-impact violations are
printed to the test output but do not fail — enforce at critical first, then
ratchet `serious` into the failing tier (`BLOCKING_IMPACTS` in the spec) once
that backlog is cleared. The scans need no LLM key and run on every trigger.

## Failure artifacts

Playwright retries failed specs up to twice on CI and records a **trace** on the
first retry (`retries` / `trace: "on-first-retry"` in `playwright.config.ts`).
On pass, fail, or timeout, the job uploads `playwright-report/` and
`test-results/` as the **`playwright-report`** artifact (14-day retention): from
the failed run's page in the Actions tab, download it, then
`npx playwright show-report playwright-report` locally to see per-spec results,
screenshots, and step-by-step traces of what the browser did.

## Optional secret (fuller coverage)

| Secret | What it unlocks | Without it |
|---|---|---|
| `ANTHROPIC_API_KEY` | The 4 LLM-dependent specs (chat rename/delete/submit, critical-path "ask a question") send a message and assert a **streamed** answer. With the key set they run and are enforced. | Those 4 specs **skip** (see `e2e/llm.ts`) instead of hanging, so the run is still green on the other 27 specs. |

The suite is green **without** any secret — the LLM specs skip themselves via
`test.skip(!process.env.ANTHROPIC_API_KEY, …)`, which keeps keyless runs (local,
and fork PRs with no secret access) green and fast. Mike supports keyless local
models through Ollama, but this CI job does not provision an Ollama server or
pull a model. Without the Anthropic secret, the four live-response tests
therefore have no model available in the CI environment and must skip. The
auto title-generation call is not the reason for the gate; failures there are
already treated as best-effort.

## Enable the LLM specs

### 1. Add the repository secret

UI path:

1. Open the repository on GitHub → **Settings**.
2. In the left sidebar: **Secrets and variables → Actions**.
3. On the **Secrets** tab, click **New repository secret**.
4. **Name:** `ANTHROPIC_API_KEY` — exactly this name; both the workflow env and
   `e2e/llm.ts` read it. **Secret:** an Anthropic API key (`sk-ant-…`) from
   <https://console.anthropic.com/settings/keys>.
5. Click **Add secret**.

CLI equivalent (repo admin):

```bash
gh secret set ANTHROPIC_API_KEY --repo Open-Legal-Products/mike
# paste the key at the prompt (or pipe it: --body "$ANTHROPIC_API_KEY")
```

### 2. The fork-PR caveat

On `pull_request` events from **forks**, GitHub withholds repository secrets, so
fork PRs — most external contributions — still run keyless and skip the 4 specs.
That is by design and keeps those runs green. Runs that actually receive the
secret and exercise the specs are:

- PRs from branches pushed to this repository (maintainer branches), and
- manual runs: **Actions → e2e → Run workflow** (`workflow_dispatch`) on any
  branch.

So after adding the secret, the quickest way to see the specs run is a
`workflow_dispatch` run from the Actions tab.

### 3. Expected cost per run

A handful of short completions: one streamed chat answer per LLM spec plus a few
small title generations (`claude-haiku-4-5`, 64-token cap). On the order of a
few cents per run — negligible next to the CI minutes.

### 4. Confirm the specs ran (not skipped)

Open the **Run Playwright** step in the Actions log:

- **Keyless run:** the summary ends with `4 skipped` / `27 passed`, and each
  skipped spec carries the reason
  `requires a model key — set the ANTHROPIC_API_KEY secret to run LLM-dependent specs`.
- **With the secret:** the summary shows `31 passed` and **no `skipped` line**;
  searching the log for `requires a model key` finds nothing.

The uploaded `playwright-report` artifact shows the same per-spec statuses.

### Model selection

When the secret is present, the shared `selectClaudeModel` helper selects a
supported Anthropic model before each gated test submits. The response checks
assert a nonempty streamed assistant answer rather than provider-specific text.
Keep that helper synchronized with the current model catalog when model ids or
display names change.

## Make it merge-blocking

The workflow failing is not enough on its own — GitHub will still allow the merge
unless the check is **required**. Enable branch protection once you have seen the
suite go green a few times (it is environment-sensitive by nature):

1. **Settings → Branches → Add branch protection rule** (or edit the rule for
   `main`).
2. Enable **Require status checks to pass before merging**.
3. Enable **Require branches to be up to date before merging**.
4. In the checks search box add **`e2e / playwright`** (the job appears in the
   list after it has run at least once on a PR).
5. Recommended alongside it: the unit/build check `backend` and the `license/cla`
   check.
6. Save. From now on a red e2e run blocks the **Merge** button.

Equivalent via the GitHub CLI (repo admin token required):

```bash
gh api -X PUT repos/OWNER/REPO/branches/main/protection \
  -H "Accept: application/vnd.github+json" \
  -f 'required_status_checks[strict]=true' \
  -f 'required_status_checks[contexts][]=e2e / playwright' \
  -f 'enforce_admins=true' \
  -f 'required_pull_request_reviews[required_approving_review_count]=1' \
  -f 'restrictions='
```

## Running the suite locally

Locally, `playwright.config.ts` starts the backend and web dev servers for you
(`webServer` is only disabled when `CI=true`), so a full local stack plus:

```bash
npm ci
npx playwright install --with-deps chromium
npm run test:e2e            # or test:e2e:ui / test:e2e:headed
```

`e2e/auth.setup.ts` reads `SUPABASE_URL` / `SUPABASE_SECRET_KEY` from the
environment or `backend/.env`, so a running local Supabase + a populated
`backend/.env` is all the setup needs.