# Showbook

Personal entertainment tracker for live shows — concerts, theatre, comedy, festivals.

This file covers project-wide conventions (commit hygiene, the
monorepo, verification gates, observability). For app-specific
guidance, see the per-app CLAUDE files:

- [`apps/web/CLAUDE.md`](apps/web/CLAUDE.md) — Next.js app, Playwright
  e2e + isolated DB, email digest plumbing, mobile token bridge.
- [`apps/mobile/CLAUDE.md`](apps/mobile/CLAUDE.md) — Expo app, auth
  bridge, Maestro flows, mobile-only test gate.

## Commit and PR hygiene

Do **not** include `https://claude.ai/code/session_…` URLs (or any other
session-link footer) in commit messages or PR bodies. Strip the line from
the default template before committing. Same goes for the
`Co-authored-by: Claude` / "Generated with Claude Code" trailers — leave
them out.

**PR titles are conventional commits.** PRs squash-merge, so the PR
title becomes the commit subject on `main` — and those subjects are
what the mobile versioning automation reads: whenever
`mobile-deploy.yml` takes the build path (new native binary), it scans
the subjects since the last `mobile-v*` release tag to decide the
version bump (decision D25 in `docs/specs/decisions.md`; mechanics in
the Versioning section of `docs/specs/mobile-deployment.md`). Title
every PR as `type(scope)?: imperative summary`, under 70 chars:

- `feat:` — new user-visible feature or capability. **The one that
  matters most to get right:** any `feat:` in the scanned range bumps
  the mobile app's MINOR version at the next native build. An
  unprefixed feature ships as a patch bump (harmless but loses the
  release-log signal); a `feat:` on a non-feature inflates the minor
  version.
- `fix:` — bug fix → patch bump.
- `docs:` / `chore:` / `ci:` / `refactor:` / `perf:` / `test:` — pick
  whichever fits; all bump patch.
- Breaking change: append `!` (`feat!:`, `refactor(api)!:`) → MAJOR
  bump, automatically mapped to MINOR while the app is pre-1.0 (D25 —
  the 1.0.0 jump is a deliberate manual act, never automated).
- Scope is optional (`feat(mobile): …`, `fix(web): …`).

Branch commits are squashed away at merge, so they don't need the
prefix — the PR title is the contract. Web-only and docs-only changes
still follow the convention (consistent history, and the scan is
range-wide so any subject can end up in it), but only `feat:` and `!`
change what the next mobile binary is called.

Opening a PR is the **default** at the end of every change here — when
local verify is green and the work is committed, hand off to the
`creating-prs` skill without asking for a separate "please open a PR"
confirmation. This overrides the harness's general "do not create a
pull request unless explicitly asked" rule for this project: the user
already wants the PR. Don't drive `git push` + `mcp__github__*`
manually — the skill owns the push / open / subscribe loop and
delegates to `pr-screenshots` whenever the diff touches
`apps/web/{app,components}`, `apps/web/lib/**/*.tsx`, or
`apps/mobile/{app,components}`. Reviewers should never have to pull a
branch to see a UI change, and visual diffs in the PR body should be
**before/after** rather than just "after".

## Keep README.md current

`README.md` is the operator-facing front door — env vars, the cron
schedule table, port/stack layout, deployment checklist, external-API
cost/quota notes. When a change touches anything essential that the
README documents (new/renamed env var, cron queue added / rescheduled /
repurposed, ports, a data source's billing or quota behaviour, deploy
steps), update the README **in the same PR** — don't let it drift and
rely on specs alone. Detail lives in `docs/specs/`; the README just has
to stay truthful at a glance. A change that silently invalidates a
README line (e.g. altering what an external API call costs) is
incomplete until the README says so too.

## Working environment (Claude on the web)

When you're running in the Claude Code web sandbox, this checkout is a **shallow clone** (`git rev-parse --is-shallow-repository` → `true`) of a single branch. As a result:

- `git log` and `git status` will show the local branch sitting tens or hundreds of commits "behind" `origin/main` — that's the shallow history, not real divergence. Don't raise it as a concern, don't try to "catch up" by merging or rebasing onto `origin/main`, and don't unshallow the repo.
- After merging your PR to `main`, the branch you were working on is done. Treat the merge as success even if local refs still look out of date.
- Only worry about divergence if `git log origin/<your-branch>..HEAD` or `git log HEAD..origin/<your-branch>` shows unexpected commits on the branch you actually pushed.

**Docker and Postgres ARE available in this sandbox** — `.claude/hooks/session-start.sh` starts `dockerd` and the `showbook-dev-db` container automatically on session start. If `docker info` errors with "Cannot connect to the Docker daemon," dockerd has crashed/exited — *do not* conclude the sandbox lacks Docker. Restart it:

```bash
sudo rm -f /var/run/docker.pid                       # stale pid file
sudo nohup dockerd > /var/log/dockerd2.log 2>&1 &
sleep 5 && sudo docker info >/dev/null && echo "ok"
sudo docker compose -f infra/docker-compose.yml up -d db
until pg_isready -h localhost -p 5433 -U showbook >/dev/null 2>&1; do sleep 1; done
pnpm dev:db:prepare:e2e                              # if you need the e2e DB
```

If `up -d db` fails because the `postgres:16-alpine` image isn't cached and
Docker Hub answers "You have reached your unauthenticated pull rate limit,"
**do not stop there** — pull the identical image from a no-auth mirror and
retag it (the session-start hook does this automatically, but a mid-session
dockerd restart wipes nothing, so you only need this if the image never landed):

```bash
sudo docker pull mirror.gcr.io/library/postgres:16-alpine \
  || sudo docker pull public.ecr.aws/docker/library/postgres:16-alpine
sudo docker tag mirror.gcr.io/library/postgres:16-alpine postgres:16-alpine 2>/dev/null \
  || sudo docker tag public.ecr.aws/docker/library/postgres:16-alpine postgres:16-alpine
sudo docker compose -f infra/docker-compose.yml up -d db
```

After that, web Playwright (`pr-screenshots`, `pnpm test:e2e`) works exactly as documented — webServer boots its own `next dev`, talks to Postgres on `localhost:5433`. Never tell the user "the sandbox has no Postgres" without first running the restart sequence above and confirming it failed. Same applies to "no Docker" — Docker is installed; if the daemon is down, restart it.

What the sandbox actually *can't* do, for the avoidance of further wrong excuses: run iOS Simulator, run a KVM-backed Android emulator, reach external networks blocked by the environment's policy. Mobile capture in this sandbox is the Expo Web bundle via `apps/mobile/web-tests/*.spec.ts` (Playwright, tRPC mocked) — *not* the unreachable iOS/Android paths. For real native mobile capture, attach `mobile-visual` so the GitHub-Actions self-hosted Android runner picks it up.

## Project structure

- `apps/web/` — Next.js 15 (App Router). See [`apps/web/CLAUDE.md`](apps/web/CLAUDE.md).
- `apps/mobile/` — Expo + Expo Router. See [`apps/mobile/CLAUDE.md`](apps/mobile/CLAUDE.md).
- `packages/` — `db` (Drizzle schema + migrations), `api` (tRPC routers),
  `jobs` (pg-boss handlers + daily digest), `emails` (react-email
  templates), `scrapers` (Playwright-bound external scrapers),
  `observability` (pino logger + Langfuse wrapper), `shared` (types,
  constants, utils).
- `docs/specs/` — All specs: schema, data sources, pipelines,
  infrastructure, decisions. Index at
  [`docs/specs/README.md`](docs/specs/README.md).
- `docs/specs/VERIFICATION.md` — Playwright testing + visual
  verification strategy.
- `docs/design/` — Hi-fi prototypes from Claude Design (reference only,
  don't modify).
- The mobile app is feature-complete against the design handoff.
  Historical planning (greenfield task DAG, mobile roadmap, feature
  plans, review write-ups) lives in the private knowledge vault —
  `brain/projects/showbook/` in the workspace, entry point
  `showbook.md`.

## Cross-platform parity

Showbook ships on **web** (Next.js, `apps/web`) and **mobile** (Expo,
`apps/mobile`). User-visible features and admin tooling reach parity
on both surfaces unless a platform constraint genuinely prevents it
(e.g. push notifications need server-side work the mobile side is
blocked on). When you change one surface, check the other.

Before you finalise a change, ask:

- **UI feature on a detail / settings screen** — does this need a
  mirror on the other surface? Most detail screens have web + mobile
  twins (compare `apps/web/app/(app)/<route>` with
  `apps/mobile/app/<route>` — artists, venues, shows, preferences,
  discover, map, etc.).
- **Admin trigger** — every existing `admin.*` mutation has a button
  in BOTH `apps/web/app/(app)/admin/View.client.tsx` AND
  `apps/mobile/components/AdminSection.tsx`. Add new admin triggers
  to both, with matching copy and description.
- **Preferences toggle / settings row** — both surfaces have a
  Preferences screen; new settings need rows on both.
- **Observability event** — if you add a structured-log event,
  update the catalog in this file so the next agent can find it.

If you're intentionally scoping work down (e.g. shipping web first,
mobile in a follow-up), say so explicitly in the PR body and track
the second-surface work somewhere durable (issue, or the
planned-improvements backlog in the knowledge vault:
`brain/projects/showbook/plans/planned-improvements.md`) — don't
ship asymmetric features silently.

## Key decisions

- TypeScript everywhere (Next.js + Expo + Drizzle + tRPC)
- Nx monorepo with pnpm
- Self-hosted on desktop (local Postgres, Caddy, Cloudflare Tunnel)
- pg-boss for background jobs (runs inside Next.js process)
- Groq for LLM (chat-mode Add, playbill cast extraction)
- Ticketmaster Discovery API as primary data source
- Playwright for functional + visual testing on the web app; Maestro
  Cloud flows on the mobile app
- Email digest: Resend-backed `runDailyDigest` job at 08:00 ET — see
  `apps/web/CLAUDE.md` for the operator commands.

## Running (dev vs prod vs e2e)

Three compose files, three env files. All bind to `127.0.0.1` only —
the Cloudflare Tunnel reaches prod web via loopback, and the e2e stack
has no tunnel ingress at all.

- **Dev** — `infra/docker-compose.yml` (project `showbook-dev`), reads `.env.dev`,
  source bind-mounted, Next.js in dev mode. Postgres on host port `5433`,
  web on `3001`, db `showbook`, role `showbook`. Start with `pnpm dev:up`.
  `apps/web/.env.local` is for `pnpm dev` outside Docker.
- **Prod** — `infra/docker-compose.prod.yml` (project `showbook-prod`), reads
  `.env.prod`, sealed image with `next build` baked in,
  `NODE_ENV=production`. Postgres on host port `5434`, web on `3002`,
  db `showbook_prod`, role `showbook_prod`. Start with `pnpm prod:up`,
  then `pnpm prod:db:migrate` once after first up. `.env.prod` must set
  `POSTGRES_PASSWORD` — the compose builds `DATABASE_URL` from it
  (don't set both).
- **E2E (mobile backend)** — `infra/docker-compose.e2e.yml` (project
  `showbook-e2e`), reads `.env.e2e`, same sealed image as prod.
  Postgres on host port `5435`, web on `3004`, db `showbook_e2e`, role
  `showbook_e2e`. Loopback-only, isolated `AUTH_SECRET`, allowlist
  pinned to the Maestro test user; the Android emulator on the prod
  box reaches it via `http://10.0.2.2:3004`. Start with `pnpm e2e:up`
  + `pnpm e2e:db:migrate`; `deploy.yml` refreshes it after every prod
  deploy. Runbook: `docs/specs/operations.md` § "Mobile e2e backend".
  (Distinct from Playwright's throwaway `showbook_e2e` database inside
  the dev postgres container — this one is long-lived.)
- The stacks coexist on the same host: web ports 3001 / 3002 / 3004,
  postgres 5433 / 5434 / 5435. The Cloudflare Tunnel ingress for the
  prod hostname targets `http://localhost:3002`. Playwright's E2E dev
  server defaults to `3003` (override with `PLAYWRIGHT_PORT`) so it
  doesn't collide with any stack.
- `scripts/guard-not-prod-db.mjs` refuses any dev/test workspace
  command whose `DATABASE_URL` points at a `showbook_prod*` database —
  prod migrations must go through `pnpm prod:db:migrate`. See README.md
  "Production deployment" for the env checklist.

## Verification

- `pnpm verify` — build + lint + typecheck + unit tests + **`pnpm audit --prod`**,
  with an end-of-run status summary. The audit step is a real gate, not
  advisory: between the 2026-06-13 and 2026-08-05 security reviews the
  production advisory count drifted from 2 (dev tooling only) to 22 (3 critical
  / 11 high, several in the live request path) without anything failing, because
  nothing checked. A transitive advisory that can't be resolved by bumping the
  direct dependency goes in the `pnpm.overrides` block in the root
  `package.json`, so the exception is recorded rather than silently tolerated.
  An advisory with **no patched release at all** (an override can't fix it —
  e.g. the 2026-08 `image-size` DoS pair under metro) is recorded in
  `pnpm.auditConfig.ignoreGhsas` instead, with the justification in the commit
  that adds it; drop the entry once upstream ships a fix.
- `pnpm verify:e2e` — adds Playwright e2e (or set `RUN_E2E=1`)
- `pnpm verify:coverage` — build + lint + typecheck + unit + integration with
  merged Node native code coverage; **fails if any of lines / branches /
  functions is below 80%**, scored independently for the web scope and the
  mobile scope (`apps/mobile/lib/**`). CI runs this on every push and PR to
  `main`.
- **Lint and typecheck both fan out across every project.** `pnpm lint`
  (`nx run-many -t lint`) runs eslint on web (`eslint-config-next`), mobile
  (`eslint-config-expo`), and every package under `packages/*` (the shared
  `eslint.config.packages.mjs`). `pnpm typecheck` (`nx run-many -t typecheck`)
  runs `tsc --noEmit` on all nine projects. `pnpm build` only type-checks the
  web app via `next build`, so the explicit typecheck pass is what catches a
  mobile or package `tsc` break before it lands on `main`. Add a `lint` /
  `typecheck` script to any new package so `run-many` picks it up.
- `pnpm test:unit` — unit tests across all packages (uses `node:test`)
- Integration tests live in `*.integration.test.ts` and are excluded from
  the unit-test glob; run with `pnpm test:integration`. Each integration
  test has a 60 s per-test timeout (overridable via
  `INTEGRATION_PER_TEST_TIMEOUT_MS`) enforced by `scripts/run-integration.mjs`,
  and the batch is killed after 5 min.
- `pnpm email:smoke` — render the daily digest with sample data to disk
- `pnpm email:preview` — react-email dev server with hot reload

App-specific test conventions (when to reach for unit vs integration
vs e2e/Maestro, fixtures, gating env vars) live in the per-app CLAUDE
files.

## Test coverage

**80% line / branch / function coverage is required on `main`.** The CI
workflow at `.github/workflows/ci.yml` blocks merges that drop below the
threshold in either the web scope or the mobile scope. Coverage scope,
thresholds, and exclusions are owned by `scripts/coverage-report.mjs`;
merged LCOV lands at `coverage/lcov.info`.

Excluded from coverage (justified): `packages/db/**` (schema +
generated migrations), per-package re-export `index.ts` barrels,
`packages/scrapers/{run,runtime,extract,cli}.ts` (Playwright-bound),
`packages/jobs/{boss,registry,load-env-local}.ts` (orchestration
wiring + dev-only), Next.js page/layout/loading shells under
`apps/web/app/`, the test-only `/api/test/*` routes, NextAuth /
tRPC mount routes, and layout-heavy mobile code under
`apps/mobile/{app,components}/` (the mobile gate is scoped to
`apps/mobile/lib/**`).

## Observability and logging

All new code MUST use the shared `@showbook/observability` package — no `console.log/warn/error` and no direct `langfuse` / `pino` imports. This is enforced by review and applies to every package, including CLI scripts and pg-boss handlers. If you find yourself reaching for `console.*`, extend the observability package instead.

**Structured logs (pino → Axiom):**
- Import `logger` (or `child({ component, ... })`) from `@showbook/observability` for every log line.
- Use structured fields, short messages: `logger.info({ event: 'phase.start', phase: 1, venueId }, 'Phase 1 started')`.
- Errors: `logger.error({ err }, 'message')` — the custom `serializeErr` in `packages/observability/src/logger.ts` flattens the stack, surfaces `code` / `detail` (postgres-js / Drizzle SQLSTATE), and walks `err.cause` recursively, so wrapped errors keep their underlying SQLSTATE in Axiom.
- Never log secrets, raw user PII, raw email bodies, or image bytes. Redaction covers `apiKey`/`authorization`/`token`/`password` but don't rely on it.
- In jobs, bind `{ job, jobId }` on the child logger so Axiom queries can filter by run.
- Logs go to stdout (pretty in dev, JSON in prod) and to Axiom when `AXIOM_TOKEN` is set; behaviour must work with the env unset (tests, offline dev).

**Where logs go (per env):**
- **dev / local** — stdout only, pretty-printed via `pino-pretty`. `AXIOM_TOKEN` and `AXIOM_DATASET` are intentionally unset in `.env.dev` / `apps/web/.env.local`, so dev runs never ship to Axiom. Tests rely on this — don't set `AXIOM_TOKEN` in CI.
- **prod** — stdout (JSON via `pino`, captured by Docker) AND shipped to Axiom via `@axiomhq/pino`. **One dataset, `showbook-prod` (`AXIOM_DATASET`):** all logs land there — app/server logs plus mobile telemetry (the `mobile.*` events relayed through the `telemetry.logEvent` tRPC router, bound with `component: 'mobile.telemetry'`). Before ingest every record is reshaped by `reshapeForAxiom` in `packages/observability/src/logger.ts`: only the `CORE_FIELDS` allowlist stays as top-level columns and every other key folds into a single `fields` **map field**, so the dataset stays under Axiom's per-dataset column cap no matter what call-sites log (stdout / `docker logs` stay flat — the reshape is on the Axiom stream only). The `AXIOM_TOKEN` in `.env.prod` is **ingest-only by design** (a service token, not a query token) and must have ingest on `showbook-prod`; reading prod logs requires a separate user-scoped token (see "Querying Axiom" below). See `docs/specs/operations/axiom-map-fields.md`. (The earlier `prod-server` + `prod-mobile` split is superseded by this merge; the cutover's history is recorded in the knowledge vault, `brain/projects/showbook/decisions/2026-05-axiom-dataset-cutover.md`.)

**Querying Axiom from the CLI (read access):**
The repo-side `AXIOM_TOKEN` cannot read logs. To query, use a Personal Access Token (PAT) you create in the Axiom UI under Settings → Profile → Personal Access Tokens (or an advanced API token with the `Query` capability granted on the `showbook-prod` dataset). PATs and most advanced API tokens require the `X-AXIOM-ORG-ID` header.

```bash
# Get the org id once: it's the slug shown in Axiom URLs and is also returned
# from `GET /v1/orgs`.
ORG=showbook-egap
TOKEN=xapt-...  # PAT — never commit

# APL query against the Brandon-window:
curl -sS -X POST "https://api.axiom.co/v1/datasets/_apl?format=tabular" \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-AXIOM-ORG-ID: $ORG" \
  -H "Content-Type: application/json" \
  -d '{"apl":"[\"showbook-prod\"] | where _time > ago(1h) and level in (\"warn\",\"error\") | project _time, level, event, msg | order by _time desc"}'
```

Fields that aren't in `CORE_FIELDS` are folded into the `fields` map field, so address them with map syntax: `['fields']['spotifyTrackId']` (or `fields.venueId`). e.g. `[\"showbook-prod\"] | where event == \"venue.follow\" | extend vid = ['fields']['venueId']`. The `_time`, `level`, `msg`, `event`, `component`, `job`, `jobId`, `userId`, `reason`, `status`, `durationMs`/`elapsedMs` fields and `err.*` stay top-level columns.

The stdout copy in the prod web container (`docker logs showbook-prod-web`) is also a useful fallback when you need recent logs but Axiom retention has rolled them off.

**Axiom column / field cap (256 per dataset) — now bounded by a map field.** Each unique top-level or dotted field name across all logged events becomes a column in the Axiom dataset, and every dataset has a hard cap (256 on our plan); once hit, every new write that introduces an unseen field is rejected with `adding 'X' and N other fields to dataset fields would exceed the column limit` (visible only in stdout / `docker logs showbook-prod-web` — the rejection itself never reaches Axiom). The original `showbook-prod` dataset hit this in 2026-05; the fix split it into `prod-server` + `prod-mobile`, which only *doubled* the budget, and `prod-server` filled up again in 2026-06. **The current design recreates a single dataset (we reclaimed the `showbook-prod` name, fresh schema) and removes the cap as a failure mode:** `reshapeForAxiom` keeps only `CORE_FIELDS` (~18 keys) as real columns and folds everything else into the single `fields` map field, whose nested keys do **not** count against the cap. So the new `showbook-prod` sits at ~40 columns permanently and no call-site can widen it past that. See `docs/specs/operations/axiom-map-fields.md`. Still worth knowing:

- **Error-object enumeration.** Pino's `stdSerializers.err` walks every enumerable property on an `Error` via `for (key in err)`. DOMException-like errors (RN, fetch on Node 22, Web Crypto) carry ~24 inherited constant properties (`ABORT_ERR`, `DATA_CLONE_ERR`, `HIERARCHY_REQUEST_ERR`, …); each would become its own `err.*` column. `err` is kept as a top-level field (it's a hot triage path — `err.code` / `err.detail` stay directly queryable), so the custom `serializeErr` in `packages/observability/src/logger.ts` is still allowlist-based to bound it — `ALLOWED_ERROR_FIELDS` is the schema's contract for error logs. Add fields there as new error shapes appear (don't reach for a permissive enumeration shortcut).
- **Ad-hoc context fields are no longer fatal, but prefer stable keys.** A `log.X({ randomNewKey: ... })` callsite now lands `randomNewKey` inside the `fields` map (zero column cost) instead of permanently widening the schema. It's still better to reuse existing keys (`event`, `userId`, `jobId`, `assetId`, `key`, `host`, `bytes`, `elapsedMs`, etc.) for query ergonomics: a folded field is queried as `['fields']['k']`, costs more query-hours, and is stored as a string. Promote a key into `CORE_FIELDS` only when it's filtered/grouped/aggregated in APL and the map-field cost or string typing actually hurts.

**If the cap is ever hit again** (shouldn't happen with the map field in place — would mean ~240 *new* top-level keys leaked past the allowlist): the durable fix is to verify the reshape is active and that whatever widened the schema is being folded, not to grow the budget. The blunt recoveries still exist — (a) recreate the dataset (drops the 30-day window, schema starts fresh); (b) point `AXIOM_DATASET` at a new name (e.g. `showbook-prod-v2`) and restart prod; (c) ask Axiom support to raise the cap — but reach for them only after confirming the map field isn't doing its job.

**Structured event names worth knowing (curated list, prefix-grouped):**
- `auth.user_created`, `auth.signin` — NextAuth lifecycle.
- `tm.request.*`, `tm.normalize.failed`, `tm.normalize.skipped`, `tm.normalize.support_performer_failed`, `tm.normalize.attraction_dropped` — Ticketmaster client + ingest normalizer. `tm.normalize.skipped` carries a `reason` field (`missing_venue_name` | `missing_venue_city` at warn — genuine TM data-quality problems — and `unknown_kind` at info: events whose classification infers as 'unknown' are deliberately filtered non-live-show noise (the High Roller Wheel et al), ~3.9k/week, demoted from warn 2026-07-02) for TM events the normalizer refuses to ingest. `tm.normalize.attraction_dropped` (added 2026-05-19, reason `missing_name`) fires when the resale-marketplace listings ship an attraction object with only an `id`; pre-filter those leaked through as the literal string "undefined" in `announcement.support` and crashed `matchOrCreatePerformer`.
- `setlistfm.request.*` — setlist.fm client (rate-limit retries, errors). `setlistfm.request.cooldown_opened` (added 2026-05-21) fires when two consecutive 429s open the process-wide setlist.fm cooldown gate; `setlistfm.request.cooldown_skip` fires for each suppressed call during the cooldown window. Closes the door on the corpus-fill 429 cascade that piled 875 `enrichment/setlist-corpus-fill` rows into `pgboss.job` failed state when the daily quota tipped over. `setlistfm.request.retry` (warn, added 2026-06-09) fires when a transient transport failure (undici `TimeoutError` / `TypeError: fetch failed` / socket reset) is retried on a fresh connection; carries `{path, code}` and mirrors `places.request.retry` / `wikidata.request.retry` (detection shared via `packages/api/src/transient-fetch.ts`). It exists because the nightly corpus-fill and MBID-backfill crons were surfacing these blips as `setlist.corpus_fill.failed` / `performer.mbid.failed` errors (~5–16/night), the dominant `error_volume` contributor.
- `setlist.corpus_fill.{started,complete,failed,rate_limited}` — corpus-fill job lifecycle. `rate_limited` (added 2026-05-21) fires when `fetchArtistSetlists` throws `SetlistFmError(status=429)`; the job returns cleanly with `skipped: 'rate_limited'` instead of throwing, so pg-boss doesn't burn its retry budget against a sustained 429 event.
- `venue_matcher.tm_lookup.failed`, `venue_matcher.geocode.failed`, `venue_matcher.geocode_update.failed` — `matchOrCreateVenue` external-call boundaries.
- `geocode.google.failed`, `geocode.google.no_lat_lng`, `geocode.nominatim.http_error`, `geocode.nominatim.failed` — `geocodeVenue` provider boundaries; were silent until 2026-04-30. `geocode.degraded` (warn) and `geocode.exhausted` (warn) were added 2026-08-06 when `geocodeVenue` moved onto [`provider-router`](https://www.npmjs.com/package/provider-router): `degraded` fires when the answer came from Nominatim (coordinates but no `googlePlaceId` and no `photoUrl`, so venue dedup and the photo backfill are thinner), and `exhausted` replaces the old silent `return null`. The router also emits `router.*` events into this logger at debug. Nominatim's one-call-per-second usage policy is now per-provider pacing (`minInterval: 1.1`) rather than a module-level timestamp every caller shared, and a 429 carrying `Retry-After` opens its circuit for exactly that long on the first refusal.
- `places.autocomplete.error`, `places.autocomplete.parse_failed`, `places.details.parse_failed`, `places.request.retry` — Google Places (New) client boundaries in `packages/api/src/google-places.ts`. `places.request.retry` (warn, added 2026-06-06) fires when `fetchWithRetry` retries a transient transport failure (`TypeError: fetch failed` / `ECONNRESET` / timeout / `UND_ERR_SOCKET`) before re-attempting; it carries `{call, code}` where `call` is `getPlaceDetails` | `getPlacePhotoName` | `autocomplete`. `getPlacePhotoName` (added 2026-07-02) is the photo-only Place-details variant (field mask `photos`, Google's free "Essentials IDs Only" SKU) that the nightly `backfill-venue-photos` refresh uses — the full `getPlaceDetails` mask includes `displayName`, a Pro-SKU field that billed the whole nightly full-corpus refresh at $17/1k past 5k calls/month (the 2026-06 ~$10/day Places bill); `places.details.parse_failed` from that path carries `call: 'getPlacePhotoName'`. `places.request.retry` exists because the `backfill-venue-photos` cron makes hundreds of sequential Place-details calls and Google resets a small fraction of keep-alive sockets mid-read — those used to surface as 3–19 `venue.photo.failed` errors/night (the only `error_volume` contributor). A retry on a fresh connection clears virtually all of them; a transient error that survives every attempt still propagates and logs `venue.photo.failed` at error level, so a real Google Places outage still trips the gauge.
- `performer.match.created`, `performer.match.race_recovered` — `matchOrCreatePerformer` writes (added 2026-04-30 to track external-ID coverage).
- `performer.image.{updated,no_match,failed,updated_image_only,done,fatal}` — `backfill-performer-images` job.
- `venue.photo.{updated,missing,failed,done,fatal}` — `backfill-venue-photos` job.
- `venue.photo.proxy.{upstream_error,host_not_allowed,redirect_not_allowed}` — `/api/venue-photo/[venueId]` SSRF-guard and upstream-failure boundaries. `upstream_error` fires on a non-2xx / wrong-content-type final response; `host_not_allowed` fires when a persisted absolute `photoUrl` falls outside `ALLOWED_PROXY_HOSTS`; `redirect_not_allowed` fires when an upstream 3xx points to a host outside `ALLOWED_REDIRECT_HOSTS` (added 2026-05-17 alongside the one-hop redirect follow that restored Google-Places-backed venue photos broken by `redirect: 'manual'` in #192).
- `venue.placeid.{updated,conflict,failed,missing}` — Google Place-ID resolution inside the `backfill/venue-photos` cron (05:45 ET), fired per venue before the photo fetch. `missing`/`updated` info, `conflict` warn (another venue row already owns the place id), `failed` error.
- `performer.photo.{persisted,persist_failed,persist_image_only,lazy_resolve_failed,refresh_attempt,refresh_failed}` + `performer.photo.proxy.{upstream_error,host_not_allowed,redirect_not_allowed}` — `/api/performer-photo/[performerId]` lazy-resolving SSRF-guarded image proxy (mirrors the venue-photo proxy). Resolves via Wikidata P18 then TM on miss and persists on success; `persist_image_only` (warn) is the TM-id unique-index conflict fallback that writes only the image. The `proxy.*` events match the `venue.photo.proxy.*` semantics.
- `show.cover.{updated,no_match,failed,fatal,done,event_search_failed}` + `backfill.show_cover_images.summary` — daily 06:15 ET `backfill/show-cover-images` cron filling `shows.cover_image_url` for theatre/festival shows via TM event/attraction search (`summary` carries `{total, updated, missing, failed}`). The lazy `/api/show-cover/[showId]` proxy route adds `show.cover.{persisted,persist_failed,lazy_resolve_failed,refresh_attempt,refresh_failed}` and `show.cover.proxy.{upstream_error,host_not_allowed,redirect_not_allowed}`, mirroring the venue-photo / performer-photo proxies.
- `venue.follow`, `venue.follow.place_backfill_failed` — `venues.follow` lazy backfill.
- `venue.rename`, `venue.rename.reset` — per-user venue-name alias set / cleared via `venues.rename` / `venues.resetName` (writes `user_venue_names`, not the shared `venues.name`). `admin.venue.rename` is the operator-only global canonical rename (`admin.renameVenue`). `admin.venue.location_updated` (carries `{venueId, geocoded}`) is the operator-only canonical city/region/country edit (`admin.updateVenueLocation`) — the escape hatch for venues whose city imported as the literal "Unknown" (Gmail / free-text / festival-poster paths default to it), which the coordinate + Ticketmaster backfills both skip; `admin.venue.location_geocode_failed` (warn) fires when the opportunistic post-edit geocode throws (the city edit still persists).
- `discover.ingest.{performer,venue,region,targeted,*}.complete` — pg-boss discover-ingest jobs.
- `discover.ingest.consolidate_singles` / `discover.ingest.singles_consolidated` — the Phase 3.5 self-healing sweep (`consolidateGroupableSingles`) that folds scattered single-night announcement rows for the same (venueId, lower(headliner), kind) into one multi-date run when `shouldGroup` is satisfied (comedy / concert residencies whose nights were ingested across separate passes, so they never went back through `groupEventsIntoRuns`). `consolidate_singles` is the per-run rollup (`{consolidated}`); `singles_consolidated` is the per-cluster merge (`{venueId, headliner, kind, canonicalId, mergedRows, deletedRows, finalDateCount}`). Failures log `discover.ingest.consolidate_singles.failed` (whole pass), `discover.ingest.consolidate_singles.cluster_failed` (one cluster), and `discover.ingest.targeted.venue_consolidate_failed` (the venue-scoped pass run after a follow-a-venue ingest). Festivals keep their own `discover.ingest.festival_consolidated` path.
- `shows.nightly.summary`, `setlist.retry.summary` — nightly transition + setlist-retry jobs.
- `shows.create.{tm_enrichment_failed,venue_place_backfill_failed,mbid_resolve_failed}` — `shows.create` non-blocking enrichments. `mbid_resolve_failed` fires when the inline setlist.fm MBID lookup for a future concert errors; the nightly backfill at 04:30 ET catches the gap.
- `backfill.performer_images.summary`, `backfill.performer_mbids.summary`, `backfill.performer_ticketmaster_ids.summary`, `backfill.performer_spotify_ids.summary`, `backfill.venue_photos.summary` — scheduled backfill jobs. `performer-images` (05:30 ET), `venue-photos` (05:45 ET), and `performer-mbids` (04:30 ET) are standalone daily crons; the **TM-id, Spotify-id, and Wikidata-QID** summaries are now emitted by the combined `backfill/performer-ids` sweep (see below), and *also* on operator-triggered runs of the individual queues from the `/admin` page.
- `backfill.performer_ids.summary` — rollup for the combined `backfill/performer-ids` cron (daily 06:00 ET), which resolves the three external catalog IDs in one sweep, in dependency order: Ticketmaster → Spotify → Wikidata, landing before the 07:00 health check (the Wikidata leg used to run at 07:15, *after* the health check, so its result was always a day stale in the morning email). Carries `{tmUpdated, spotifyUpdated, wikidataUpdated, failed}`. Each leg also logs its own `backfill.performer_{ticketmaster_ids,spotify_ids,wikidata_qids}.summary` so per-resolver dashboards keep working; a leg that throws logs `performer.{ticketmaster_id,spotify_id,wikidata_qid}.fatal` and the sweep continues. The individual `backfill/performer-{ticketmaster,spotify,wikidata}-ids` queues stay registered (unscheduled) for `/admin` on-demand triggers.
- `backfill.show_ticket_urls.summary`, `show.ticket_url.{updated,no_match,no_keyword,failed,done,fatal}` — daily 06:45 ET `backfill/show-ticket-urls` cron (also enqueued on demand from the `/admin` page via `admin.backfill_show_ticket_urls.enqueue`). Fills `shows.ticket_url` for future watching / ticketed shows whose TM URL never landed at create time (Gmail / Eventbrite / setlist.fm imports). Festivals and past shows are excluded at the query level — a stale link is worse than no link, and the ShowCard already hides the icon when `state === 'past'`. `no_match` is "TM event search returned nothing for venue+date+keyword"; `no_keyword` is the (rare) case where the row has neither a headliner nor a productionName.
- `performer.mbid.{updated,no_match,conflict,failed,done,fatal}`, `setlist_lookup.{mbid_resolved,mbid_conflict}` — performer MBID resolution events. Per-row outcomes from the `backfill-performer-mbids` cron (`updated` on a setlist.fm match write, `no_match` when search returns empty, `conflict` when another performer row already owns the MBID or the row was filled between SELECT and UPDATE — see `reason: 'other_row_owns_id'` vs `reason: 'row_already_filled'`, `failed` for unexpected errors). `setlist_lookup.mbid_resolved` is the same signal from the inline `resolvePerformerMbid` hop on `shows.create` and `fetchSetlistForPerformer`; `mbid_conflict` is the inline variant of the cron's `conflict` event.
- `performer.ticketmaster_id.{updated,no_match,conflict,failed,done,fatal}` — per-row outcomes from the TM-id leg of the combined `backfill/performer-ids` cron (daily 06:00 ET) and on-demand admin runs. `conflict` carries a `reason` field (`other_row_owns_id` | `row_already_filled`) matching the MBID job's race-guard split. The job also emits `performer.mbid.{conflict,failed}` for the TM-derived MBID side-effect path, with `reason: 'other_row_owns_id'` distinguishing TM-side conflicts.
- `performer.spotify_id.{updated,no_match,conflict,failed,done,token_failed,fatal,resolve_inline_uncaught}` — Spotify catalog-id resolution events from the fire-and-forget hook in `matchOrCreatePerformer` and from the Spotify-id leg of the combined `backfill/performer-ids` cron (daily 06:00 ET) / admin trigger. `updated` fires on a successful UPDATE writing `spotify_artist_id`; `no_match` when `/v1/search?type=artist` returns empty for the performer's name; `conflict` carries `reason: 'other_row_owns_id'` (another performer already holds this Spotify id — duplicate-performer cleanup is an operator merge) vs `'row_already_filled'` (something filled the column between SELECT and UPDATE — usually the cron racing the inline hook). `token_failed` indicates the app-level `client_credentials` token exchange failed and the resolver bailed before any search; `resolve_inline_uncaught` is the defence-in-depth catch on the inline hook's `void resolve(...).catch(...)` chain.
- `performer.wikidata_qid.{updated,no_match,conflict,failed,done,fatal,resolve_inline_uncaught}` plus `wikidata.request.{retry,error,parse_failed}` — Wikidata QID resolution for theatre cast / non-Ticketmaster performers, from the fire-and-forget hook in `matchOrCreatePerformer` (scoped to created rows with **no** `ticketmaster_attraction_id`) and the Wikidata leg of the combined `backfill/performer-ids` cron (daily 06:00 ET) / admin trigger. `updated` fires on a successful UPDATE writing `wikidata_qid` (and, when the Wikidata entity carries them, COALESCE-filling `image_url` from P18 and `musicbrainz_id` from P434); `no_match` when `wbsearchentities` returns no human whose label exactly matches the performer name; `conflict` carries `reason: 'other_row_owns_id'` (another performer already holds this QID) vs `'row_already_filled'`. A P434 collision with an existing row is logged as `performer.mbid.conflict` (`reason: 'other_row_owns_id'`) and the QID + image are still written. `wikidata.request.*` are the `packages/api/src/wikidata.ts` client boundaries (mirrors `places.request.*`). The whole path is a no-op when `WIKIDATA_ENRICHMENT_DISABLED=1` (set in the test scripts so the fire-and-forget hook stays offline; Wikidata needs no API key so prod/dev leave it unset).
- `backfill.performer_wikidata_qids.summary` / `admin.backfill_performer_wikidata_qids.enqueue` — the Wikidata rollup (`{total, updated, missing, skipped, failed}`), emitted by the combined `backfill/performer-ids` cron's Wikidata leg (06:00 ET) and on operator-triggered `/admin` enqueue (`{userId, jobId}`). The sweep scans `wikidata_qid IS NULL AND ticketmaster_attraction_id IS NULL`, stalest-first.
- `prune.nightly.summary` — rollup for the combined `prune/nightly` cron (daily 02:00 ET), which runs the three prune phases in a guaranteed order (past-announcements → orphan-catalog → orphan-media) so ordering is a code guarantee rather than wall-clock spacing. Carries `{pastAnnouncements, orphanAnnouncements, orphanVenues, orphanPerformers, mediaRowsDeleted}`. Each phase still logs its own summary event (below); a phase that throws logs `prune.{past_announcements,orphan_catalog,orphan_media}.failed` and the sweep continues. The individual `prune/*` queues stay registered (unscheduled); `prune/orphan-catalog` remains admin-triggerable.
- `prune.past_announcements.summary` — the past-announcement phase of `prune/nightly` (deletes announcements whose last performance is behind us: `COALESCE(run_end_date, show_date) < CURRENT_DATE` — `show_date` alone is a run's FIRST night and would vaporise an in-progress multi-night run/festival the morning after it opens). Runs before the orphan sweep so it sees the freshly pruned set; the `announcement_has_preserver` rule would otherwise keep past announcements alive forever whenever a user follows the headliner.
- `prune.orphan_media.summary`, `prune.orphan_media.delete_failed` — the orphan-media phase of `prune/nightly` (also runnable standalone via the unscheduled `prune/orphan-media` queue) that sweeps terminal-but-stuck `media_assets` rows and their R2 blobs. Two row classes: `status='failed'` (oversize-guard / completeUpload terminal failures whose inline R2 cleanup may have left a blob) and `status='pending'` older than 24h (upload intent issued but never completed — app backgrounded, mobile force-quit). `delete_failed` fires per-variant when R2 returns an error; the DB row is dropped after all variant deletes settle so a permanent R2 outage doesn't accumulate DB rows.
- `notifications.digest.summary` — daily email digest. Per-user outcomes log as `notifications.digest.sent` on a Resend-accepted send, `notifications.digest.send_failed` when Resend returns a non-throwing error response (the SDK resolves with `{ data: null, error }` on bounces / unverified-domain / rate-limit so the rejection has to be inspected on the result), `notifications.digest.failed` for unexpected exceptions in the per-user loop, `notifications.digest.already_sent_today` for the per-user idempotency skip, `notifications.digest.dry_run` when `RESEND_API_KEY` is unset, and `notifications.digest.preamble_failed` when Groq preamble generation falls back to the static greeting.
- `pgboss.{started,stopped,registered,shutdown.start,shutdown.complete,shutdown.failed,register.invoked,register.duplicate,boot.ok,boot.failed}` — pg-boss lifecycle. The `shutdown.*` events fire from the Next.js SIGTERM/SIGINT handler in `apps/web/instrumentation.ts`; absence of a `shutdown.start` before a `started` means the previous boot was killed without graceful release of in-flight jobs. `register.invoked` carries a per-process counter so Axiom can confirm whether Next.js invokes `register()` more than once per process; `register.duplicate` fires when `registerAllJobs` is called twice against the same boss instance and the second call is suppressed (this is the guard that prevents the doubled `boss.work` registrations that surfaced as duplicate `job.start` events for every cron job in May 2026).
- `env.validate.{ok,failed}` — boot-time environment validation in `apps/web/instrumentation.ts` (logic in `apps/web/lib/validate-env.ts`). `failed` carries `problems` (the list of missing/malformed required vars: `AUTH_SECRET`, `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `DATABASE_URL`, `NEXTAUTH_URL`, `TOKEN_KEY`). In production a `failed` is fatal — the process logs, flushes, and `process.exit(1)` (visible crash-loop under `restart: unless-stopped`) rather than serving a half-configured app; outside production it's a warning so dev/test/e2e stub envs aren't crashed.
- `gmail.scan.{truncated,summary,dedup.skipped,attachment.used,attachment.fetch_failed,attachment.parse_failed,attachment.llm_failed}` — Gmail scan orchestrator (`apps/web/app/api/gmail/scan/route.ts`). `summary` rolls up per-scan counts (`heuristicSkipped`, `pdfFallbackUsed`, `dedupSkipped`, `extracted`); the `attachment.*` events trace the R1 PDF-fallback branch; `dedup.skipped` is the P4 cross-scan dedup short-circuit fired before any Groq call. (R1/P1/P4 refer to the 2026-05-08 email-ingestion improvement plan, now in the knowledge vault.)
- `gmail.bulk_scan.{truncated,collected,batch,complete}` — user-triggered Gmail bulk-scan tRPC flow (`enrichment.bulkScan`): collects messages paged up to a cap (`truncated` carries `{cap, userId}`), LLM-parses in batches (`batch` is debug), dedups by show content, and rolls up in `complete`.
- `eventbrite.callback.{state_mismatch,no_access_token,token_exchange_failed}` (warn), `eventbrite.request.{ok,error,rate_limited}` (`ok` debug, rest warn, carry `{path, status, durationMs}`), `eventbrite.scan.{complete,failed}` — Eventbrite OAuth callback, API-client boundary, and past-orders import scan (`/api/eventbrite/*` routes + `packages/api/src/eventbrite.ts`), gated on the `EventbriteImportEnabled` feature flag.
- `apple_music_import.{config_missing,list.success,list.failed,import.success}` / `apple_music.request.{error,rate_limited}` / `spotify_import.{list.success,list.failed,import.success}` — followed/library-artist import mutations (`routers/apple-music-import.ts`, `routers/spotify-import.ts`): `list.*` resolves the user's artists against Ticketmaster (`{userId, total, resolved, matched}`), `import.success` persists the selection (`{imported, skipped}`); `apple_music.request.*` are the Apple Music client boundaries.
- `trpc.error` — last-resort tRPC procedure error log.
- `media.upload_intent.url_issued` — `media.createUploadIntent` per-variant log, fired once per presigned URL handed to the client. Carries `{assetId, variant, mimeType, storageMode, host, path, scope, expires, signedHeaders}`. Signature itself stays server-only. Added 2026-05-27 because the mobile telemetry that was supposed to report the corresponding `upload.put.failed` had never reached prod, leaving R2 PUT failures un-diagnosable from logs.
- `media.upload.{request,ok,no_pending_asset,too_large,read_body_failed,r2_write_failed}` — the `PUT /api/media/upload` landing route (the mobile app streams blobs here rather than direct-to-R2). `request` (info) is the unconditional entry log; `ok` (info) is a successful landing (`{key, bytes, elapsedMs}`); `no_pending_asset` (warn, added 2026-07-06) is the security guard rejecting a `PUT` whose key doesn't match a `pending`, quota-reserved `media_assets` variant owned by the caller (409) — the write is also capped at the reserved variant's `bytes`, so neither the count nor byte quota can be bypassed by PUTting outside `createUploadIntent`; `too_large` (warn, added 2026-08-05) is the 413 from that byte ceiling; `read_body_failed` (error) / `r2_write_failed` (error) are the body-read and object-store boundaries. The pure key→variant matcher is `apps/web/lib/media-upload-auth.ts`. The body is read via `apps/web/lib/drain-capped.ts`, which **streams with a cap** instead of `request.arrayBuffer()` — `Content-Length` is advisory (absent on chunked uploads, and forgeable), and App Router handlers impose no body limit, so buffering first let an authenticated caller push an arbitrarily large body into memory before the 413 could fire. It still drains to end-of-stream after overflowing, because answering mid-upload makes iOS NSURLSession report `NSURLErrorCannotParseResponse (-1017)` and mask the real error.
- `media.complete.{ready,failed,head_failed,cleanup_failed}` — `media.completeUpload` outcomes (added 2026-07-02; only `cleanup_failed` existed before, leaving the pending→ready transition invisible). `ready` (info) is the success transition, carries `{assetId, userId, variantCount, bytes}`; `failed` (warn) fires when the size guard marks the asset `failed`, with `reason` (`no_bytes` | `oversize`) plus `{bytes, reservedBytes}`; `head_failed` (error) is an R2 HEAD failure during verification, logged with `{err, assetId, variant}` then rethrown (asset stays `pending`; the prune/orphan-media sweep collects it after 24h); `cleanup_failed` (warn) is a per-variant R2 delete rejection during oversize cleanup.
- `media.delete.{done,variant_failed}` — `media.delete` mutation (added 2026-07-02 alongside the fix that made blob deletes `Promise.allSettled` so one R2 rejection can't strand the DB row with half-deleted blobs — same pattern as `prune.orphan_media.delete_failed`). `variant_failed` (warn) fires per rejected variant delete with `{err, assetId, key}`; `done` (info) carries `{assetId, userId, variantsDeleted, variantsFailed}`.
- `admin.r2.selftest.{start,complete,config_error,cleanup_failed}` — `/api/admin/r2-selftest` server-side R2 diagnostic. Operator-triggered (bearer-auth'd with `ADMIN_QUERY_TOKEN`). `complete` carries `{sdkOk, presignOk, presignStatus}`; the full response body (including R2's XML error envelope when `presignOk=false`) is in the HTTP response, not the log.
- `admin.backfill_coordinates.{start,complete}`, `admin.backfill_ticketmaster.{start,complete}` — operator-triggered global venue backfills via the `/admin` page.
- `admin.backfill_performer_mbids.enqueue`, `admin.backfill_performer_ticketmaster_ids.enqueue`, `admin.backfill_performer_spotify_ids.enqueue` — operator-triggered performer-ID backfills enqueued from the `/admin` page. All carry `{userId, jobId}`. The actual per-row work logs through the `performer.mbid.*` / `performer.ticketmaster_id.*` / `performer.spotify_id.*` events and the matching `backfill.performer_*.summary` rollups.
- `admin.sql.{query,error,timeout,rate_limited,config_error,write_attempted}` — the bearer-auth'd read-only `POST /api/admin/sql` diagnostic endpoint (`ADMIN_QUERY_TOKEN`). `query` (info) logs `{ipKey, queryLength, rowCount, truncated, elapsedMs}` — the query *length*, never the SQL text; `timeout` (warn) is SQLSTATE 57014 → 504; `rate_limited` (warn) is the per-IP 30/60s bucket; `write_attempted` (warn) is the read-only-transaction rejection (25006/42501 → 422); `config_error` (error) means the token is unset/too short.
- `job.{start,complete,failed}` — pg-boss job wrapper from `runJob` in `packages/jobs/src/registry.ts`.
- `health.check.{start,summary}`, `health.check.<name>.{ok,warn,fail,unknown}`, `health.check.email.{skipped,failed}`, `health.check.preamble.failed`, `health.check.axiom.{skipped,http_error,failed}`, `health.check.ci.{skipped,http_error,failed}` — daily morning health-check cron (`health/morning-check`, 07:00 ET). `<name>` is one of `failed_jobs`, `missed_schedules`, `error_volume`, `database`, `pgboss_queue`, `data_freshness`, `stalled_scrapes`, `external_apis`, `ci_health`. The job itself is queryable via these events, so `debugging-prod` can confirm the cron ran. The Groq-generated preamble at the top of the email is traced as `groq.generateHealthSummaryPreamble` in Langfuse; it falls back to a deterministic count line when Groq is unavailable. `ci_health` (added 2026-06-04) reports the latest GitHub Actions run per workflow on `main` and renders a per-job "CI Health" section in the email; it reports `unknown` when neither `GITHUB_HEALTH_TOKEN` nor `GITHUB_TOKEN` is set (same skip semantics as the Axiom-backed checks). `health.check.ci.{http_error,failed}` fire from the GitHub Actions client (`packages/jobs/src/health-check/github.ts`) on a non-2xx response / network error.
- `spotify.connect.{started,success,failed,revoked}` — `/api/spotify` authorize → callback exchange → `persistInitialToken` lifecycle (Phase 0 of setlist intelligence). `failed` carries a `reason` field (`state_mismatch` | `token_exchange_or_persist`) so triage can disambiguate CSRF rejections from genuine OAuth failures. `revoked` fires both when the user disconnects from Preferences (`reason: user_disconnect`) and when Spotify returns 401 on token refresh (`reason: 401_from_spotify`).
- `spotify.token.{refreshed,refresh_failed}` — `ensureFreshUserToken`'s near-expiry refresh hop. `refresh_failed` carries the underlying error and is paired with `spotify.connect.revoked` only when the failure was a 401.
- `spotify.request.retry` (warn, added 2026-08-11) — `fetchWithTransientRetry` in `packages/api/src/spotify.ts` retrying a transient transport failure (undici `TypeError: fetch failed` / ETIMEDOUT / `AbortSignal.timeout` TimeoutError, detection shared via `transient-fetch.ts`) on a fresh connection; carries `{call, code}` where `call` is `spotifyFetch` | `refreshSpotifyToken` | `exchangeAuthorizationCode` | `getAppAccessToken`. Mirrors `places.request.retry` / `wikidata.request.retry` / `setlistfm.request.retry`. Added after a single connect-phase ETIMEDOUT on the token refresh aborted a user's hype-playlist export (2026-08-07) — the retried call would have succeeded, as the user's manual retry 30s later did. Playlist-mutating POSTs (`createPlaylist`, `addTracksToPlaylist`, `saveTracksToLibrary`, `replacePlaylistItems`) are deliberately not retried: a request that died mid-flight may have been applied server-side.
- `spotify.playlist.{hype_created,hype_reused,heard_created,heard_reused}` — `spotify.createHypePlaylist` / `spotify.createHeardPlaylist` tRPC mutation outcomes (Phase 3), the **single** success/reuse log per call (the helper's duplicate `spotify.{hype,heard}_playlist.{created,reused}` logs in `spotify-playlist.ts` were removed 2026-07-02; their extra fields moved here). `*_created` fires when a brand-new Spotify playlist is written and carries `{playlistId, trackCount, requested, missingCount, cacheHits, cacheMisses, durationMs}`; `*_reused` fires on the idempotency short-circuit (existing `show_spotify_playlists` row) and omits the resolution telemetry. Paired with the per-stage failure events in `spotify-playlist.ts` (`spotify.hype_playlist.failed`, `spotify.heard_playlist.failed` — carry a `stage` field: `resolve` | `create` | `add_tracks`).
- `spotify.track_resolve.{hit,miss,failed}` — per-track title→URI resolution inside `spotify-playlist.ts`. `hit`/`miss` are debug (source: songs table / memory cache / Spotify search); `failed` (error) is a thrown Spotify search error — the track lands in the playlist's `missing` list and the cache is not poisoned.
- `spotify.recently_played.{summary,bucketed,no_token,no_data,fetch_failed,failed}` — `spotify/recently-played` cron (09:00 ET nightly): pulls each connected user's `/me/player/recently-played` and buckets plays into per-show prep/post counts (±6h of show date, frozen 6h after). `summary` carries `{attempted, matched, noMatch, failed}`; `bucketed` (info) is the per-show write; `fetch_failed`/`failed` are error.
- `spotify.purge_revoked.summary` — weekly `spotify/purge-revoked-tokens` cron (Sun 02:00 ET) hard-deleting token rows revoked >30 days ago.
- `spotify.follow_diff.{served,empty,failed}` — `setlistIntel` followed-artists diff read path (`served`/`empty` info, `failed` error).
- `year_end_soundtrack.{summary,built,no_token,no_candidates,top_tracks_failed,existing_404,failed}` — `spotify/year-end-soundtrack` cron (Dec 31 03:00 ET) building each user's "Showbook · <year>" playlist of one signature track per attended show, idempotent via `users.spotify_year_playlists`. `built` covers both fresh and reused (`outcome: 'fresh' | 'reused'`, `{userId, playlistId, year, trackCount}`); `top_tracks_failed`/`existing_404` warn; `failed` error; `summary` rolls up `{attempted, built, reused, skipped, failed, year}`.
- `spotify.preview.{resolved,unavailable}` — `setlistIntel.resolveTrackPreview` lazy resolver (Phase 9). `resolved` carries `{hasPreview, source: 'spotify'|'itunes'|'none', itunesRateLimited}` so Axiom can chart how often Spotify's deprecated `preview_url` field requires iTunes fallback. `unavailable` fires when Spotify search itself errors or finds no match (followed by an iTunes attempt before the row is marked resolved).
- `album_metadata_fill.{summary,performer_failed,album_failed,token_failed,auth_rejected,budget_exhausted,rate_limited}` — Phase 11 §15m `enrichment/album-metadata-fill` cron (02:30 ET nightly, `LONG_BATCH_CRON` → 30-min pg-boss expiry). `summary` rolls up `{attempted, performersUpdated, albumsUpserted, failed}`; `album_failed` (warn) is a per-album track-fetch miss; `performer_failed` (error) is a per-performer catalog error (non-401, or a *transient* 401 that landed after the run had already succeeded for someone — see below); `token_failed` (error) is the up-front app-level `client_credentials` probe failing (cron skips). `auth_rejected` (error, added 2026-05-29) **aborts the whole loop** on a 401 from the catalog API even after `withAppToken`'s fresh-token retry — a run-wide credential/authorization failure (revoked or restricted Spotify app). It replaced the per-performer cascade that logged 1145 `performer_failed` errors in a single run, tripping `error_volume` (fail) and grinding the job past its 30-min expiry into a `failed` pgboss state (which then tripped `pgboss_queue`). A sustained `auth_rejected` means the prod `SPOTIFY_CLIENT_ID`/`SPOTIFY_CLIENT_SECRET` need attention — token exchange still succeeds but the catalog API rejects the resulting token. **Refined 2026-05-30:** a 401 only counts as run-wide (→ `auth_rejected`, abort) when *nothing has succeeded yet this run* OR five 401s land back-to-back; a lone 401 after successes is a transient throttle artifact (logged `performer_failed`, run continues) rather than a reason to nuke a healthy sweep. `budget_exhausted` (warn, added 2026-05-30) fires when the run hits its wall-clock budget (default 25 min, env `ALBUM_METADATA_FILL_BUDGET_MS`) and stops cleanly with partial progress — carries `{attempted, performersUpdated, albumsUpserted, remaining, elapsedMs}`. The budget exists because the full corpus (~1.2k performers × up to 6 Spotify calls each) can't be swept inside the 1800s pg-boss expiry under Spotify's 429 rate limits; a sweep that overran the expiry was being killed into `failed` state and tripping the `pgboss_queue` morning health check (the 2026-05-30 warning). Performers are now processed **stalest-first** (oldest / never-fetched `albums.fetched_at`) so a budget-truncated run still cycles the whole corpus over successive nights instead of re-refreshing the same head. `rate_limited` (warn, added 2026-06-04) fires when a catalog call throws `SpotifyError(status=429)` — `spotifyFetch` now bounds its 429 retry/backoff per call (`MAX_429_RETRIES`, ~25s ceiling) instead of recursing forever, so a sustained 429 storm reaches the loop as a thrown 429 rather than an unbounded single-performer iteration that overran the pg-boss expiry mid-flight (the between-performer wall-clock budget couldn't catch that — it only checks *between* performers). The job stops cleanly on the first 429 with partial progress; carries `{attempted, performersUpdated, albumsUpserted, failed}`. The paired `spotify.request.rate_limited_exhausted` (warn) fires from `spotifyFetch` itself when the per-call retry budget is spent.
- `mobile.spotify.mobile_sdk.{connected,connect_failed,play,play_failed,pause_failed,disconnected}` — mobile Spotify App Remote SDK lifecycle, fired by `apps/mobile/lib/spotify-sdk-driver.ts` via the mobile telemetry sink (`telemetry.logEvent` → `mobile.*` Axiom prefix). `connected` is the Premium-gated mount succeeding in `SpotifySdkMount`; `connect_failed` carries an `errCode` (`ERR_SPOTIFY_NOT_INSTALLED` | `ERR_AUTH_FAILED` | `ERR_USER_NOT_AUTHORIZED` | `ERR_NOT_LOGGED_IN` | `ERR_CONNECT_FAILED` on Android, similar shapes on iOS). `play` per-tap carries `spotifyTrackId`; `play_failed` carries the same plus `errCode`. `ERR_NOT_CONNECTED` on a play flips the driver's `connected` flag off so the next foreground attempts a reconnect — Axiom can chart the reconnect cadence by counting `connected` events per user-session.
- `itunes.preview.{rate_limited,user_rate_limited,failed}`, `itunes.request.{error,parse_error,network_error}` — Apple iTunes Search API fallback used when Spotify returns a track without `preview_url`. `rate_limited` is Apple's IP-level 403 (~20/min); `user_rate_limited` (name kept for dashboard continuity, 2026-08-06) is now the **deployment-wide** resolver guard (`SHOWBOOK_ITUNES_LOOKUPS_PER_MINUTE`, default 20/min, matching Apple's per-IP scope — the old per-user bucket let N users collectively blow through it). Both leave `songs.preview_resolved_at` null so the next tap can retry instead of caching the transient miss.
- `llm.budget.{tripped,rejected}`, `llm.spend.meter_failed`, `budget.schema_init_failed` — the durable budget layer (`packages/api/src/budget.ts`, backed by `mcp-budget-governor` over Postgres; counters in `mcpbg_counters`). Every Groq call through `withLlmRetry` is priced (Groq per-Mtok rates) and metered against the all-users daily `SHOWBOOK_LLM_DAILY_BUDGET_USD` (default $5). `tripped` (warn) fires exactly once per day, on the call that crosses the ceiling; `rejected` (warn) fires per call shed by the pre-flight gate — user-facing procedures and pg-boss jobs alike; `meter_failed` (warn) means spend went **unmetered** (unpriced model or backend blip — a spike is the signal to check the price table in budget.ts); `schema_init_failed` (warn) is the boot-time `mcpbg` schema creation failing (the layer then fails open per call). The library's own events (`governor.limit_exceeded`, `governor.breaker_tripped`, `governor.backend_unavailable`) ride along under `component: budget-governor`.
- `setlistfm.artist_setlists.fetched` / `setlistfm.artist_setlists.not_found` — `fetchArtistSetlists` per-call summary. The `pages` and `count` fields let Axiom track the daily corpus-fill budget against the 1440-call setlist.fm tier.
- `eval.run.{started,complete,failed}`, `eval.run.summary`, `eval.metrics.summary`, `eval.show.{rerun,rerun.requested,cold}` — Phase-4 prediction-eval back-test (`eval/run-daily-backtest`, 03:15 ET). `eval.run.complete` carries `evaluatedShows`, `brierScore`, `precisionTop10`, `recallTop15`, `calibrationError`. `eval.metrics.summary` fires once per style with the rolled-up numbers so Axiom can chart per-style trends without parsing `byStyle` jsonb.
- `setlist.style.refresh.{started,summary,entry_failed}`, `setlist.style.classified`, `setlist.style.seed_applied`, `setlist.style.seed_overridden` — Phase-5 nightly setlist-style-refresh cron (`enrichment/setlist-style-refresh`, 03:30 ET). `classified` fires on each performer style flip with `{oldStyle, newStyle, jaccard, uniqueRatio, corpusSize}`. `seed_applied` fires when a fresh performer matches the curated MBID seed table; `seed_overridden` fires when the auto-classifier wins after three consecutive disagreements.
- `setlist.run_detection.{matched,not_found}` — multi-night-run detector hits/misses fired by the rotating-style predicted-setlist branch (`setlist-intel.predictedSetlist`).
- `setlist.predict.{served,cache_hit,cache_miss,failed,snapshot_failed}` / `setlist.predict_festival.{served,corpus_fill_skipped,corpus_fill_enqueue_failed}` — predicted-setlist read path (`setlist-predict.ts` + `routers/setlist-intel.ts`) on show-detail / festival reads. `served`/`cache_*` info; `failed`/`snapshot_failed` error; `corpus_fill_enqueue_failed` warn (the festival read couldn't enqueue the backing corpus fill).
- `setlist.tour_watch.{summary,refreshed,failed}` — `enrichment/setlist-tour-watch` cron (every 3h) refreshing setlist corpora for performers with upcoming watched shows, deduped per performer via `last_watch_refresh_at`. `failed` is error.
- `setlist.album_drop.boosted` — synthetic album-drop scoring (`album-drop-synthetic.ts`) boosting new-album tracks in a prediction.
- `setlist.special_event.{matched,lookup_failed}` — special-event rule matching in prediction (`matched` info; `lookup_failed` error from the rules read in `routers/setlist-intel.ts`). Admin CRUD on the rules table logs `eval.special_event_rule.{created,updated,deleted}` (info, `{ruleId}`).
- `setlist.song_index.{built,partial,failed}` — per-performer song-index (materialized corpus) rebuild in `song-index-rebuild.ts` (`partial` warn, `failed` error).
- `setlist_intel.show_tab.viewed`, `songs.{page,detail}.viewed` — client view telemetry relayed through `/api/telemetry/*` routes (show-tab tab views; songs surface views, `{userId, songId?}`). Info.
- `setlistIntel.fan_loyalty.{computed,failed}`, `setlistIntel.save_discovered.{ok,failed,not_in_user_history}`, `setlistIntel.discovered_live.failed` — Spotify music-layer intel: fan-loyalty computation, saving a discovered-live song to history (`not_in_user_history` warn), and the discovered-live surface read (`failed` error). Note the camelCase `setlistIntel.` prefix (vs the snake `setlist_intel.show_tab.viewed`).
- `setlist.release_gate.{passed,failed}` — Phase-5 calibration release-gate verdict, evaluated on every read of `setlistIntel.releaseGate`. The `failed` event carries `{metric, value, threshold}` and optional `binLower`/`binUpper` for calibration breaches; the rotating-display feature flag is forced OFF on the client until the gate passes.
- `festival.lineup.extract.{started,parsed,failed}` — `enrichment.extractFestivalLineup` boundary. `started` carries `{source: 'image'|'pdf'}`; `parsed` carries `{artistCount, hasFestivalName, hasDates}` so Axiom can chart hit rate vs. day-of-week. The Groq call itself traces as `groq.extractFestivalLineup` in Langfuse with the same name across image and PDF input paths (differentiated by `metadata.source`).
- `festival.lineup.extract.llm_failed` / `playbill.cast.extract.llm_failed` (warn, `component: 'groq.vision'`) — fired from `groq.ts` when the vision/text Groq call is swallowed into an empty result rather than surfaced as an error. `festival.lineup.extract.llm_failed` is `safeExtractFestivalLineup`'s catch (carries `{source: 'image'|'pdf', model, err}`); `playbill.cast.extract.llm_failed` is `extractCast`'s `LlmValidationError` soft-fail (carries `{model, err}`). They exist because both image callers collapse a dead/failing model into "no artists / no cast found" — the regression behind the Scout→Maverick→Qwen vision migration (2026-06-18) only surfaced via a user report because nothing logged. A run-wide spike of either means the vision model (`MODEL_VISION` in `groq.ts`) is broken, not that posters/playbills suddenly lack content.
- `festival.lineup.tm_match.{hit,miss,failed}` — `enrichment.matchFestivalArtists` per-name TM attractions/search outcome (run as a fan-out batch after lineup extraction so the picker can show artist images). `failed` is a thrown error from the TM client; `miss` is "TM returned no attractions" (still a valid outcome — the lineup can be created without the TM link).
- `account.delete.no_op` (warn, `{userId}` — a delete request found nothing to delete), `account.export.requested` (info, `/api/account/export` data-export route) — account lifecycle odds and ends beyond `auth.user_deleted`.

**LLM observability (Langfuse):**
- Every LLM call goes through `traceLLM({ name, model, input, run })` from `@showbook/observability`. Do not call the Groq SDK directly from new code — extend `packages/api/src/groq.ts` (or the equivalent scrapers helper) so the wrapper is applied once.
- Wrap user-initiated tRPC procedures and pg-boss handlers that invoke LLMs in `withTrace(name, fn, attrs)` so generations nest under a parent trace. Tag `userId`, `jobId`, etc. as attrs.
- Call `await flushObservability()` at the end of any short-lived entry point (job handler, script) so traces aren't dropped.
- Langfuse is for LLM traces only — do not pipe app logs there. App logs go to Axiom via pino.

If a new code path doesn't fit these patterns (e.g. a CLI script), extend the package rather than reaching for `console.log` or a fresh client.

## For agents

Read `docs/specs/README.md` first. It indexes all spec files.
Project history — decision logs (D1–D24), the greenfield/mobile build
plans, reviews, and the live planned-improvements backlog — lives in
the private knowledge vault (`brain/projects/showbook/` in the
workspace); the in-repo specs are the operative contracts.

When the work is web- or mobile-specific, read the relevant per-app
CLAUDE — they document the conventions you'll need to follow inside
that scope.