# CLAUDE.md — agent guidance for pqp

Open-source Discord-like voice + text chat (**pqp.gg**). Repo: [rafaelcg/pqp](https://github.com/rafaelcg/pqp).

For current product status and open work, see [`docs/HANDOVER.md`](./docs/HANDOVER.md). Deeper design: [`ARCHITECTURE.md`](./ARCHITECTURE.md). Deploy: [`docs/DEPLOY.md`](./docs/DEPLOY.md), [`docs/deploy-vultr.md`](./docs/deploy-vultr.md) (production and its rollback plan); [`docs/deploy-fly.md`](./docs/deploy-fly.md) is historical, kept for self-hosters on Fly.

## Stack

- **Monorepo:** pnpm workspaces (`pnpm-workspace.yaml`)
- **Client:** React 19 + Vite + Tailwind + Clerk (`@clerk/clerk-react`)
- **Server:** Node HTTP API + WebSocket (`/ws`) + Postgres
- **Shared:** Zod schemas / protocol types (`@pqp/shared`)
- **Desktop:** Electron shell (loads web client)
- **Auth:** Clerk JWT (Bearer on HTTP; first WS message `{ type: "auth", token }`)
- **Voice:** the server picks the media path per room and pins it: peer-to-peer WebRTC for DM calls and small servers, a LiveKit SFU for communities and servers of ten or more (live in production). Signaling and presence stay on the same WS in both

## Monorepo layout

| Path | Package | Role |
|---|---|---|
| `client/` | `@pqp/client` | SPA (landing + `/app`) |
| `server/` | `@pqp/server` | API, WS chat + voice signaling, optional static serve |
| `packages/shared/` | `@pqp/shared` | Shared types / Zod / voice config |
| `electron/` | `@pqp/electron` | Desktop shell |
| `docs-apps/` | `@pqp/apps-docs` | Public docs site for people who build pqp apps (VitePress, `pnpm docs:apps`). Not deployed yet, see its README |
| `ios/` | (Xcode) | Native SwiftUI iOS client |
| `android/` | (Gradle) | Native Kotlin + Compose Android client |

## How to run (local)

```bash
pnpm install
cp .env.example .env
cp .env.example client/.env
# Fill Clerk keys, or enable DEV_AUTH_BYPASS (see below)

docker compose up -d postgres
pnpm dev
# Client http://localhost:5173 — Server http://localhost:3001 — WS /ws

# Optional MinIO for file attachments (creates the bucket too)
docker compose --profile storage up -d postgres minio minio-init

# Optional desktop shell (Vite must be up)
pnpm electron:dev
```

**Dev auth bypass** (no Clerk): set `DEV_AUTH_BYPASS=true` in root `.env` and `VITE_DEV_AUTH_BYPASS=true` in `client/.env`, then restart server.

With the bypass on, API boot seeds a **Sandbox** hall (text + voice, dummy members, a few messages, one bot). Opt out with `DEV_SEED=false`. Vitest suites truncate: set `TEST_DATABASE_URL` to a copy (`pqp_test`), never to `DATABASE_URL`.

**A second local user** (voice, watch party, DMs, friends, reactions — anything that needs two people). The bypass signs *every* browser in as one shared "Dev User", so two windows are the same account and a two-person feature looks broken rather than untested. To get a genuinely separate account, set a suffix in the second window's console **before** loading `/app`:

```js
localStorage.setItem("pqp:dev-user-suffix", "bob")  // then reload
```

That window becomes `dev_user_bob`, a real row in the database with its own onboarding, presence and seat in a voice room. The first window is untouched. Any `[a-z0-9_-]{1,32}` suffix works, so `alice` / `bob` / `carol` give you three. The server half is `devBypassIdentity` in `server/src/auth/clerk.ts`; the client half is `devAuthToken` in `client/src/lib/dev-auth.ts`.

Same-machine testing needs no second browser profile and no private window, because the suffix lives in `localStorage`, which is per-origin *and* per-profile — but two normal tabs on the same profile share it, so set the suffix in one and **only** one.

## Env vars (names only — never commit `.env`)

See `.env.example`. Important names:

| Area | Names |
|---|---|
| Server | `DATABASE_URL`, `TEST_DATABASE_URL`, `CLERK_SECRET_KEY`, `PORT`, `DEV_AUTH_BYPASS`, `DEV_SEED`, `CHARACTER_ACCOUNTS_ENABLED`, `INSTANCE_MODERATOR_CLERK_IDS`, `ADMIN_METRICS_TOKEN` (machine token for the operator dashboard. Five reads, `GET /api/admin/metrics`, `GET /api/admin/voice-occupancy`, `GET /api/admin/user-activity` (daily actives and signup retention from `user_activity_days`, see `server/src/services/user-activity.ts`), `GET /api/admin/acquisition` (signups by source, medium, campaign, ref and landing path, aggregate counts only; `?days=` default 30, at most 90, and `?hours=` 1 to 168 for a surge window; the per-signup trail is the log line `user.acquired userId= source= medium= campaign= ref= landing= gclid=true|false`, landing path only, never the gclid value; see `server/src/services/acquisition.ts`) and `GET /api/admin/bots` (server bots: adoption, use, the funnel, spend against `BOTS_MAX_USD_PER_DAY` and refused calls over 31 São Paulo days, counts and money only, see `server/src/services/bot-admin.ts`), plus twenty-one operator routes, eight reads and thirteen writes. Reads: find a server, list its voice channels, the watch party waitlist, the runtime feature flags, which app one named person is on (`GET /api/admin/user-client?userId=|handle=|tag=`: platform, app version, OS and browser family and last seen from `user_client_platforms`, families only, never the user agent or IP, 90 day retention; the moderator form is `GET /api/admin/users/:userId/client`; see the admin README), the feedback queue (`GET /api/admin/feedback`: whole text, the author's tag and handle, and the device and call context captured at submit, `feedback.context`), the apps catalog (`GET /api/admin/apps`) and one app's servers (`GET /api/admin/app-servers`). Writes: watch party availability and low latency per server, a channel's transport pin, a channel's SFU region, declining a server's waitlist rows, the mixed channel list migration, a feature flag globally, a feature flag for one server, confirming or closing a feedback item (which on a bug grants the caça-bugs badge), and three on the apps catalog (an app's status, global switch, community unlock and current version `PUT /api/admin/apps`; a version's review `PUT /api/admin/app-review`; a server's operator switch `PUT /api/admin/server-apps`, which ends that server's live sessions of the app when it goes off; and the two that end live sessions: `POST /api/admin/app-disable-everywhere` (master switch off, every live session of the app ended with `app-disabled`, the count on the audit row) and `POST /api/admin/app-session-end` (one session, reason `operator`). Each audited in `app_admin_audit`, with the app id in the body or query because the table has no path patterns). What the token can reach is one flat table, `ADMIN_MACHINE_ROUTES` in `server/src/api/index.ts`; account termination is deliberately not on it. See `tools/admin-dashboard/README.md`), `LOAD_TEST_TOKEN` (**staging only**: a scoped auth path that resolves `Bearer <token>:<suffix>` to a throwaway identity per suffix, so a load harness can drive hundreds of accounts against a public host without the repo-constant dev bypass. Inert unset; refused on any Fly app not named `-staging`, and off Fly under `NODE_ENV=production`. See `server/src/auth/load-test.ts` and `docs/STAGING.md`), `RATE_LIMIT_ANON_CAPACITY` / `RATE_LIMIT_ANON_REFILL` / `RATE_LIMIT_SOCKET_CAPACITY` / `RATE_LIMIT_SOCKET_REFILL` (the two address-keyed backstops; defaults unchanged, tunable because a load harness runs from one IP), `VOICE_MESH_RESUME_REQUIRES_CAP` (**default off**; `true` stops the server holding a **mesh** seat for the 90s resume window unless the socket declared the `mesh-resume` capability. Web and Electron declare it; phones do not and cannot keep the promise. Ships dark on purpose: an API deploy reconnects open tabs without reloading them, so on the deploy that carries it every browser in a mesh call is still on a pre-capability bundle. Flip once `voice.seats.meshResumeSockets` on the dashboard has caught up with the browser share; `voice.seats.meshHoldsRefused` then says whether it is doing anything. See `meshHoldAllowed` in `server/src/ws/voice.ts`), `WATCH_PARTY_WAITLIST` (`on` / `off`; unset follows `LIVE_HLS_ENABLED`, so a self-host that cannot run a watch party never shows the waitlist teaser. Read per request. See `docs/WATCH_PARTY.md` §"The waitlist"), `VOICE_IDLE_ALONE_MINUTES` (default 90; `0` turns it off. The idle hangup: somebody alone in a voice room this long gets a `voice-idle-warning` a minute before and is then disconnected through the moderator's disconnect path with `reason: "idle"`. Alone is the condition, never idle-while-talking. Any self-initiated frame or the warning's "still here" button starts the clock over. Counted as `voice.seats.idleAloneWarned` / `idleAloneDisconnected`. See `sweepIdleAloneSeats` in `server/src/ws/voice.ts`) |
| Runtime feature flags | **These env vars are now DEFAULTS, overridable live from the dashboard (controles → interruptores) with no deploy and no restart:** `WATCH_PARTY_WAITLIST` (also per server), `LIVE_HLS_CAMERA`, `LIVE_HLS_CAMERA_480`, `LIVE_HLS_VOICE_TRACK`, `LIVE_HLS_MIC_ARCHIVE`, `LIVE_HLS_REAP_ORPHANS`, `HLS_SHARER_RESUME_HOLD`, `LIVEKIT_REGION_REQUIRE_CAP`, `VOICE_MESH_RESUME_REQUIRES_CAP`, `VOICE_GATE` (default **on**, client-only: the adaptive gate after RNNoise in Voz limpa; off bypasses every running gate on the next config refresh, mid-call; `docs/NOISE_SUPPRESSION.md` §"The voice gate"), `SFU_REGION_SCOPED_CALLS` (default **on**; off restores moderation that asks every SFU box and waits for the slowest, `docs/plans/SFU_REGIONS.md` §"The control plane"), `SFU_VIDEO_BUDGET_PER_BOX` (default **on**; the video budget `VOICE_PROMOTION_MAX_SFU_MBPS` is one media box's, so a camera or share is priced against only the rooms pinned to the same SFU region as its room, unknown region = home, instead of every room on the cluster; each box may have its own ceiling in `VOICE_PROMOTION_MAX_SFU_MBPS_<REGION>` (`_MIA`, `_LON`), unreadable falls back and never uncaps; the watch party ladder and camera egress are priced against the home box only; off is the old cluster-wide sum. `voice.videoAdmissionRefused` and `voice.transportPromoted` carry `region`, `roomsOnBox`, `peopleOnBox`. `docs/CAPACITY.md` §"The video budget is per box"), `SFU_VIDEO_PRICE_CALIBRATED` (default **on**; in a room of 20 or fewer the first camera costs each participant 1.5 Mbit/s and each further camera 0.4, the 360p layer `adaptiveStream` hands a grid tile, instead of 1.5 each; shares, rooms above 20 and watch parties are priced exactly as before; off is the flat 1.5), `TURN_PREFER_STATIC`, `READ_CACHE`, `USER_STATS` (default **on**: per person and per São Paulo day, call minutes and screen-share minutes from a one-minute sampler over live seats in rooms of two or more real people, tracks queued in calls, and the watch parties watched, copied out of `hls_session_viewers` before its 24h prune; counts only, never who was in a call with whom; the Retrospectiva's data, which did not exist before this shipped; off stops every write; `userStats` on `GET /api/admin/metrics`, where `blindTicks` above 0 on the process running the jobs means it cannot see a seat; see `server/src/services/user-stats.ts` and `docs/plans/RETROSPECTIVA.md`), `DEFAULT_TEXT_FILTER` (`default_text_filter`, per server, default **off**, only `true` turns the variable on: pqp's built-in text filter, a short list of slurs and threats (kill-yourself, rape threats, child sexual content) in English, Portuguese and Spanish, refused on server messages and edits, forum posts, stories, Baú comments and posts and profile names with no AutoMod rule written by anybody; insults and everyday swearing are not on it and direct and group messages are never read by it. **Nothing is filtered until somebody turns it on**: the operator globally or per server from the dashboard, or a server's owner for their own server in AutoMod settings (`server_text_filter_settings`, either direction, the owner's choice wins for that server); the App Review server ("pqp review", `review-seed.ts`) has it on; no staff bypass once on, the audit row and log keep the class and never the word; a profile name follows only the global answer; App Store 1.2; `docs/CONTENT_SAFETY.md` §"The built-in text filter", `packages/shared/src/automod-baseline.ts`), `WS_AUTH_ADMISSION` (on by default: socket arrivals queue at the WebSocket during a reconnect herd instead of on the pg pool; off is every arrival at once, see the "Boot schema and reconnect herd" row), `COMMUNITY_HOME_ENABLED`, `COMMUNITY_HOME_VIP_ENABLED`, `COMMUNITY_HOME_TRANSLATION` (per server, default off; also needs `OPENROUTER_API_KEY`, see the Baú row), `PARTY_NEWCOMER_EXPERIENCE` (per server, default off: phone chat floor, newcomer context strip and no app invite for a new account in a live party; client presentation only, see `docs/FEATURE_FLAGS.md`), `PARTY_FAST_START` (per server; client-only first-frame path, `docs/WATCH_PARTY.md` §"Fast first frame"), `WATCH_PARTY_SERVER_AUDIENCE` (per server, default off: the in-app watch party count becomes the server's distinct-account count from every machine's presence row, sent as `viewers` on `channel-live`; the operator's `liveViewers` is the same number unflagged; `docs/WATCH_PARTY.md` §"How many people watched"), `WATCH_CAMERA_SYNC` (per server, default off, only `true` turns it on, client-only: the viewer's camera player follows the film's wall clock with a rate nudge of at most 5 % or a seek, never touching the film; off, the camera plays loose exactly as before; turn it on for one server from the dashboard, check it on a party, then globally; `docs/WATCH_PARTY.md` §"The camera follows the film"), `CLIENT_FORCE_UPDATE` (default off, global: "announce now". A web or desktop client not on the latest build is normally told of a new build once a day at 09:00 UTC, through a small indicator in the rail; with this on it is told at once and the indicator is drawn with emphasis. It blocks nobody and reloads nobody in a call; it used to put up a blocking screen on every deploy. Served by `GET /api/client-update/config`; turn it off once the rollout is done; `CLIENT_MIN_BUILT_AT`, an ISO date or epoch ms, is its environment-only sibling for bundles built before a moment. `CLIENT_FORCE_UPDATE_IN_CALL` and `CLIENT_CRITICAL_MIN_BUILT_AT` make an update CRITICAL: the only thing that blocks a screen (a small dialog) or reloads a call. `docs/PWA.md` §"Nobody stays on an old bundle"), `RAISED_HAND_STICKY` (global, default **on**: a raised hand stays up until the person, a moderator or leaving the room lowers it; letting somebody speak in audience mode no longer lowers it, your own number on a tile or in the queue lowers your hand, and a moderator with MUTE_MEMBERS gets "Baixar todas" (`POST /api/channels/:id/voice-lower-hands`, 403 when off); served as `raisedHandSticky` on `GET /api/voice/config`; off restores the old audience-mode lowering and hides the controls, `docs/RAISED_HANDS.md`), `SHARE_HIGH_MOTION_GUARD` (per server, default off; the presenter's client steps a starved screen share down in place and the desktop shell raises its process priority while sharing, `docs/DESKTOP.md` §"A share next to a game at a very high frame rate"), `SHARE_FAST_START_QUALITY` (per server, default off, client only: a viewer asks for the share's 720p layer before the SFU binds the subscription, the share is not republished when the room crosses twenty, and the large-room 720p capture cap actually applies; served by `GET /api/share/config`, measured in `client/e2e/share-fast-start/`, `docs/FEATURE_FLAGS.md`), `LINUX_DESKTOP_SYSTEM_AUDIO` (global, default off; the Linux desktop app may carry the computer's sound minus the call on a share, via a `pactl` bus built only for a request the page armed; needs a desktop build that publishes `capabilities.linuxShareAudio`, served by `GET /api/share/config`, `electron/lib/linux-share-audio.js`), `MAC_DESKTOP_SYSTEM_AUDIO` (global, default **on**, `false` / `0` / `off` is the kill switch; the macOS desktop app (14.2+) may carry the computer's sound minus the call on a share, through Chromium's CoreAudio Tap with its own audio process excluded, for a request the page armed; needs a desktop build that publishes `capabilities.macShareAudio` and the Info.plist key `NSAudioCaptureUsageDescription`, and the person grants System Audio Recording once; served by `GET /api/share/config` as `macDesktopSystemAudio`, `docs/DESKTOP.md` §"macOS share audio", `electron/lib/mac-share-audio.js`), `WATCH_NOW_BANNER` (per server, default off, client-only: the strip above the conversation that says somebody is sharing a screen in a voice channel, or a watch party is live, with one tap to watch it as an audience seat with the microphone never opened; computed from the rosters and `channel-live` the client already holds, no new frame and no write per viewer; served as `watchNowBanner` on `GET /api/live-hls/config?serverId=`, the deployment-wide answer is what a conversation reads; turn it on for one server first, `docs/plans/WATCH_NOW.md`), `FORUM_ANNOUNCEMENT` (per server, default off, client presentation only: the one-time "O fórum abriu" corner card, with Abrir o fórum and Agora não, for members of that server who can see a forum there; never in a call, a share, an app or a live party; stamped per browser per server by either button, the X, or opening the forum; served as `announce` on `GET /api/forum/config?serverId=`, only ever true beside `forum_channels`; a campaign: on for one server, off when done; `docs/FEATURE_FLAGS.md`, `client/src/lib/forum-announcement.ts`), `HINT_DIET` (global, default off, client-only: hides the hints that repeat what the app shows, one id list in `client/src/lib/hint-diet.ts`, nothing stored for a hidden hint so the flag off restores every card), `HINT_BUDGET` (global, default off, client-only: no unsolicited hint while in a call, an app or game, a share or party, the composer is focused or has a draft, or a dialog is open; held, not dropped; `client/src/lib/hint-quiet.ts`; both served as `hintDiet` / `hintBudget` on `GET /api/onboarding/config`, `docs/ONBOARDING.md` §"The diet and the quiet contexts"), `SHARE_SOUND_PROMOTION` (global, default **on**: an iPhone starting a screen share in a mesh call, a DM included, sends `audioNeedsSfu` and the server promotes the room to LiveKit so the share carries its sound; yields to a mesh pin, a seat that cannot follow, another live share and the budget; off is picture only on mesh as before; `docs/voice-backends.md` §"The share-sound trigger"), `SOUNDBOARD` (per server, default off: the call soundboard, its settings tab and every play; the play gate reads it per click, so off silences a running call; served as `soundboard` on `GET /api/voice/config?serverId=`, `docs/SOUNDBOARD.md`), `STREAM_START_NOTIFICATIONS` (per server, default off, **the one that can page people**: when a stream starts and stays up 20 s the server tells, once per channel per 30 minutes, the members of a server of at most 200 people who are not the sharer, not in the room, not on DND, not muted and may connect; a server above 200 members and every community notify nobody unless the person opted in for it; claimed by one atomic upsert so two API machines yield one notice; `streamAlerts` on `GET /api/admin/metrics`; served as `streamStartNotifications`, the per-person switch is `notifications.streamAlerts[serverId]`, `docs/plans/WATCH_NOW.md` section 3), `LIVE_PREVIEW` (per server, default off, only `true` turns it on: the signed-out live preview. On a community (`is_community`, not suspended, `COMMUNITIES_ENABLED` on), a visitor with no account who opens `/c/<slug>` or a community invite while a watch party is live confirms their age on their own device (the account gate's date fields and `MINIMUM_AGE_YEARS`, nothing sent or stored) and watches the film through the existing playlist proxy for `LIVE_PREVIEW_SECONDS` (default 300), then is asked to sign up and lands in that channel. Only watch party channels @everyone can VIEW. The window is a signed ticket the visitor holds and the playlist token (viewer token, purpose `preview`) expires with it; no seat, no socket, no chat, no camera, no party pass, not counted as a viewer. `LIVE_PREVIEW_RESET_HOURS` (default 24) is how long a used ticket keeps being refused. `livePreview` on `GET /api/admin/metrics`; `docs/WATCH_PARTY.md` §"Watching without an account"), `LOCADORA_GUEST` (GLOBAL, default off, only `true`: Locadora without an account at `pqp.gg/locadora/jogar` and `/apps/locadora/play`, plus the `/apps/locadora` door, one huge "Botar a fita" that opens the picker or takes a dropped file and opens the player with it already in. Needs `apps_landing`, `APPS_ORIGIN` and Locadora released (`live` + `global_enabled`), not `apps` or `emulator_room`, so it is the emulator room's legal question for the open web: only with Rafael's go. No age question and nothing collected (Rafael's decision 2026-10-21, on the lawyer list as `docs/EMULATOR_ROOM.md` item 10); a session that lives in the page, the shelf in the browser's own `guest:pqp-locadora` store, no socket and no API call but the public config, the file never leaves the device (`client/e2e/locadora-guest/` canary). Served as `guest` on `GET /api/public/apps/config`; off, both paths redirect to `/`. Sign-ups carry `?ref=locadora-guest`. `docs/APPS.md` §"Guest play"), `LOCADORA_SAVES_ANNOUNCED` (GLOBAL, default off: only lets the public Locadora copy say "saves na nuvem", otherwise "em breve"; served as `cloudSaves` on the same route; turn on once `locadora_save_sync` is on for everybody). `VOICE_CALM_DETECTOR` (default **on**, global, client-only: the web call's speaking detector on a 20 Hz timer, 4 Hz while the window is hidden, instead of every animation frame; unknown is off on the client; served as `calmDetector` on `GET /api/voice/config`; `docs/FEATURE_FLAGS.md`). Precedence: per-server override, global row, the variable, the code default; with no row a flag answers exactly what the variable always did, so a self-host changes nothing. `FEATURE_FLAGS_TTL_MS` (default 10000) bounds staleness when `CLUSTER_BUS` is off; with it on, a flip reaches the sibling in milliseconds. Before reading one of these variables to explain production behaviour, check the dashboard or `flags` on `GET /api/admin/metrics`: a row beats the env. Adding a flag, what is deliberately not a flag, and why: `docs/FEATURE_FLAGS.md`, code `server/src/lib/flags.ts` |
| Multi-instance (both default off in the repo; production's Vultr box sets both to `postgres` to run `api-a` + `api-b` as two instances, see pitfall 11) | `CLUSTER_BUS` (`postgres` shares chat fan-out over LISTEN/NOTIFY), `VOICE_REGISTRY` (`postgres` copies the voice peer map and transport pins into the `voice_*` tables and builds rosters from them; with `CLUSTER_BUS` also on, room events cross instances. M1 and M2 of `docs/plans/MULTI_INSTANCE_VOICE.md`; not yet enough to run two machines, M3 and M5 first), `VOICE_REGISTRY_BATCH` (**default off**, and the default IS the rollback until it is measured; **restarts-api**. On, the registry's seat writes for this instance are queued and flushed together every `VOICE_REGISTRY_BATCH_MS` (default 50) or when `VOICE_REGISTRY_BATCH_MAX` (default 200) rows are waiting, as one multi-row statement per kind inside one transaction on one pooled connection, at most one flush in flight. It exists because with `VOICE_REGISTRY=postgres` every join, state change and leave is its own round trip: on staging, 2026-09-18, 800 seated sockets rejoining at once put 4,604 callers on the pool's wait queue and opened the database breaker, and ~700 leaving at once held `checks.postgres.ms` at 1,225 for a minute. 500 joins plus 500 leaves cost 2,000 statements unbatched and 20 batched. Per-peer ordering is preserved, which is pitfall 13 and not a nicety: the queue holds one op per peer and the "is this seat still in the map" check is re-asked at FLUSH time, so a `set-voice-state` behind a leave cannot resurrect the row. Watch `voice.registry.batch` on the dashboard — `rowsCoalesced / batchFlushes` is the compression ratio and about 1 means the flag is buying nothing; `flushFailures` and `staleDropped` belong at zero. Item A2 of `docs/plans/WATCH_PARTY_POSTMORTEM_2026-09-12.md`; see `server/src/voice/registry-batch.ts`) |
| Music queue | `YOUTUBE_API_KEY` (optional; search only, links resolve without it. See `docs/MUSIC.md`) |
| Game connections | `PUBLIC_APP_URL`, `STEAM_WEB_API_KEY`, `BATTLENET_CLIENT_ID`, `BATTLENET_CLIENT_SECRET`, `TWITCH_CLIENT_ID`, `TWITCH_CLIENT_SECRET` (off per provider until set; see `docs/CONNECTIONS.md`) |
| Ambient runner (`tools/ambient`) | `PQP_API_URL`, `AMBIENT_TOKENS_FILE`, `AMBIENT_STATE_DIR`, `AMBIENT_CONFIG`, `AMBIENT_MODEL`, `ANTHROPIC_API_KEY`, `AMBIENT_KILL_SWITCH` |
| Client | `VITE_CLERK_PUBLISHABLE_KEY`, `VITE_API_URL`, `VITE_WS_URL`, `VITE_DEV_AUTH_BYPASS`, `VITE_VOICE_BACKEND` (leave empty to follow the server; `mesh` forces peer-to-peer), `VITE_PQP_BUILD_ID` (overrides the build id baked into the bundle and `/version.json`; CI leaves it unset and the id is the deployed commit from `VITE_FARO_APP_VERSION`), `VITE_UPDATE_MAX_STALE_HOURS` (hours past which an out-of-date page reloads without asking, default 12, `0` off) |
| Beta join links (public, never secrets) | `VITE_TESTFLIGHT_URL` (iOS; a public default lives in `client/src/lib/testflight.ts`, so this only overrides it). `VITE_ANDROID_APK_URL` (Android APK; empty uses GitHub `releases/download/android-beta/pqp.apk`, a single space hides the button). Do not point this at `/releases/latest` — that tag is Electron. `VITE_PLAY_STORE_URL` (Google Play went live 2026-09-28; empty now uses a code default of our own listing, `https://play.google.com/store/apps/details?id=gg.pqp.app`, same shape as the APK variable above — a self-host overrides it with their own listing or hides the badge with a single space, reverting `/android` to the pre-Play, APK-only page. Set, the official Play badge leads everywhere Android is offered — the landing hero, `/download`, the footer, `/vem`, the corner card, `/android` — and the APK drops to a small secondary "ou baixa o APK" link on `/android` only. See `client/src/lib/play-store.ts` and `docs/ANDROID_RELEASE.md` §4b). The join surfaces are `/beta` and `/android`. |
| Hosted-only tags (never set on a self-host) | `VITE_UMAMI_WEBSITE_ID` / `VITE_UMAMI_SRC`, `VITE_GOOGLE_ADS_ID` / `VITE_GOOGLE_ADS_SIGNUP_LABEL` (the tag does not load on `/apps*` or `/locadora*`, a path guard in the inline scripts, `google-ads-tag.ts`), `VITE_ANDROID_APK_CLICK_URL`, `VITE_APPS_DOCS_URL` (public, optional: where the "Build yours" section of `/apps` links the apps docs; empty hides the link, like the donation links). The first two pairs gate a third-party tag that a Vite plugin injects into `index.html` at build time; unset means the tag is absent from the built HTML, which is the point (AGPL, self-hosters must not inherit our analytics or our advertising). Google Ads also needs the label, and only reports one event: an account being created. `VITE_ANDROID_APK_CLICK_URL` is a public POST on `pqp-admin` that counts taps on the `/android` download button; empty means the client never beacons. See `client/src/lib/google-ads-tag.ts`, `client/src/lib/google-ads.ts`, `client/src/lib/android-apk-click.ts` |
| Grafana Faro (frontend errors + RUM, hosted-only, never set on a self-host) | `VITE_FARO_URL` (the Faro collector URL — with it empty the SDK never initialises, so a self-host makes no request and defines no global; same AGPL posture as the analytics tags above), `VITE_FARO_APP_NAME` (defaults `pqp-web`), `VITE_FARO_APP_VERSION` (CI sets it to the deployed commit). The URL carries a write-only ingest key that ships in any client bundle, so it is a repository VARIABLE, not a secret. Captures uncaught errors, unhandled rejections, console errors and Web Vitals; attaches no user identity (no PII). Wired in `deploy-web.yml`'s build env. Build-time source-map upload is gated on `FARO_SOURCEMAP_API_KEY` (a CI secret) plus `FARO_SOURCEMAP_ENDPOINT` / `FARO_SOURCEMAP_APP_ID` / `FARO_SOURCEMAP_STACK_ID` (non-secret ids from the Faro app); all absent means no upload and the build still succeeds. See `client/src/lib/faro.ts` and `client/vite.config.ts` |
| Donations page (hosted-only, never set on a self-host) | `VITE_SPONSOR_URL`, `VITE_PIX_KEY`, `VITE_PIX_BRCODE` (optional "Pix copia e cola" string). Public values, never secrets. `/apoie` and `/support` plus the footer link exist only while the sponsor URL or the Pix key is non-empty; both empty redirects the routes to `/` and hides the link, which is what a self-host wants (same reasoning as the row above: nobody inherits our donation links). Donating unlocks nothing. See `client/src/lib/support-links.ts` |
| Speech-to-text (provider seam only, nothing calls it yet) | `CLOUDFLARE_AI_ACCOUNT_ID`, `CLOUDFLARE_AI_API_TOKEN` (the `workers-ai` provider, `server/src/speech/providers/workers-ai.ts`; secrets, never commit) |
| ICE / TURN (API preferred) | `TURN_URL`, `TURN_USERNAME`, `TURN_CREDENTIAL`, `CLOUDFLARE_TURN_KEY_ID`, `CLOUDFLARE_TURN_API_TOKEN`, `METERED_API_KEY`, `METERED_DOMAIN`, `TURN_PREFER_STATIC` (rollback switch, see below) |
| Client TURN fallback (avoid in prod) | `VITE_TURN_URL`, `VITE_TURN_USERNAME`, `VITE_TURN_CREDENTIAL` |
| SFU | `LIVEKIT_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET` (implemented; the HOME box, the only one watch parties, egress and remux ever use); `LIVEKIT_REGIONS` (**unset by default, and unset is today's single-region behaviour byte for byte**; **restarts-api** to change. `mia:wss://sfu-mia.pqp.gg,lon:wss://sfu-lon.pqp.gg` names the other boxes, each with its own `LIVEKIT_API_KEY_<ID>` / `LIVEKIT_API_SECRET_<ID>` falling back to the home pair), `LIVEKIT_HOME_REGION` (default `sao`), `LIVEKIT_REGION_COUNTRIES` (`US:mia,GB:lon`; the switch that actually routes people. A room follows where its server's people are: members seen in the last 30 days by their last seen country, `users.last_country`, two letters, never an IP, recorded at WS auth only with regions on; 5+ known and one region at 60%+ wins, 5+ without that stays on the default, fewer falls back to the first joiner's `CF-IPCountry`; `server/src/voice/region-audience.ts`), `LIVEKIT_REGION_DEFAULT` (default home). A room's region is pinned at first join beside its transport (`voice_rooms.sfu_region`), carried in the resume token, stated by `POST /api/voice/token`'s `url`; watch party channels stay home, and so do rooms opened by a client without the `sfu-region` cap only when the rollback `LIVEKIT_REGION_REQUIRE_CAP=true` is set (default off: every shipped client, phones included, dials the URL it is handed). Runbook: `docs/plans/SFU_REGIONS.md`, code: `server/src/voice/regions.ts`); `CLOUDFLARE_REALTIME_*` (stub); `VOICE_PROMOTION_MAX_SFU_MBPS` (default 600, the ceiling past which a mesh room is *not* moved to the SFU to fit another camera; 0 turns promotion off) |
| Attachments (S3/R2) | `S3_ENDPOINT`, `S3_BUCKET`, `S3_REGION`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`, `S3_FORCE_PATH_STYLE`, `S3_PUBLIC_BASE_URL`, `MAX_ATTACHMENT_BYTES`, `ATTACHMENT_URL_TTL_SECONDS` |
| Voice note transcription + transcode (worker only) | `VOICE_STT_PROVIDER` (`none` default, `workers-ai`, `groq`; a name with its key missing is `none`, logged once), `VOICE_STT_DAILY_SECONDS` (audio seconds per UTC day for every worker together, default 36000, reserved in `speech_usage_daily`), plus the provider's key: `CLOUDFLARE_AI_ACCOUNT_ID` / `CLOUDFLARE_AI_API_TOKEN` or `GROQ_API_KEY` (secrets, never commit). On the Vultr box they live in `/opt/pqp/worker.env`, which only the `worker` service reads: the API queues jobs and never calls a provider. Whether transcripts exist is the runtime flag `voice_note_transcription` (**default off, per server**; off also hides stored ones). The AAC copy of Opus notes needs ffmpeg, which is only in the `worker` Dockerfile target (`<sha>-worker` tag); a process without it never claims a transcode. Queue: `speech_jobs` (lease + `SKIP LOCKED`, NOTIFY `pqp_speech_job_due`); code in `server/src/services/speech-*.ts` and `voice-transcription.ts` |
| Audience mode | `AUDIENCE_MODE` (runtime flag `audience_mode`, per server, default off; the dashboard override is how it goes on for one server. "Modo plateia" in voice calls; see `docs/plans/AUDIENCE_MODE.md`) |
| Server bots | `CUSTOM_BOTS` (runtime flag `custom_bots`, **per server, default off**: the whole feature; off hides the settings tab and the member list row, no bot answers, no welcome or reminder goes out, and the run and reminder jobs do not query. A reminder that falls due while its server is off waits, it is never cancelled), `BOT_TEST_CHAT` (runtime flag `bot_test_chat`, per server, default off: the wizard's practice chat, each turn a model call), `ANTHROPIC_API_KEY` (a secret; the only provider production offers by default, so without it bots are off in production. Goes in `/opt/pqp/.env`, which the API and the `worker` both read: the API serves the config and the practice chat, the worker runs the replies and the reminders), `BOTS_MAX_USD_PER_DAY` (default 5, the whole deployment per São Paulo day), `BOTS_REPLIES_PER_SERVER_DAY`, `BOTS_RUNS_PER_MEMBER_HOUR`, `BOTS_ACTIONS_PER_HOUR`, `BOTS_TEST_TURNS_PER_MEMBER_DAY`, `BOTS_MODEL_CHOICES` / `BOTS_MODEL` / `BOTS_PROVIDER`. `GET /api/admin/bots` and the dashboard's "bots · beta" section are the operator's view. Rollout and limits: `docs/plans/CUSTOM_BOTS.md`, flag notes `docs/FEATURE_FLAGS.md` |
| Forum channels | `FORUM_CHANNELS` (runtime flag `forum_channels`, per server, default off; gates creating a forum channel and its posts, never reading them; served on `GET /api/forum/config?serverId=` as `enabled`. Contract and the meaning of every forum field: `packages/shared/src/forums.ts`; plan `docs/plans/FORUM_CHANNELS.md`; flag notes `docs/FEATURE_FLAGS.md`) |
| Apps platform and Locadora | `APPS` (runtime flag `apps`, per server, default off: the whole platform, every apps route 404s and nothing renders while off), `APPS_ORIGIN` (the cookieless site that serves bundles, hosted `https://pqp-apps.pages.dev`, local `http://localhost:5175`; unset means no app can run), `EMULATOR_ROOM` (runtime flag `emulator_room`, per server, default off: Locadora and the old emulator dialog, served on `GET /api/emulator/config?serverId=`), `LOCADORA_SHARE` (runtime flag `locadora_share`, per server, default off: an emulator app's pane share and spectators), `LOCADORA_SAVE_SYNC` (runtime flag `locadora_save_sync`, per server, default off, only `true` turns it on: Saves na conta, a person's own Locadora saves on their account through the `S3_*` bucket, never the game). `emulator_room`, `locadora_share` and `locadora_save_sync` each wait for a written lawyer answer (`docs/EMULATOR_ROOM.md`, bottom). Repo variable `APPS_PAGES_PROJECT` names the Pages project `deploy-apps.yml` publishes to (empty skips). Platform: `docs/APPS.md`; flags: `docs/FEATURE_FLAGS.md` |
| Jam channels | `JAM_CHANNELS` (runtime flag `jam_channels`, per server, default off; a `jam` channel holds songs, each a pile of recorded tracks that play together. Gates creating a jam channel and every song or track write (create, edit, upload ticket, add, vote), never a read, and never the three removals (delete a song, delete a track, unvote); served on `GET /api/jam/config?serverId=` as `enabled` beside `uploads`. Uploads use jam-specific limits (audio 32 MiB, video 100 MiB, up to 10 min) through the same bucket, not the message attachment path. Contract and the meaning of every field: `packages/shared/src/jam.ts`; operator page `docs/JAM.md`; flag notes `docs/FEATURE_FLAGS.md`) |
| Communities | `COMMUNITIES_ENABLED` (default off — read `docs/CONTENT_SAFETY.md` §Communities first; it changes the instance's legal category, not just its features) |
| Channel list | `MIXED_CHANNEL_LIST` (runtime flag `mixed_channel_list`, per server, default off). On for a server, the migration job moves its loose channels into a new "Canais" category within a minute (snapshot kept, `POST /api/admin/channel-layout` reverts) and `GET /api/servers/:id/channel-layout` answers `mixed: true`: categories are the only headers and types mix. On globally, a new server is born with Canais. See `server/src/services/channel-layout.ts` and the fixture all clients run, `packages/shared/fixtures/channel-layout-cases.json` |
| Baú (Community Home) | `COMMUNITY_HOME_ENABLED`, `COMMUNITY_HOME_VIP_ENABLED` (both default off; server-side, read per request; `community_home_vip` is also a per-server runtime flag, overridable one server at a time from the dashboard; the client follows `GET /api/community-home/config`, `?serverId=` for the VIP answer. Not `COMMUNITIES_ENABLED`. See `docs/COMMUNITY_HOME.md`). **Translation** (readers in another language see a post in their own): the runtime flag `community_home_translation` (env `COMMUNITY_HOME_TRANSLATION`, **default off, per server**, flipped from the dashboard) AND `OPENROUTER_API_KEY` (a secret; unset means the feature is cleanly off, never an error). Tuning, all optional: `COMMUNITY_HOME_TRANSLATION_MODEL` (default `google/gemini-3.1-flash-lite`), `COMMUNITY_HOME_TRANSLATION_DAILY_CHARS` (default 200000, shared by every API machine through a row in `community_home_translation_usage`), `COMMUNITY_HOME_TRANSLATION_MAX_CHARS` (per post and language, default 6000), `COMMUNITY_HOME_TRANSLATION_BASE_URL` (another OpenAI-compatible endpoint; the key is sent to it, so env only). Counters under `communityHomeTranslation` on `GET /api/admin/metrics`; see `docs/COMMUNITY_HOME.md` §Translation. **Baú+ payments** (a fan pays the creator for VIP inside pqp; the creator is the merchant, pqp never holds money): the runtime flag `bau_payments` (env `BAU_PAYMENTS`, **default off, per server**; off stops new checkouts, never a month already paid) AND `BAU_PAYMENTS_PROVIDER` (the adapter; unset is the built-in sandbox under the dev bypass and nothing anywhere else, `none` off; the sandbox is refused on production). Sandbox tuning: `BAU_PAYMENTS_SANDBOX_AUTOPAY_SECONDS` (default 8, `0` never), `BAU_PAYMENTS_SANDBOX_CARD` (`false` hides card), `BAU_PAYMENTS_SANDBOX_SECRET`, `BAU_PAYMENTS_PIX_TTL_MINUTES` (default 15). Counters under `creatorHub.payments`; see `docs/BAU_PAYMENTS.md` |
| Watch party recording | `LIVE_HLS_MIC_ARCHIVE` (**default off**; `true` records the watch-party host's voice to its OWN file beside the HLS segments, so a clip can be cut with the film and the voice on separate tracks. Nothing about the stream changes — the audience still hears the browser's mix. On, the host's browser publishes its processed mic a second time under the LiveKit track NAME `mic-archive` (a `Microphone` source, because a publish grant is an allowlist of sources — pitfall 14 — so the name is the contract), every client drops and unsubscribes from that name, and the API starts a **Track Egress**: no transcode, Opus straight into `live/<channelId>/<startedAt>-mic.ogg` in the `LIVE_HLS_S3_*` bucket, on its own `hls_sessions` row with `rung = 'mic'` so retention and `keep_replay` collect it with the party. `liveHls.micArchive` on the dashboard is how many sessions are recording. See `docs/WATCH_PARTY.md` §"The host's voice as its own file") |
| Watch party camera | `LIVE_HLS_CAMERA_480` (**on by default**; `false` / `0` / `off` is the rollback to the 360p camera, read per start, no client rebuild). The presenter's webcam is recorded and served at 854x480 when they publish 480 lines or more (a 360p publication is never upscaled), and `GET /api/live-hls/config` says `cameraHeight: 480` so the presenter's browser caps its camera at 480p with a 360p fallback layer while the share bids at `priority: "high"`. The slot keeps its name `cam360p30` (object prefixes, rows, downloads and the playlist path all key on it). `LIVE_HLS_CAMERA=false` still turns the camera transcode off entirely. See `docs/WATCH_PARTY.md` §"The presenter's camera at 480p" |
| Watch party session lifecycle | `WATCH_PARTY_DRAFT_TTL_MINUTES` (default 30, `0` off: a `draft` older than this whose host has no socket anywhere is **cancelled** by the minute sweep, so an abandoned setup sheet stops blocking its channel), `WATCH_PARTY_DRAFT_STALE_MINUTES` (default 10, `0` off: how long a draft may be the reason somebody else's `POST /api/channels/:id/watch-parties` 409s. Past it, or when its host has no socket, or when the requester IS its host, the create **supersedes** it: cancelled through the transition table, broadcast like any other ending, and the new party takes the channel. A `scheduled` or `live` party is never superseded), `WATCH_PARTY_HOST_GONE_MINUTES` (default 5, `0` off: how long a `live` party outlives its host's last socket before the sweep ends it. The default is 5 rather than something longer because it is the same window `claimHost` uses, and a sweep window past the claim window is a stretch where nobody may take the party and nothing will end it. The sweep can **never** end a party with a stream on it: `channelHasLiveStream` asks `hls_sessions` for an unended row started in the last 12 hours, which is the one authority true on the API and the worker alike and covers the conventional ladder and the LL remux together, and it fails safe by holding). Counters on `GET /api/admin/metrics` under `watchParty.*`; logs `watchParty.sweptDraft` / `sweptHostGone` / `supersededDraft` / `staffOverride`. See `docs/WATCH_PARTY_LIFECYCLE.md` §4 |
| Watch party playlists at the edge | `LIVE_HLS_PLAYLIST_BASE_URL` (**unset by default**; **restarts-api**. Set to `https://hls.pqp.gg` to point playlist URLs at `tools/hls-edge/`, a Cloudflare Worker that validates the viewer token itself and collapses every viewer's 2-4s poll of a rendition into one origin fetch per rung per 2s, instead of one per viewer — see `docs/plans/RELOAD_STORM.md` for the numbers this exists for. Same path shape as the API-relative URL it replaces, so no client rebuild. Applied per recipient in `stampViewerStream` (`hls-viewer-token.ts`), after the token is minted and not before — never in `viewerPlaylistUrl` itself, which stays relative always; getting that order backwards once meant every viewer got a token-less URL and the edge Worker 401'd everyone. **Signed off 2026-09-14.** The edge cache's revocation gap (a ban or lost VIEW, `hls-revocation.ts`, reaching a revoked viewer for as long as OTHER viewers keep a rung's cache warm) is now bounded to 30s wherever the KV denylist below is provisioned, and unbounded-outside-production party passes are refused outright once it is not — see `tools/hls-edge/README.md` §"What this Worker does NOT make faster" and §"Enabling in production". Rollback: unset the var; the Worker itself needs no rollback and can stay deployed idle. See `docs/WATCH_PARTY.md` §"Playlists at the edge". Two more (**restarts-api**, both optional): `LIVE_HLS_VIEWER_TOKEN_TTL_MS` (default unchanged, one hour — moves the API's own `?t=` proxy token off its hardcoded TTL; `hls-revocation.ts` reads the same live value for its own pruning window, so the two move together) and `LIVE_HLS_PARTY_PASS_TTL_MS` (a SEPARATE, longer-lived `?pp=` credential the edge Worker alone checks, so a party past an hour does not need the reactive token refresh; default and hard ceiling six hours regardless of the env value, `0` disables minting entirely. Signed with a different derived key than the viewer token — see `partySecret()` — so the origin's own proxy can never verify one, by construction, keeping its own TTL exactly what it always was). `HLS_EDGE_KV_ACCOUNT_ID` / `HLS_EDGE_KV_NAMESPACE_ID` / `HLS_EDGE_KV_API_TOKEN` (API-side, all three optional and all-or-nothing: write the party-pass/token revocation denylist the edge Worker's `HLS_REVOKED_USERS` KV binding reads, the moment `hls-revocation.ts` records an eviction; unset is a supported no-op, see `server/src/voice/hls-edge-revocation.ts` and `tools/hls-edge/README.md` §"Enabling in production"). See `tools/hls-edge/README.md` §"The party pass" and `docs/WATCH_PARTY.md` §"Two TTLs, two audiences") |
| Watch party segments at the edge | `LIVE_HLS_SEGMENT_BASE_URL` (**unset by default**, read per render so it needs no session restart). Set to `https://hls.pqp.gg` to point conventional segment lines at the same Worker, which serves them from its colo cache off an R2 binding (`LIVE_SEGMENTS`) instead of presigned ENAM URLs. The Worker needs `HLS_SEGMENT_TOKEN_SECRET` (HMAC of the Clerk key with `pqp-hls-segment`) and the binding BEFORE the flag, or viewers get 401/503. Unset is the rollback. See `tools/hls-edge/README.md` §"Segments at the edge" |
| WebSocket compression | `WS_COMPRESSION` (**on by default**; `off` / `false` / `0` is the rollback switch). `permessage-deflate` on `/ws`, measured at ~3x fewer bytes on the wire for ~2.5x the processor on the fan-out. Every stack that connects negotiates it (Chromium, Firefox, WebKit, iOS `URLSessionWebSocketTask`, Android OkHttp 5); a client that does not offer the extension is served uncompressed and connects normally. Settings and the measurements behind each one: `server/src/lib/ws-compression.ts`. The operator dashboard shows what fraction of live sockets actually negotiated it |
| Read cache | `READ_CACHE` (**on by default**; `off` / `false` / `0` is the rollback switch). Coalesces identical Postgres reads during a reload storm — a channel's latest message page, a server's channel list, a channel's watch-party state — behind a short TTL, in-flight coalescing, and stale-while-revalidate, so N concurrent callers with access to the same data share one query instead of issuing N. Per-viewer authorization is never cached and still runs on every request; only the shared data fetch is. Born from the 2026-09-12 watch party postmortem (A2), where 141 reconnecting tabs pinned the Postgres pool. `server/src/lib/read-cache.ts`; counters on `GET /api/admin/metrics` under `readCache.*` |
| Postgres statement timeouts | `PG_QUERY_TIMEOUT_MS` (default **15000**) and `PG_WORKER_QUERY_TIMEOUT_MS` (default **120000**, read only when `WORKER_MODE=worker`, which falls back to `PG_QUERY_TIMEOUT_MS` before its own default). How long a statement issued through the main pool may take before this process stops waiting for it. `0` disables the bound, which is the rollback switch. Postgres's own `statement_timeout` is set to the value and the client-side `query_timeout` to one second more, deliberately: a genuinely slow query is cancelled by the server (`57014`, connection still usable) and pg's own read timer fires only for a reply that is never coming. Added after a staging reconnect storm on 2026-09-17 left both API processes at `total:22 / idle:0 / busy:22 / waiting:15 / pressure:"saturated"` with the breaker open for minutes, while every one of the 45 API backends sat `idle` in `ClientRead`: Postgres had finished, the replies never arrived, and nothing bounded the wait. A timeout alone is not the whole fix: `query_timeout` rejects the promise without cancelling the statement or marking the connection unusable, so a client borrowed through `connect()` and returned by an ordinary `finally { client.release() }` would go back into the idle list still waiting on the missing reply. Any query ending in pg's read timeout or a connection-level error poisons that client and the release destroys it, counted by the `db.pool.clientDestroyed` log line; `57014` deliberately does not, because Postgres resynchronised the protocol and the connection is fine. Boot DDL (`schema.sql`, `CREATE INDEX CONCURRENTLY`) runs on a connection with the bound lifted and destroyed afterwards, so a big instance cannot fail to boot. The pool also sets `keepAlive` with a 10s initial delay. `runtime.pool.longestCheckoutMs` / `checkedOutOver10s` on `GET /api/admin/metrics` and the `db.pool.stuckClient` log line are how a stuck checkout is visible before it takes the pool. See `resolvePgQueryTimeoutMs` in `server/src/db.ts`, `server/src/lib/pool-checkouts.ts`, and `docs/DB_RUNBOOK.md` §3 |
| Boot schema and reconnect herd | `BOOT_SCHEMA_MODE` (**`changed` by default**, `always` is the rollback; **restarts-api**): a boot runs `schema.sql` only when its text differs from the last applied (`schema_boot_state`, under an advisory lock), because re-running an unchanged file still held ACCESS EXCLUSIVE on 27 core tables until it committed, and on every rolling deploy that was while the sibling absorbed the reconnect herd (pitfall 22). Idempotent invariant sweeps that sat at the top level of the file run on every boot regardless (`BOOT_EVERY_TIME_SWEEPS`, and a test fails on an unclassified one). `BOOT_SCHEMA_LOCK_TIMEOUT_MS` (default 2000, `0` lifts it): a run that does happen waits at most this per lock, rolls back and retries rather than holding the sibling's tables. `WS_AUTH_CONCURRENCY` (default `PG_POOL_MAX / 4`): socket arrivals running auth through the voice rosters at once, the rest FIFO at the socket, fail-open after 4 s; the watch-party host write and party catch-up an arrival starts go through a second gate of the same width that never goes over it (the host write waits, the catch-up is dropped after 10 s, both skipped if the socket closed meanwhile, so a late host write can never cancel a real host-gone clock). Both gates' on/off is the `WS_AUTH_ADMISSION` runtime flag; `runtime.wsAuth` on `GET /api/admin/metrics` is the arrival gate. `DRAIN_RATE_PER_SECOND` (default 100, `0` is the old 50 per batch). `VOICE_CATCHUP_CONCURRENCY` (default 4; `0` is the rollback to one unbounded query per room): the voice catch-up answers its cache misses in one statement per socket (`accessibleChannelIds`, `voice.roster.catchUpQueries` / `catchUpChecks`) and runs what is left at most this wide. The dashboard's pool card reads `runtime.poolWait` (per-checkout wait histogram per minute, `server/src/lib/pool-wait.ts`) and goes yellow only on a wait over 1 s, not on the peak since start that every deploy pushes to 22 of 22. Numbers and the pooler decision: `docs/DB_RUNBOOK.md` §3; reproduce with `tools/reconnect-herd-harness/run.mts` |
| Process role | `WORKER_MODE` (unset = one process does everything, today's default; `api` = listeners only, batch jobs skipped, set on `api-a` / `api-b`, which only works while a worker runs; `worker` or `1` = batch jobs + `/health` only, set on the `worker` service). Inventory of what moved: `docs/plans/COLD_PATHS.md` |
| Electron | `VITE_APP_URL` |
| DB backup (`tools/api-host/db-backup.sh`, runs on the API box itself) | No secrets of its own. Reads `DATABASE_URL` and `LIVE_HLS_S3_*` straight out of `/opt/pqp/.env`, the same values the API containers use, and the same R2 bucket (`pqp-live-enam`) the watch party recording feature already writes to. A root cron job (`23 4 * * *` UTC, installed by `tools/api-host/provision.sh`) runs `pg_dump -Fc` against the Vultr managed cluster and uploads with `rclone` to `backups/pqp-db/`, 30-day retention. Replaces the Fly app `pqp-db-backup` (a scheduled machine), destroyed 2026-09-24 after it had been failing silently since the 09-14 DB password rotation. See `docs/DB_RUNBOOK.md` |

**Rule:** never commit `.env` / secrets. Prefer serving ICE via `GET /api/ice-servers` (the API) over baking TURN into the Pages build.

## Architecture (short)

```
Browser/Electron → Clerk (auth)
                 → HTTPS API (servers, channels, messages, /api/ice-servers)
                 → WSS /ws (chat + presence + WebRTC offer/answer/ICE relay)
                 → P2P mesh (audio); TURN when cross-NAT
                 → S3/R2 direct (attachment bytes, presigned; never via the API)
```

- **Transport per room:** mesh caps at `MESH_VOICE_LIMIT` (8) and is what DM calls and servers under `LARGE_SERVER_MEMBER_THRESHOLD` (10) get; listed communities and larger servers go to LiveKit automatically (`server/src/voice/transport-policy.ts`), which is **configured on production today** (`/ready` reports it). **LiveKit SFU is implemented** — set `LIVEKIT_URL` / `LIVEKIT_API_KEY` / `LIVEKIT_API_SECRET` and the server advertises it via `GET /api/voice/backend` (no client rebuild). Presence stays on `/ws` in both modes; only media moves. A room's transport is decided by the server, pinned for the room's lifetime and stated in `welcome` — a client that cannot use it is refused rather than silently split off from the call. `cloudflare-sfu` is still a stub that falls back to mesh. See [`docs/voice-backends.md`](./docs/voice-backends.md).
- **Attachments:** S3-compatible storage (R2 hosted, MinIO local). The API only signs URLs — the browser PUTs and GETs the bytes itself. Off entirely unless `S3_*` is configured. Size is enforced twice: `Content-Length` is signed into the presigned PUT (verified against MinIO, not yet against R2), and the claim `HEAD`s the object — which is also what catches "never uploaded" and a stored type that differs from the signed one. That HEAD runs *before* the claim transaction opens; nothing between `BEGIN` and `COMMIT` may touch the network. See [`docs/ATTACHMENTS.md`](./docs/ATTACHMENTS.md).
- **Watch party availability is data, not configuration.** `servers.live_hls_enabled` is a tri-state column read per request: TRUE is on, FALSE is off *even when the environment says yes*, NULL falls back to `LIVE_HLS_SERVER_ALLOWLIST` exactly as before. `LIVE_HLS_ENABLED` is still the master switch above all three. The operator flips it from the dashboard with no deploy and no restart; a self-host with no dashboard keeps the environment variable it always had. `resolveLiveHlsForServer` in `server/src/voice/hls-egress.ts`, and `docs/WATCH_PARTY.md` §"Widening it is a click now". Low latency follows the same pattern: `servers.live_hls_ll_enabled` (TRUE on, FALSE off even when `LIVE_HLS_LL_ALLOWLIST` names the server, NULL falls back to that variable byte for byte), read per request by `liveHlsLLServerOverride` in `server/src/voice/hls-remux.ts` (only when a live party asked for LL, and never with `LIVE_HLS_LL` off), written through the same `PUT /api/admin/server-live-hls` with `lowLatency`, and `LIVE_HLS_LL` plus the edge playlist front stay the master switches; `docs/WATCH_PARTY.md` §"Low latency is a click too".
- **Watch party waitlist.** On a server whose live-hls config answered `enabled: false`, the sidebar's create slot shows a "Watch party · Acesso antecipado" teaser to everybody; it opens an explainer and a form (`watch_party_waitlist`: a `request` from whoever holds MANAGE_CHANNELS / MANAGE_SERVER or owns it, `interest` from everybody else). Approval is by hand: any operator write that leaves the server effectively on (the dashboard's waitlist "Ativar", which also sets low latency, or the plain "ligar") approves that server's waiting rows and notifies them (socket frame over `CLUSTER_BUS`, push, and a durable card). `WATCH_PARTY_WAITLIST` (`on` / `off`, unset follows `LIVE_HLS_ENABLED`) is the campaign switch. Public page `pqp.gg/watch-party` carries `?intent=watch-party-waitlist` through sign-up. `docs/WATCH_PARTY.md` §"The waitlist".
- **Communities:** a public directory of joinable servers, entirely behind `COMMUNITIES_ENABLED` (default off, exposed to the client via `GET /api/communities/config` like the attachments/GIF configs). A community is a `servers` row with two switches: `is_community` is the **public address** (`pqp.gg/c/<slug>` answers, whoever holds the link joins with one tap, Manage Server may set it) and `is_community_listed` is the **directory** (browsable and searchable by any account, owner-only, requires an address). Every directory read runs auth and hides servers the viewer is banned from; joining reuses `redeemInvite`'s semantics without an invite row. Reports about a community go to the **instance** queue, never to that community's own owner, and the operator can unlist one with a single `UPDATE servers SET is_community_suspended = TRUE`. See [`docs/CONTENT_SAFETY.md`](./docs/CONTENT_SAFETY.md) §Communities — turning it on moves the instance out of Brazil's private-messaging liability exemption (STF, Art. 19, 26 Jun 2025).
- **Handles:** `users.handle` is a *second*, genuinely unique name (`username` is only unique paired with `discriminator`), claimed first-come, NULL for most accounts, one rename per 30 days. It addresses `pqp.gg/@rafa` — the public profile page, served by the unauthenticated endpoint that answers with a person: display name, avatar, an optional uploaded banner (`users.banner_key/banner_url`, on the avatar machinery), public-community badges, **opt-in** Steam / Battle.net / Twitch connections (`visibility = public`), the newest six **approved** depoimentos, and a join month. No id, no tag, no email, no presence. Communities get the same treatment at `pqp.gg/c/<slug>` (`servers.community_slug`, derived from the name when the address goes on, unique among addressed communities); that page is a poster — name, tagline, category, member count, pictures — and never a member list. Open Graph tags for both are injected at the edge by a Cloudflare Pages middleware (`client/functions/`), because a static SPA's client-side `<head>` is invisible to every unfurler. Both public CTAs carry an intent through sign-up (`?add=<handle>`, `?join=<slug>` + a `localStorage` stash — `client/src/lib/handle-intent.ts`). See [`docs/HANDLES.md`](./docs/HANDLES.md) and [`docs/CONNECTIONS.md`](./docs/CONNECTIONS.md).
- **Data model:** Server → Channels (`text` \| `voice`) → Messages (+ `message_attachments`); a compatibility rank of `owner` / `admin` / `member` under a staff ladder of cargos whose real authority is permission bits plus per-channel overwrites; usernames `name#1234`.

## Deploy targets (hosted)

| Piece | Where | URL (as of 2026-08-20) |
|---|---|---|
| Static SPA | Cloudflare Pages project `pqp` | https://pqp-3yr.pages.dev |
| API + WS | Vultr box in `gru` (São Paulo): `api-a` + `api-b` behind Caddy behind Cloudflare (`tools/api-host/`, `docs/deploy-vultr.md`) | https://api.pqp.gg — `wss://api.pqp.gg/ws` |
| Worker (batch jobs) | `worker` service on the same Vultr box, `WORKER_MODE=worker`, runs `node server/dist/worker.js` (`tools/api-host/compose.yaml`) | private `/health` only |
| Staging SPA | Cloudflare Pages branch `staging` of project `pqp` | https://staging.pqp-3yr.pages.dev |
| Staging API + WS | Fly.io app `pqp-api-staging`, region `gru` | https://pqp-api-staging.fly.dev — `wss://pqp-api-staging.fly.dev/ws` |

**Production rollback plan.** Production left Fly on 2026-09-17 and the Fly apps `pqp-api`, `pqp-worker` and `pqp-log-shipper` were destroyed on 2026-10-22, so there is no second production to fail over to. The plan is the box itself: a bad deploy is put back by hand with `sudo /usr/local/bin/pqp-deploy <older sha>` (the same rolling script CI runs, one container always serving; any sha still in GHCR works), the merge freeze (`DEPLOY_FREEZE`) stops the next merge redeploying the bad commit, a lost box is rebuilt from `tools/api-host/` with the secrets from wherever they are kept, and the database is restored from the nightly R2 dump (`docs/DB_RUNBOOK.md`). See [`docs/deploy-vultr.md`](./docs/deploy-vultr.md) §7. A self-host can still deploy to Fly (`fly.staging.toml` is the Fly config in the repo, `docs/deploy-fly.md` the guide).

CI workflows: `.github/workflows/ci.yml`, `deploy-web.yml`, `deploy-api-vultr.yml` (gated on `vars.DEPLOY_TARGET == 'vultr'`, so do not delete that variable; API auto-deploys from `main` — a merged schema/endpoint change is live minutes later), `electron.yml`, `deploy-staging.yml` (staging web + API from the `staging` branch or `workflow_dispatch`; see `docs/STAGING.md`).

**GitHub Actions secrets (names):** `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`, `VITE_CLERK_PUBLISHABLE_KEY`, `VITE_API_URL`, `VITE_WS_URL`; staging adds secret `FLY_API_TOKEN_STAGING` and repo variable `STAGING_CLERK_PUBLISHABLE_KEY`.

**API secrets (names):** `DATABASE_URL`, `CLERK_SECRET_KEY`, plus TURN/ICE vars above, `S3_*` if attachments are wanted, and `PUBLIC_APP_URL` plus `STEAM_WEB_API_KEY` / `BATTLENET_*` / `TWITCH_*` if game connections are wanted. On production they live in `/opt/pqp/.env` on the Vultr box (never in git); on a self-host on Fly they are `fly secrets`. **Worker secrets (names):** `DATABASE_URL`, `CLERK_SECRET_KEY`, `S3_*` if attachments are on (`DATABASE_SSL` if the API has it), and the voice note keys in `/opt/pqp/worker.env`; nothing else. A stale Railway copy may still answer at api-production-206d.up.railway.app; nothing points at it. Do not put Clerk secret, TURN credentials, S3 keys, or those provider secrets in Pages/client secrets.

## Pitfalls already hit

1. **Cross-NAT voice FAILED** — STUN-only is not enough; dead Open Relay creds were unreliable. **Fixed (2026-07-11):** Railway `TURN_*` (ExpressTURN) via `/api/ice-servers`, plus client Retry / ICE restart. Retest: hard-refresh both clients and rejoin voice.
2. **Clerk `getToken` remount loop** — unstable token getter in React deps caused remount storms; keep token access stable (ref / memoized callback), don’t put a fresh `getToken` identity in effect deps every render.
3. **Pages without API URLs** — empty `VITE_API_URL` / `VITE_WS_URL` makes `/app` hit same-origin Pages (no API). Set GH secrets and redeploy web.
4. **Clerk origins** — allow Pages + API origins in the Clerk dashboard.
5. **`@pqp/shared` on Railway** — production resolution needed a dedicated fix; rebuild/shared packaging matters for Docker deploys.
6. **pnpm version** — CI uses pnpm matching the lockfile (pnpm 10); don’t downgrade casually.
7. **Electron Linux artifacts** — scoped package name broke `.deb` paths; fixed in CI metadata.
8. **`curl` against `/api/...` returns 401** — there is no public-route allowlist; `handleApi` resolves a Bearer token before the router runs, so even `/api/attachments/config` and `/api/gifs/config` need one. Locally: `-H "Authorization: Bearer dev-local-token"` with `DEV_AUTH_BYPASS=true` (the bypass is ignored when `NODE_ENV=production`). The exceptions, all handled *before* that resolution and none of them router routes: `/health` (the Vultr box's Docker healthcheck, and Fly's check on a self-host — **process liveness only** as of A3.1 of `docs/plans/ALWAYS_ON.md`: draining, and that this HTTP server is listening, nothing that touches Postgres. Used to run `SELECT 1`; on 2026-09-12 that coupling was the incident — Postgres collapsed, the pool saturated, `/health` failed with it, and the platform stopped routing to the only machine, taking WebSockets, the HLS playlist proxy and every cached read down too, none of which needed Postgres at that instant. A DB-dependent route now answers its own fast 503 instead, via the circuit breaker in `server/src/db.ts` — see the pitfall below), `/ready` (the deep check for external monitors: Postgres, pool, LiveKit, storage; 503 when any is not ok; see `docs/MONITORING.md`. Because `/health` no longer proves a broken release can reach Postgres, `deploy-api-vultr.yml`'s post-deploy verification checks `/ready` too, so routing health and deploy safety are two different checks), `/up`, `/status.json`, the embed-image proxy, `/api/avatars/:userId`, `/api/users/:userId/banner`, `/api/emoji/:emojiId` (a custom server emoji's image, 302 to a presigned URL like a server icon), `/api/bot-faces/:color/:face.png` (a server bot's Breu face, the same thirty PNGs for every server, served from `server/src/assets/bot-faces`, cached a year behind `?v=`), webhook execution, `GET /api/public/profiles/:handle`, `GET /api/public/communities/:slug` (also behind `COMMUNITIES_ENABLED`, and 404 when it is off), `GET /api/public/communities/config` (the landing page's `{enabled}` and nothing else, the same `COMMUNITIES_ENABLED` read as the authenticated config, 200 `{enabled:false}` when off, `public, max-age=60`, own address bucket; matched before the `:slug` route), `GET /api/public/apps/config` (`{enabled, guest, cloudSaves, locadora}`: `enabled` from the global runtime flag `apps_landing`, default off, whether the public pqp Apps pages `/apps`, `/apps/locadora` and `/locadora` exist; `guest` and the guest player's manifest slice from `locadora_guest`, see the flags row; same shape, cache and own bucket as the communities one, never reads Authorization; off redirects the three paths to `/` in the router and, once it has read the answer, at the Pages edge; see `client/src/lib/apps-landing.ts`, `apps-meta.ts` and `docs/FEATURE_FLAGS.md`), and `GET /api/public/invites/:code` (the signed-out invite gate's preview: server name, icon URL and member count only; unknown, expired, exhausted and suspended all answer the same 404; own address-keyed bucket under the anon backstop; see `servePublicInvitePreview`), and the signed-out live preview's three, `GET /api/public/live-preview/communities/:slug`, `GET /api/public/live-preview/invites/:code` and `POST /api/public/live-preview/start` (not even matched while `live_preview` is off everywhere, so they 401 like any unknown path then; each with its own address-keyed bucket under the anon backstop; see `server/src/api/live-preview-routes.ts`). The playlist proxy's header-less door also accepts the preview's token, and only that door: every other reader of a viewer token refuses purpose `preview` unless it asks for it by name.
9. **Cloudflare TURN was configured, deployed and never once used**. `getIceServers` returned on the first match and static `TURN_*` was first, so the Cloudflare credentials sitting on the API were dead code, and a static relay that was throttled or down took cross-network voice with it because nothing else was ever consulted. Compounding it, the Cloudflare response parser required `iceServers` to be an **array** and tested `.length`, but Cloudflare answers with a single object, so even when it was reached it returned null. Two silent failures that looked identical to "working, on the other provider". **Fixed (2026-08-26):** order is now Cloudflare, then Metered, then static as the fallback; the parser accepts either shape; `TURN_PREFER_STATIC=true` restores the old order in one command without a deploy. Pinned by `server/src/services/ice.test.ts`, which asserts *which* relay comes back rather than that the list is well formed.
10. **Persistent "Realtime connection closed" on hosted deploy** — no WS heartbeat/reconnect, plus any thrown WS handler error crashed the whole server (unhandled rejection → Railway restart → every client dropped). **Fixed (2026-07-11):** server ping/pong heartbeat + try/catch around WS handlers + `pool.on("error")`; client auto-reconnect with backoff that resolves a fresh Clerk token per attempt (`client/src/lib/realtime.ts`).
11. **Merging to `main` restarts the API.** A server or `packages/shared` change is `restarts-api`. Since the 17 September 2026 cutover, production is the Vultr box (`deploy-api-vultr.yml`, `tools/api-host/`): two API containers, `api-a` and `api-b`, behind Caddy, sharing chat fan-out and the voice peer map through Postgres (`CLUSTER_BUS` / `VOICE_REGISTRY`, both on). `pqp-deploy.sh` recreates them one at a time (`api-a` confirmed healthy on the new tag, only then `api-b`), so one container is always serving and a redeploy no longer closes every `/ws` at once; a socket on the container being recycled reconnects onto its sibling. Losing the whole box is still full downtime, and there is no second production behind it: the Fly path (its workflow, config and apps) was removed on 2026-10-22, so the rollback plan is the box itself (`pqp-deploy <older sha>` by hand, the nightly R2 dump, `docs/deploy-vultr.md` §7). On the container that is recycled, that machine drains rather than drops: `/health` goes 503 first so the proxy stops routing to it, then the sockets get 1001 paced at about 100 a second (`drainPlan` in `server/src/lib/drain.ts`; `DRAIN_RATE_PER_SECOND=0` is the old 50 every 100 to 150 ms), and the reconnects land on the container that stayed up. After #162, refreshed **web and Electron** keep the media session and resume the same peer id (90s orphan window in-process; HMAC token lasts hours so a long call still reconstructs after an API restart). **iOS in a LiveKit room** keeps its media across the restart the same way (it declares `resume` and presents the token). **iOS in a mesh room, Android**, failed resume, and tabs that have not refreshed still drop out of voice. Client-only Pages deploys do not restart the API. This is the usual case. It is not a hangup. Apply `drops-voice` only when the change itself will hang up a live call even after resume. An API restart alone does not earn that label. Say `restarts-api` in the PR when it applies, before claiming the PR is safe to merge. Provider keys (Steam, Twitch, Battle.net) live in `/opt/pqp/.env` on the box, not in git; merging connection code without those secrets must not 500 production.
12. **Roster deltas were deployed and never once sent in production.** `sendRoster` had `registryOn() ? null : foldRoomEvents(events)`, production runs with `VOICE_REGISTRY=postgres`, and every delta test ran with it off. `voice.roster.deltas` read 0 for days and was blamed on browsers that had not reloaded. Same shape as pitfall 9: the flag that changes the code path was not the flag the tests exercised. **Fixed (2026-09-07):** with the registry on the delta is a diff of the rows against what the process last sent (`diffSentRoster`), pinned on a real Postgres by `server/src/ws/voice-roster-delta-registry.test.ts`. When a feature is gated by a flag that production sets, test it with the flag set, and read the counter that proves it runs.
13. **Voice seats outlived the people in them, for fifteen hours.** Ten of nineteen live rooms held exactly one person on 2026-09-08, `orphaned_at` NULL, and the process agreed the sockets were alive. The seat that started it was released correctly (`voice.leave`, `roomSize=0`, id retired) and then **re-inserted six milliseconds later** by the `set-voice-state` frame the client had sent just before hanging up. `trackRowWrite` claimed to order a channel's writes but joined them with `Promise.all`, so both statements were already in flight on two pooled connections and Postgres ran the DELETE first and the UPSERT second. The resurrected row carries a **live** `instance_id`, and `reconcileVoiceRegistry` refuses by design to touch a row whose owner is answering its heartbeat, while the owner has no peer left to release it: immortal, and on every roster. **Fixed (2026-09-08):** `trackRowWrite` takes a thunk and starts the write inside the chain, keyed **per peer** so a room's other seats and their hangups never queue behind one slow statement (completion stays per channel, which is what the roster reads wait on); `writePeerRow` refuses to write a peer the map no longer holds; and `sweepOwnStaleVoicePeers` reclaims, every 15s on the instance heartbeat, any row this instance owns that its map does not, with a 60s grace and `orphaned_at IS NULL`. That last clause is the definition, not an optimisation: a ghost is a row the close path never stamped, while an orphan is a seat deliberately held for the 90s resume window, and the two windows overlap. Narrowing the chain also made an old documented race reachable, so `deleteVoicePeer` is now two statements instead of one CTE on one snapshot, and two simultaneous last-leaves clear the room row instead of each seeing the other still seated. `voice.seats` on `GET /api/admin/metrics` reports idle seats, the oldest, and both counters. Two lessons, both already on this list: a fire-and-forget write is not an ordered write, and every sweep needs a case for **its own** rows. A related half came from the iOS audit (#411): `join-voice-room.resume` is a self-report, its schema says in as many words that phones omit it, and an iOS build sent it for every room anyway because it read `GET /api/voice/backend`, which answers what a *new* room on this deployment would be pinned to rather than what *this* room is. Production runs LiveKit, so it said yes to mesh rooms too, and that build tears its peer connections down on reconnect. Every conversation call is mesh by policy, so **every DM call from an iPhone left a phantom participant for 90 seconds**. The server cannot catch a build that lies about itself: it knows the room's transport exactly and has no signal at all for whether a client can rebuild mesh media, and guessing from a fingerprint (which caps a build happens to send, its user agent) is pitfall 12 waiting to happen. So the fix is a narrower promise the client has to opt into, `mesh-resume`, with `VOICE_MESH_RESUME_REQUIRES_CAP` treating a client that never makes it as unable. That reaches every build already on a phone without an app update, and it is off until browsers have cycled onto the bundle that declares it. Beside it, a second finding of the pitfall-9 shape: `startHeartbeat` in `server/src/ws/index.ts`, its two-strike rule and the whole of `heartbeat.test.ts` were **dead code**: `server/src/index.ts` ran its own inline one-strike copy. There is one reaper now, and `heartbeat.test.ts` reads the entry point to keep it that way.
14. **The web client published every microphone with no source, and a watch party's audience could not be heard at all.** Sampling a live production party on 2026-09-09 showed the host's microphone as `source: 0` (`SOURCE_UNKNOWN`) beside a screen share correctly tagged `3`. `new LocalAudioTrack(raw)` starts at `Track.Source.Unknown` and `publishTrack` only overwrites that when a `source` is passed; the client passed one for the camera and for both halves of a screen share, and not for the mic. Android's publish carries a comment saying it copies "both halves of what the web client publishes", about the two halves it did copy. **This was not cosmetic.** `liveKitPublishGrant` sends `canPublishSources: ["microphone"]` to anybody holding SPEAK and not STREAM, and LiveKit treats a non-empty list as an allowlist that UNKNOWN is not in (`VideoGrant.GetCanPublishSource`, livekit/protocol). In a `watch_party` channel the stream bit is START_WATCH_PARTY, which no ordinary member holds, so **exactly that grant is what an invited speaker gets**: the media server refused their microphone while their own app showed them live and unmuted. Nothing on either side could see it, because the server's own moderation reads `type === AUDIO` and works on an untagged track, so every test and every audit of the mute path passed. Two lessons. A grant written against an enum is a contract with the client, and nothing checks that the client keeps it: the load harness at `tools/watch-party-load` set the source correctly, so the rig was more correct than the product. And when a picker or a grant keys on a field, assert the field on the **publish** side too, which is what `client/src/lib/livekit-session-sources.test.ts` now does for all three sources. **Fixed (2026-09-09):** client-only, so tabs that have not reloaded keep the old behaviour until they do.
15. **A watch party's transcode leaked handlers onto the media box, and a stalling stream was the only symptom.** The same live party logged two `voice.hlsStarted` for one channel nine minutes apart, same presenter, **with nothing at all logged in between**, and the first pair's egress handlers were still running two minutes after the second pair started. Two causes, both of the usual shape. `rungHealth` returns "ended" for two different reasons and only one of them means the handler is over: LiveKit saying so is final, while "the playlist has not moved for 20 s" is a rule that exists precisely *because* a killed egress node is reported ACTIVE forever, and its mirror case, a handler still transcoding while its output stalled, was dropped from `rooms` and **never stopped**, then had a fresh ladder started beside it by `scheduleRestart`. And three of `stopRoom`'s five callers logged nothing, while `pushLiveHls` resolved the channel's server id only when it had a sharer and passed `null` otherwise, so an ordinary end of share was torn down by the not-allowlisted branch (silent) instead of the no-share branch (which logs). **Fixed (2026-09-09):** `stillRunning` tells the two halves of "ended" apart and a stalled rung is stopped; every teardown carries a `reason`; and `reapForeignEgresses` stops, on every monitor tick, any ACTIVE egress in a room this process is presenting that is not one of its own rungs, counted by `liveHls.orphansStopped`, which belongs at zero. Scoped to rooms we hold a session for on purpose: "stop everything I do not recognise" would kill what `adoptLiveHlsSession` exists to inherit across a deploy. The lesson beyond the bug: **`LIVE_HLS_MAX_SESSIONS` counts sessions, and a cap only means what it says while a session is a fixed number of processes.**
16. **A dead Clerk JWT vetoed a valid capability token, and stalled every web viewer of every watch party.** The playlist proxy takes two credentials: a Bearer header, and a `?t=` capability this server signs itself, which names the user, the channel and the session and is minted only after a real access check. `handleApi` resolves a Bearer **before the router**, so any `Authorization` header had to succeed or the request was 401 before the token in the URL was ever looked at, and the token-only door was gated on `!req.headers.authorization`. `hls.js` attaches a Clerk JWT through `xhrSetup` **on top of** the `?t=` already in the URL, caches it in a closure and refreshes it every 30 s without `forceRefresh`, while a Clerk JWT lives about 60 s. So roughly once a minute every playlist request carried an expired token and was rejected: the player stalled, retried, recovered, and did it again a minute later, for the whole film. **Why nothing caught it**: every server test and every `curl` sent only `?t=`, which is the one request shape that never fails, and native iOS and Safari send only `?t=` too. There was even a test named "serves hls.js (header AND token)" and it used a VALID header, so it exercised the shape and not the failure inside it. The failure existed solely in the combination a browser produces. **And the proxy logged nothing on a 401**, so from the server every measurement said the stream was healthy; it took a screenshot of Rafael's network panel after an afternoon of looking. **Fixed (2026-09-09):** the capability is tried again after the Bearer resolution FAILS, which is late enough that there is no other caller to confuse it with (trying it first would let a token serve a caller whose header resolves to somebody else, a rule this route already had and keeps); the client stops attaching a header when the URL already carries a token; and `voice.hlsPlaylistRejected` says which of missing, malformed, bad-signature, expired, wrong-channel, wrong-session it was, rate limited per channel per reason. Three lessons: **a second credential is a second thing that can fail**, and "belt and braces" is only true if neither strap can veto the other; a test that sends one request shape proves nothing about the shape the product sends; and an endpoint that refuses somebody must say why, or the next one of these costs an afternoon too.
17. **`/health` ran `SELECT 1`, so a Postgres collapse took down everything, not just the database.** On 2026-09-12 Postgres saturated the pool; `/health` opened its own fresh connection for the check and failed along with it; Fly (and any load balancer routing on it) had nowhere healthy to send traffic and stopped routing to the only machine — dropping every WebSocket, the HLS playlist proxy and every cached read, none of which touch Postgres to do their job. The health check the proxy routes on had become a proxy for "is the database up", which is a different question from "should traffic keep reaching this process". **Fixed (A3.1, 2026-09-13):** `/health` is process liveness only (draining, HTTP server listening) — database health moved entirely to `/ready`, which was already built for exactly this and was never on `fly.toml` or the Docker healthcheck. A small circuit breaker in `server/src/db.ts` (`createDbBreaker`, `server/src/lib/db-breaker.ts`) watches pool queue depth and its own `SELECT 1` probe — on a dedicated connection, never the shared pool, so a stalled query cannot itself consume the capacity the breaker exists to protect — with hysteresis (needs `DB_BREAKER_GRACE_MS` of sustained badness to open, `DB_BREAKER_HALF_OPEN_SUCCESSES` consecutive good trials to close). While `open` **or `half-open`**, every `pool.query`, `pool.connect()` and the `PoolClient.query` of whatever `connect()` returned (transactions included — a first review of this change caught that only `pool.query` was wrapped at first, leaving every `BEGIN`/`COMMIT` write in the codebase able to hang the pool anyway, and that `half-open` let ordinary traffic through instead of admitting only the breaker's own recovery trial) fast-rejects with `DatabaseUnavailableError` instead of queueing for `connectionTimeoutMillis`, and the few chokepoints that funnel almost every DB-dependent HTTP route (`handleApi`'s central catch, the admin machine-route catch, the HLS token doors in `server/src/api/index.ts`) turn that into a 503 with `Retry-After: 5` and `{ error: "database_unavailable" }` in milliseconds. The HLS playlist proxy keeps rendering FRESH from storage through a database outage, on the strength of a recent "this session is live" (see pitfall 18 for why replaying the last body was not enough), for up to `STALE_ON_BREAKER_MAX_MS` (3 minutes) past the last check that actually confirmed the session live, and the master's rung list is served from its last read for the same bound. A chat send that hits the breaker before `createMessage` commits gets `message-rejected` with reason `database-unavailable` (retriable, unlike `no-access` or `automod`) instead of a silent drop that only resolves as a generic failure after the client's 10s timer; a failure in a *post*-creation step (recording mentions, the thread-chip update) still propagates as a real error rather than telling an already-successful sender to retry into a duplicate post. `DB_BREAKER=off` is the rollback switch; `db.breaker.state` / `.opened` / `.rejected` are on `GET /api/admin/metrics` under `runtime.db.breaker`, sampled live even when the cached half of that same payload is itself serving a stale snapshot because the breaker is open. The lesson: **the health check a proxy routes on must fail only for reasons that make routing away correct** — anything a request doesn't actually need should not be able to veto it. A related lesson from getting the breaker itself reviewed: **a circuit breaker that only wraps the entry point everyone remembers (`query`) is not a circuit breaker for the paths that go around it (`connect()`, a "trial" state that quietly admits everyone) — enumerate every door before trusting the one you built for.**

18. **A 61 s database blip would have frozen every watch party and hung up calls across the two machines.** 2026-09-23 21:42:46Z to 21:43:47Z, Vultr managed Postgres unreachable from the API box, breaker open on both replicas. Nothing broke that night, but the audit for the Saturday party found four ways it could have. (a) The playlist proxy's A3.1 fallback REPLAYED the last rendered body for 30 s: a live playlist that stops advancing is a stall, so a viewer froze about twelve seconds in and got `database_unavailable` at thirty, and before the breaker even opened each render sat up to 16 s on a dead pool connection with every viewer coalesced behind it. The only thing a render asks Postgres is "is this session still live"; the picture is in the bucket. So `confirmSessionLive` now renders fresh from storage on a recent confirmation, waits at most `HLS_LIVENESS_WAIT_MS` for the check, lets a late "ended" still end it, and `liveHls.rendersWithoutDb` counts it. (b) The web player's restart hold (`enterRestartHold`) stops the loader on a run of playlist 5xx and asks `/live` for a NEW session; when the answer was the SAME session nothing ever started loading again, so a viewer reached "A transmissão caiu" however soon the database came back. (c) On recovery every other instance's lease was a minute or two stale, and the first machine to reconcile read its live twin as dead: orphan, delete and retire every seat over there, `peer-left` to its own clients, which tears down the mesh legs that cross machines. `otherLeasesTrustworthy` in `voice/registry.ts` now holds that judgment (and the HLS ownership passes) until this instance has been beating again for a full TTL, and the first good beat re-writes every seat this instance holds. (d) A roster sent while the rows were unreadable was this instance's own peers only, as an authoritative snapshot; it is now the last roster sent plus local changes. The lesson is pitfall 17's, one level down: **a fallback has to keep the thing moving, not just keep answering**, and a lease that went stale while *you* could not see the database is evidence about you, not about the other machine. Reproduce with `tools/db-blip-harness/` (two real APIs through a black-hole TCP proxy, ffmpeg live HLS, headless hls.js, two dev users in a call).
19. **A rolling deploy ended a live watch party twice, and nothing restarted it.** 2026-09-24 rehearsal, channel `ad99074f`, `api-a` then `api-b` recycled. `api-a`'s new process adopted the ladder at boot; the presenter's socket resumed on `api-b`; `api-b` refused to adopt a session a live machine owned (correct); `api-a`, finding no sharer among ITS OWN peers, ended the session after the 5 s grace (`hlsStopped reason=no-share`). A minute later the same thing happened in the other direction, and after that the host sat "live and sharing" with no stream, because the end of a session on one machine produced no event on the machine holding the host. Each machine was right by its own rules: "no sharer here" was being read as "no sharer", the machine with the sharer had no way to take the session, and the only way back was an event nobody was going to send. **Fixed (2026-09-24):** the owner asks the registry where the presenter went before the grace may end anything (`locateVanishedSharer` in `ws/voice.ts`): seated and sharing on another live machine hands the session there with nothing stopped (`releaseLiveHlsSession` re-stamps the rows, the sharer's machine adopts the same egresses or remux session); a seat held for its resume window holds the broadcast for that window; a genuine stop still ends it in 5 s. The machine holding a sharer reconciles when it hears a remote stop, and `sweepHlsSharersWithoutSession` runs every 15 s as the backstop. The relay throttle now coalesces instead of dropping (a dropped "your presenter is here now" is the same outage). Pinned by `server/src/ws/voice-hls-rolling-deploy.test.ts`, which fails on the old code with exactly the rehearsal's `hlsStopped reason=no-share`. The lesson: **with more than one process, "not in my map" is not an answer to "is it there"**; every per-process judgment about a shared resource needs the shared view before it may end something.
20. **Every transcode restart was a new watch party session, so every viewer re-attached.** 2026-09-24, one conventional show: an egress died 22 s after go-live (`Timestamping error on input streams`) and a presenter resumed after a deploy republished the screen on a new sid; each minted a new `startedAt` (new URL, token, master) for what was the same party, and the camera lost ~100 s because the adopting process had never heard the presenter's "separada" declaration. **Fixed (2026-09-24):** the ladder restarts IN PLACE (`restartRoomInPlace`): same room, same `startedAt`, camera and archive untouched, a new egress **run** per rung under `-r<ms>` names, and a durable media-sequence `base` per run on `hls_sessions.runs`, which the playlist proxy stitches with `#EXT-X-DISCONTINUITY` identically on every process (never an in-process rebase: two machines would disagree and hls.js goes fatal on a sequence that runs backwards). Replay and the film download stitch every run; the live render drops the `#EXT-X-ENDLIST` a stopped egress writes, because the row, not the transcoder, says a session is over. Only a genuine end (share gone past its grace, host ended, presenter gone, restart budget out) ends the session. See `docs/WATCH_PARTY.md` §"A restart keeps the session", `tools/restart-harness/`. The lesson: **the viewer-facing identity of a thing and the physical identity of what produces it are two numbers**; tying them together makes every internal repair visible to the audience.
21. **A window could stay on an old bundle for as long as it lived, and Reload did not fix it.** 2026-09-30 field test: the Windows desktop app ran the new shell around the previous deploy's website, so a share went out with no sound. Reproduced with real builds (`client/e2e/stale-bundle/`). The service worker was `registerType: "prompt"` with no `skipWaiting`: a new worker installed and WAITED for every window to close, and until then the OLD worker's precache answered every navigation, so three reloads in a row came back on the previous bundle. Nothing looked for a new build in a window that never navigates, and the update card was the only way out (snoozed, hidden in a call). Found on the way: production's `pqp.gg/sw.js` answers `max-age=14400` while the Pages origin answers `no-cache`, because the Cloudflare zone's Browser Cache TTL beats `_headers` for cacheable extensions (`.js`), so a rule in `_headers` is not proof of what a browser sees; check the real host. **Fixed (2026-09-30):** the worker takes over on install, every bundle carries its build id and `/version.json` names the deployed one, the page polls it (start, focus, about 12 minutes) and reloads at a safe moment (idle, never in a call, typing or watching a party), and `client_force_update` puts a blocking screen up for everybody when a bundle is bad. Applying an update asks the active worker which build it is and purges the caches when it is not the target, because a stale `sw.js` at a CDN finds nothing to install and looks identical to "already current". Electron clears its cached site once per shell version. **Same day, a second report:** a web user hard-reloaded onto the new site and then changed the language, and the old site came back. A hard reload bypasses the worker for ONE load, the language change is a full navigation (`location.replace`) and a worker that is still the old one answers it from the old precache (a second open tab keeps the old worker in control, so a waiting one cannot take over). So navigations are now answered network-first by the worker (`client/src/lib/sw-build-script.ts`), and a worker from before the fix is replaced within seconds of the fixed one's install because it skips waiting. Lessons: **a service worker that waits is a cache you cannot invalidate from the page**, a fix for "it updates on reload" has to be tested with a reload against a real worker, and a header written in the repo says what you asked for, not what the CDN sends. `docs/PWA.md` §"Nobody stays on an old bundle".
22. **Every rolling deploy re-ran `schema.sql`, and its table locks were what filled the sibling's pool.** The operator card said "o pool encostou no teto hoje ... 22 de 22 ... pico da fila 161" three times in three days, while the API box sat at 3% CPU, Postgres was idle and Grafana's once-a-minute `/ready` never saw more than 3 in use. The suspect was the reconnect herd (the drain sends one container's sockets to the other), and it is half the story: each arrival fanned out one access check per occupied room in the cluster, all at once, so a few sockets landing together could take the pool for a tick. The other half was in the logs at exactly those deploys: `deadlock detected` between the boot DDL and roster reads (2026-09-27 21:09Z with the breaker opening, 09-29 16:48Z, 09-30 20:02Z and 23:33Z) and, at 20:02Z, `Failed to start server: error: deadlock detected` twice with a 24 s boot. `initDb` sends the whole file as ONE query, so it is one implicit transaction, and `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` takes ACCESS EXCLUSIVE before it checks whether the column exists: re-applying an unchanged file held 27 of the busiest tables until commit, measured on a scratch database. `pqp-deploy.sh` recreates `api-a` then `api-b`, so the replacement's DDL ran precisely while its sibling absorbed the herd; waiting for one reader, it held every table it had already altered, and every query behind those locks held a pooled connection. **Fixed (2026-10-01):** the file runs only when its hash changed (`BOOT_SCHEMA_MODE=always` is the rollback), under an advisory lock and a 2 s `lock_timeout` with retries when it does; the catch-up's per-room checks are one query per socket (`accessibleChannelIds` priming the cache) and at most 4 wide after that; arrivals queue at the WebSocket behind a fail-open gate (runtime flag `ws_auth_admission`); the drain is paced at 100 a second; and the pool card reads the per-checkout wait (`runtime.poolWait`) instead of the peak since start. Reproduced with `tools/reconnect-herd-harness/run.mts` (two real APIs, 400 sockets, 240 voice seats, 1 ms of database latency): before, the same `roster membership check failed: deadlock detected` lines as production, 3 breaker flips, a pool queue of 14,706, 9,743 checkouts waiting over a second and only 22 of 240 seats resumed; after, no deadlock, a queue of 6, the longest wait 13 ms, `ready` at p95 8 ms and 240 of 240 resumed. Lessons: **an idempotent statement is not a free one** (`IF NOT EXISTS` decides after the lock); a high-water mark with no duration cannot tell a burst from an outage, so measure the wait; and when the pool fills on a box doing nothing, look for a lock before looking for load.
23. **"De 10 em 10 segundos tem atualização", and "as telas param e fica só a voz" (2026-10-13).** A desktop user, in capitals, on a day with nineteen web deploys. Three causes. `client_force_update` had been ON since 2026-10-04 and meant a blocking screen for every open window on every deploy. Nothing in the client had a notion of "not yet", so every deploy was announced to every window. And the second half was not a reload at all: every deploy deletes the old hashed assets, a page open since before it 404s on the first lazy chunk (the emoji and GIF pickers, opened from a call chat), `recoverFromChunkLoadError` correctly refuses to reload a call and defers, and the only boundary above the lazy panel was the ROOT `ErrorBoundary`, whose "pqp updated, you are in a call" fallback replaced the whole app while the voice session, which lives outside React, kept the audio going. **Fixed (2026-10-14):** a new build is announced once a day (`announceAtFor`, 09:00 UTC) as a small indicator with a popover instead of a card, the force switch only announces at once and never blocks (only a critical update does), an automatic reload is one per build and never in a call, a share or a watch party (asked again at the last moment inside `applyUpdate`), the lazy pickers have their own `ContainedChunkBoundary`, and the desktop shell's native box is shown once per version and never over a call. Lessons: **a flag whose off state is "nothing happens" and whose on state is "everyone is interrupted on every deploy" is a flag that must never be left on across a deploy day**; the unit that failed to hold was a screen, and the fix that reads as an update bug was an error-boundary bug; and a test that renders the failure (`error-boundary.test.tsx`, third case) is what proved the mechanism, since the production session was sampled out of Loki. `docs/PWA.md` §"De 10 em 10 segundos".

## Agent norms

- Do not invent secret values in docs or commits.
- Shipping and PR loop: [`AGENTS.md`](./AGENTS.md) (run locally, one feature per branch, babysit CI/Farol, do not merge unless asked, say `restarts-api` when the PR redeploys the API). Release notes (What's New and `/blog` are the same files) are a weekly catch-up, not a per-PR duty. Do not write one in an unrelated PR. When Andre asks, follow `.cursor/skills/whats-new/SKILL.md`. Never edit a note already on `main`.
- Point humans to `docs/DESIGN.md` for the design system (tokens, the `ui/` primitives, the rules the bench enforces, and the `/qa/ui` token sheet under the dev bypass); `docs/I18N.md` for adding UI copy (i18next, `{name}` slots, `_one`/`_other`, Electron menus); `docs/ONBOARDING.md` for every first-run, hint and corner-card surface (one shell, one queue, one store); `docs/HANDLES.md` for public handles, the `/@` profile page and its edge-injected SEO; `docs/CONNECTIONS.md` for Steam / Battle.net / Twitch linking; `docs/NOW_PLAYING.md` for o recado (the free-text status line under a name, `users.custom_status`, broadcast on `profile-update`) and the design note for now-playing music; `docs/DISCORD_IMPORT.md` for copying a Discord Guild Template layout; `docs/WATCH_PARTY.md` for watch party channels (the `watch_party` type, `START_WATCH_PARTY`, the HLS seam, `VITE_WATCH_PARTY_CHANNELS`); `docs/MUSIC.md` for the music queue in a voice call (a synced YouTube player and a shared queue over `/ws`, no audio through our servers, `GET /api/music/resolve`, `Permission.MANAGE_MUSIC` for skip and pause); `docs/JAM.md` for jam channels (songs made of recorded tracks that play in sync: the model, the `jam_channels` flag, the routes, the upload limits and the frames; the wire contract is `packages/shared/src/jam.ts`); `docs/RAISED_HANDS.md` for raising your hand in a voice call (the roster's `handRaisedAt`, the server-stamped queue order, `voice_raised_hands`, and what lowers a hand); `docs/COMMUNITY_HOME.md` for Baú, the durable media feed (`COMMUNITY_HOME_ENABLED` / `COMMUNITY_HOME_VIP_ENABLED`, not `COMMUNITIES_ENABLED`) and `docs/BAU_VIP_STRATEGY.md` for the VIP pricing analysis; `docs/ambient-deploy.md` for character accounts and the ambient-life runner (the house cast); `docs/BOT_SEND.md` for the character HTTP send (`POST /api/channels/:id/messages`); `tools/support-bot/README.md` for the QG support bot, its fact file and the disclosure seam (it is a **disclosed bot**, not a resident, and the QG still gets no AI residents); `docs/CLERK_SETUP.md` for Clerk CLI setup; `docs/STAGING.md` for the staging environment (separate Clerk dev instance, `pqp-api-staging`, Pages branch deploy); `docs/SSO.md` for SAML/enterprise domain joining; `docs/NOISE_SUPPRESSION.md` for the microphone's three noise-suppression settings (the RNNoise worklet, where it sits in the mic chain, and the one line that flips the default); `docs/voice-backends.md` for SFU notes; `tools/sfu/README.md` for the self-hosted LiveKit box (every config file on it, the installer that rebuilds it from a fresh Ubuntu image, and how the API key pair is kept out of git) with the rebuild runbook in `docs/plans/SELF_HOSTED_LIVEKIT.md` §7; `docs/ATTACHMENTS.md` for R2/MinIO setup; `docs/CONTENT_SAFETY.md` for image scanning, what is
 *not* scanned, and the CSAM reporting runbook; `docs/MONITORING.md` for what watches production (GitHub Actions probes, `/ready`, and the Grafana Cloud Loki dashboard and alert rules fed by `tools/log-shipper/`); `docs/EVENT_RUNBOOK.md` for running a watch party as an operator end to end (T-24h/T-1h checks, roles, the merge freeze, the event Grafana alerts and dashboard in `tools/monitoring/`, the `seat-churn` load rehearsal in `tools/watch-party-load/`, and after-party cleanup); `docs/PWA.md` for the mobile/installable app; `docs/ANDROID.md` for the native Android client, why it is native rather than a TWA, and what is actually verified; `docs/ANDROID_TELECOM.md` for the `android.telecom` integration (self-managed `ConnectionService`, the `CallStyle` notification, the FCM contract a ringing call needs beyond an ordinary push); `docs/TESTFLIGHT.md` for the iOS beta / App Review demo account; `tools/admin-dashboard/README.md` for the operator dashboard (`pqp-admin` Worker) and its `ADMIN_METRICS_TOKEN` / Basic Auth secrets.
- Production Postgres: `docs/DB_RUNBOOK.md` for the nightly backup (root cron on the API box, `tools/api-host/db-backup.sh`), how to verify a dump restores, the managed cluster's `trusted_ips` allowlist, the "cluster degraded, restore into a fresh cluster and repoint the API" procedure (written against Fly Managed Postgres, which no longer exists: read it for the order of steps, not the commands), and the `PG_POOL_MAX` connection budget. Never resize the managed cluster during a live event.
- Update `docs/HANDOVER.md` + `docs/PLAN_STATUS.md` when phase status changes.