---
name: release
description: Use when cutting, shipping, or announcing a Breeze RMM release — "cut a release", "ship vX.Y.Z", "release notes for X", "what do self-hosters need to know", "draft the release announcement", "do the release write-up". Orchestrates the whole release-comms pass: a self-hoster-focused GitHub Release body (breaking changes, required env vars, migrations), then /update-breeze-release-notes (marketing site), /update-breeze-docs (technical docs), and a Discord announcement. Also covers the post-release deploy + agent-fleet promote steps that the tag alone does NOT do.
---

# Breeze Release

This skill drives the **communication and rollout** pass for a release: the GitHub Release body (especially the self-hoster section), the marketing release notes, the technical docs, and the Discord announcement — plus the post-release deploy/promote steps the tag alone doesn't handle.

It assumes the **code is already merged to `main` and green**. Tagging is the trigger; this skill is about everything that surrounds it.

> **Infra specifics live in `internal/ops/release-infra.md`** (gitignored; read it from the
> MAIN checkout at `~/breeze/internal/ops/release-infra.md` when working in a worktree).
> This skill refers to those values as `$EU_IP`, `$US_IP`, `$EU_FW`, `$US_FW`, `$JUMP`.
> Read that file FIRST when you reach Step 6 — do not guess or reconstruct the values.

## How a release actually fires

`.github/workflows/release.yml` triggers on **`push: tags: v*`**. The tag's *contents are whatever commit it points at*, and the version is the tag minus the `v`. There is no VERSION file — cutting a release **is** pushing the tag.

- CI builds + pushes GHCR images (api/web/portal/binaries) and creates the GitHub Release with `generate_release_notes: true` — i.e. an **auto-generated commit list** ("What's Changed" + Full Changelog). It does **not** write the curated Summary / Added / Improved / Fixed / Security / Self-Hosting sections. **You add those afterward** via `gh release edit`.
- Versions are bare semver everywhere except git tags. The git **tag** is `v0.84.0`; the agent version string, `BREEZE_VERSION`, and ldflags are `0.84.0`.

## Step 0 — Know the blast radius BEFORE tagging

A tag off `main` HEAD ships **all accumulated merged-but-unreleased work**, not just "the latest thing", and **auto-runs every pending migration** on prod via `autoMigrate` on API boot. This has surprised us (the 0.83.1→0.83.2 "hotfix" carried 327 files + 10 migrations). So always look first:

```bash
PREV=v0.83.3            # the last released tag
NEW=v0.84.0             # the tag you're about to cut
git -C ~/breeze fetch --tags
git -C ~/breeze diff $PREV..HEAD --stat | tail -1          # total churn
git -C ~/breeze diff $PREV..HEAD --name-only -- apps/api/migrations/   # migrations that will auto-run on prod
git -C ~/breeze log --oneline $PREV..HEAD                  # everything that will ship
```

- **Full release of everything on main** (the normal case): tag off `main` HEAD.
- **True surgical hotfix**: branch + tag off `$PREV`, cherry-pick *only* the fix. Do **not** tag off main HEAD if you don't intend to ship all of main.

If migrations are listed, you're committing prod to those migrations the moment the new API boots. Read them; confirm there are no large-table rewrites/backfills before you proceed (call those out in the upgrade notes if present).

**Smoke-test privileged migrations as a NON-SUPERUSER role (learned the hard way, v0.95.0→v0.95.1).** CI and the local docker-compose test DB both migrate as the Postgres **superuser** (`POSTGRES_USER=breeze`), which silently passes statements a least-privilege role would be *denied*. Prod is DO-managed **`doadmin` — NOT a superuser**, so a migration that only a superuser can run passes every test and then **crash-loops the API on deploy** (this is exactly what took EU down on v0.95.0: `ALTER FUNCTION ... OWNER TO <role>` requires the *new owner* to hold `CREATE` on the schema; a superuser bypasses that check, doadmin does not → `permission denied for schema public`). So: any migration in the range that does `CREATE ROLE`, `ALTER ... OWNER TO`, `GRANT`/`REVOKE` on a schema, `CREATE EXTENSION`, or a `SECURITY DEFINER` function — **run it against a non-superuser role before tagging.** Two ways: pipe the migration to a real managed DB (or a droplet's `doadmin`) inside `BEGIN; ...; ROLLBACK;` with `ON_ERROR_STOP=1`, or `SET ROLE <nosuperuser-createrole-role>` on a local DB that already has the full schema. If it only works as a superuser, fix it forward (grant the owning role the privilege for the one statement, then revoke) **before** it reaches prod.

## Step 0.2 — Pre-cut gates: the sweep doc and the lab rigs

**Every release ships the agent.** `release.yml` builds and registers the full agent family on every tag, and the fleet promote is a standing rollout step — so "needs an agent release" is never a blocker or a special to-do, it is just what a release is. What an agent-side change *does* need is its real-host check done **before** the tag.

1. Read the newest `docs/testing/release-sweeps/*.md` and its **"Before the release cut"** list, plus `docs/release-notes/next-release-draft.md` → "Release to-do". Any sweep fix PR named there merges before the tag.
2. **Rows a sweep left `BLOCKED` / `PARTIAL` for want of a live agent are run on the SSH lab rigs before tagging — not deferred to after the cut, and not left for a reporter to find.** Use the non-prod rigs (the second Windows Server VM and the KIT `lab-ubuntu-src` VM are lab-enrolled against a local stack; the first Windows VM and the Ubuntu KVM rig are prod-enrolled — leave those alone unless the row is about prod upgrade behaviour). Recipe: `pnpm wt-stack up` at the commit to be tagged → cross-compile the agent family from that commit (`GOOS=… CGO_ENABLED=0 go build -ldflags "-X main.version=<v>"`) → back up the rig's agent config → re-point `server_url` at the Mac's Tailscale/LAN address and re-enroll → run the rows → restore the config. Record results in the sweep doc.
3. A row that still cannot be run (no hardware for it) is carried into the release body's known-gaps note with the exact prerequisite, by name.

## Step 0.4 — Author the in-app "What's New" entry (BEFORE tagging)

The post-upgrade splash (`WhatsNewSplash`, shown on dashboard load) renders an entry from
`WHATS_NEW_ENTRIES` in `apps/web/src/lib/whatsNew.ts` — a list **bundled into the web image
at build time**. No entry for the release = no splash after the upgrade, regardless of what
the server version says. This got skipped for 0.106.0 and 0.107.x, so the splash never fired
in prod even though the feature shipped in 0.105.0.

Ordering is the whole point: this is the **one comms artifact that is pre-tag**. The entry
must be merged to `main` before the tag is pushed, or the build won't contain it. Everything
else (release body, marketing notes, docs, Discord) happens after.

Add a new entry at the **top** of `WHATS_NEW_ENTRIES` (newest-first):

```ts
{
  version: '0.X.Y',   // bare semver — must equal the tag minus the 'v'
  date: 'YYYY-MM-DD', // release date
  title: '<one-line theme — same theme as the release body>',
  highlights: [
    // 2–5 short, user-facing bullets. Technician/MSP language, not commit
    // subjects. Same audience as the marketing notes, tighter than both.
  ],
  learnMoreUrl: 'https://breezermm.com/release-notes',
},
```

- Source the bullets from the same `$PREV..HEAD` sweep as Step 1 — the 2–5 things a
  technician would actually notice in the UI. Skip infra, deps, internal refactors. A
  mostly-internal release can honestly say "hardening and reliability" — or skip the entry
  entirely (skipping is a valid choice, just a deliberate one).
- Sanity-check: `cd apps/web && npx vitest run src/lib/whatsNewState.test.ts src/components/whatsNew/WhatsNewSplash.test.tsx`
- **Already tagged without an entry? Do NOT backdate.** Users' `lastSeenVersion` floor in
  localStorage is already baselined at the version they're running, so an entry versioned at
  or below the shipped release will never auto-show for them. Version the entry as the *next*
  release and fold the missed highlights into it.

## Step 0.5 — After tagging: WAIT for the build, confirm ALL artifacts exist

Pushing the tag kicks off `release.yml`, which is a **long, multi-job build** (~20–40 min): api/web/portal GHCR images, agent binaries for every platform, the **signed** Windows MSI, the **notarized** macOS agent/viewer/helper, watchdog + user-helper, and the signed `release-artifact-manifest.json` (+ `.ed25519`/`.minisig`) and `checksums.txt`. The GitHub Release is created by the **final** job — you cannot `gh release edit` the body until then, and **you must not roll out until every artifact is present**.

A partial or half-failed build can still publish a Release with **missing binaries** — installers and agent auto-update would then 404 in prod. So gate explicitly:

```bash
# Block until the run finishes (re-invokes you on completion):
RUN=$(gh run list --repo lanternops/breeze --workflow Release --branch $NEW --limit 1 -q '.[0].databaseId' --json databaseId)
gh run watch "$RUN" --repo lanternops/breeze --exit-status

# Then confirm success AND a full asset set — compare the count to the previous release:
gh release view $NEW  --repo lanternops/breeze --json assets -q '.assets | length'
gh release view $PREV --repo lanternops/breeze --json assets -q '.assets | length'   # expected baseline (~40)
gh release view $NEW  --repo lanternops/breeze --json assets -q '.assets[].name' | sort
```

Before proceeding, verify the new release has the **same asset families** as the prior one — the macOS `.pkg`/`.dmg`/`.app.tar.gz`, the viewer/helper installers, the `-unsigned` BYO-signing inputs, and `release-artifact-manifest.json` + `.ed25519` + `.minisig` + `checksums.txt`. A short asset list = signing or notarization failed midway.

- **If the run failed**, re-run just the failed jobs (`gh run rerun "$RUN" --failed`) and re-watch. Don't hand-publish a release off a red build, and don't roll out.
- **`gh run watch` can exit non-zero while the run is still in progress** (seen 2026-07-02) — before treating a watch failure as a build failure, check `gh run view $RUN --json status`: `in_progress` means keep waiting (a `status`-polling loop is more reliable than `gh run watch`).
- You *can* draft the release body and the Discord message (Steps 1–5) while the build runs — just don't **publish** the body or **deploy** until this gate is green.

## Step 0.6 — Abuse-asset gate + hosted signing lane (fork-era releases, ≥ v0.105.0)

Since the installer-trust fork, the public release must contain **nothing abuse-usable**, and the signed hosted agent build comes from a separate private lane. Both are release-blocking; the tag lands as a **draft** (`RELEASE_DRAFT_FIRST=true`) and stays one until the hosted cutover completes.

**1. Verify the public draft's assets — bytes and manifest, not job color:**

- Every agent-family Windows exe and the MSI must be **unsigned** (`edition=self-host` in `release-artifact-manifest.json`; zero `edition=hosted` entries). Check the actual bytes: PE security-directory size 0 on the exe, no `DigitalSignature` stream in the MSI, and the `-unsigned` asset byte-identical (`cmp`) to its non-suffixed twin.
- Only Viewer/Helper installers are Authenticode-signed (currently an OV cert — SmartScreen reputation warnings after a cert change are expected, not a failure).
- Spot-check `checksums.txt` against downloaded assets.

**2. Dispatch the private hosted signing lane.** Repo name, dispatch command, edition choice (gap vs strict), and token gotchas live in `internal/ops/release-infra.md` ("Hosted signing lane") — read it, don't reconstruct. Key facts:

- The lane verifies the signed official manifest + tag↔sourceCommit binding before building, rebuilds the agent family as the **hosted** edition (control-plane allowlist baked in), signs with the production provider, and publishes only a **private** GHCR image — never public assets.
- It runs against the **draft** release, so its verify job needs the cross-repo PAT described in the infra doc (draft releases are invisible to read-only credentials — the PAT needs contents **read and write** on the public repo).
- Signing consumes production signing quota: get Todd's explicit go for the dispatch, per run.
- Order (fork runbook): public build green → asset gate above → hosted lane green → **droplet flip** → only then publish the draft → fleet promote. Each arrow is a separate explicit go from Todd — approval for one step never carries to the next.

## Step 1 — Source the content (PRs + git only)

**Read `docs/release-notes/next-release-draft.md` FIRST.** It is the running
scratch list where operator-visible details get written down at the moment
they're introduced — new env vars, new log lines, new metrics, behaviour changes
— rather than being reconstructed from a commit subject weeks later. Fold each
entry into the body below (most of it belongs in Self-Hosting / Upgrade Notes),
then **clear the file in the same PR that publishes the release**, so a stale
entry can't be shipped twice. It is a supplement to the git history, not a
replacement: still do the `$PREV..$NEW` sweep, because anything nobody thought to
write down only exists there.

Release content otherwise comes from **merged PRs and git history only** — never blog posts, design docs, or product docs (those describe *planned*, not *shipped*, work).

```bash
git -C ~/breeze log --oneline $PREV..$NEW
gh pr view <N> --repo lanternops/breeze --json title,body
git -C ~/breeze log --format="%B" -1 <SHA>   # expand a squash-merge body
```

Skip from user-facing notes: `chore(deps):`, `chore:`, `ci:`/`fix(ci):`, internal `refactor:`, internal `docs:`/`test:`. Keep anything a self-hoster or operator would notice.

## Step 2 — Write the GitHub Release body

The GitHub Release is the **operator/self-hoster source of truth**. Model the body on the most recent release for structure and tone:

```bash
gh release view $PREV --repo lanternops/breeze   # reference template
```

Structure (omit empty sections):

```markdown
Breeze RMM **vX.Y.Z** — one-line theme of the release.

## Summary
- **Area** — what shipped, in operator language (#PR).

## Added
- **Feature** — what it does for the user (#PR)

## Improved
- ... (#PR)

## Fixed
- ... (#PR)

## Security
- Hardening summary; "Self-hosters are encouraged to upgrade." (#PRs)

## Self-Hosting / Upgrade Notes
<the most important section — see below>
```

### The Self-Hosting / Upgrade Notes section (do not skip)

Self-hosters run their own droplets and read this section to decide whether an upgrade is safe and what they must touch. Always answer these four questions explicitly, even when the answer is "nothing required" — silence reads as "I don't know":

1. **Upgrade command.** Give both paths, because the right one depends on how the install pins its images (`grep '^BREEZE_API_IMAGE_REF=' .env`):
   - **Digest-pinned** (value ends in `@sha256:…` — every `guided-setup.sh` install for v0.112.0+): `bash guided-setup.sh --upgrade 0.X.Y` (fetch the current script from `main` first). It verifies the signed image inventory and rewrites `BREEZE_VERSION` and all four `BREEZE_*_IMAGE_REF` digests together, then pulls and restarts.
   - **Tag-form** (value ends in `:${BREEZE_VERSION}`): bump `BREEZE_VERSION`, then `docker compose pull api web portal && docker compose up -d` (and `pnpm install` if they build from source).

   Link the [Upgrade Guide](https://docs.breezermm.com/deploy/upgrades/#docker-compose-upgrade). **Never write "set `BREEZE_VERSION`, then pull" as the only upgrade line.** On a digest-pinned install that re-pulls the images already running, so nothing upgrades. Until v0.118.2, `/health` also reported the edited version because Compose copied `BREEZE_VERSION` into `APP_VERSION` (#7024), so operators believed they had upgraded. Only the web footer showed the real version. Every release note from v0.112.0 to v0.118.2 gave that line, and self-hosters were stranded on 0.115→0.116 and 0.117→0.118.2. Keep `portal` in the tag-form pull list: it is a separate container and was left behind for 11 days (stuck on 0.94.0 while api/web ran 0.98.1) when the line pulled only `api web`.

2. **Database / migrations.** State the count and that they're idempotent and auto-apply on boot via `autoMigrate` (unless `AUTO_MIGRATE=false`). **Explicitly flag any large-table rewrite or backfill** — that's the difference between a 2-second upgrade and a stalled boot. If nothing: "**Database — nothing required.**"
3. **New required environment variables.** Anything the config validator now refuses to boot without (e.g. past examples: `RELEASE_ARTIFACT_MANIFEST_PUBLIC_KEYS`, `IS_HOSTED`). A new required env var must *also* be mapped in the `api`/`web` `environment:` block of their compose, not just `.env` — say so. If none: "**No new required environment variables.**"
4. **Behavior changes & feature flags.** Anything whose default changed (call out grandfathering of existing orgs), and any feature gated behind a flag (e.g. `PUBLIC_ENABLE_EDR_INTEGRATIONS`) — name the flag and its default.

End with the compare link: `**Full Changelog:** https://github.com/LanternOps/breeze/compare/$PREV...$NEW`

### Breaking changes

If there are any, set the tone up front (a bolded **⚠️ Breaking changes** callout near the top *and* its own bullet in upgrade notes). Be concrete: what breaks, who is affected, what action they must take, and whether there's a grandfather/opt-out. If there are none, say "No breaking changes." somewhere in the upgrade notes — self-hosters look for that reassurance. Most Breeze releases are non-breaking; behavior-default changes (like the PAM UAC opt-in flip) are the usual "soft" breaking case and must be flagged.

### Publishing the body

CI already created the release with the auto-generated "What's Changed". **Preserve that** — append your curated sections above it rather than clobbering:

```bash
gh release view $NEW --repo lanternops/breeze --json body -q .body > /tmp/auto-notes.md
# Write curated sections to /tmp/curated.md, then:
cat /tmp/curated.md /tmp/auto-notes.md > /tmp/release-body.md
gh release edit $NEW --repo lanternops/breeze --notes-file /tmp/release-body.md
```

## Step 3 — Marketing release notes

Invoke **`/update-breeze-release-notes`**. That skill owns the breezermm.com content schema, the dev→user language translation, and the `categories` grouping for major releases. Don't duplicate its rules here — hand it the version range and let it work. (Its notes are for MSP owners/technicians; tone is lighter and more marketing than the GitHub body.) Remember the site only publishes via its deploy script (`npm run deploy:notify`) — a commit alone changes nothing live.

## Step 4 — Technical docs

Invoke **`/update-breeze-docs`** in diff mode for the version range (e.g. "update docs since $PREV"). It maps changed code → affected docs pages, filters by audience, and bumps its review tracker. Required env vars, new feature workflows, and behavior changes from this release should land in the Starlight docs.

## Step 5 — Discord announcement

Draft a short, friendly announcement for the community Discord. Keep it scannable — highlights, not the full changelog — and **always** include the self-hoster upgrade line and any breaking/behavior change. Show it to Todd before posting; don't post it yourself unless asked.

Template:

```
🚀 **Breeze vX.Y.Z is out**

<one-line theme>

**Highlights**
• <feature 1>
• <feature 2>
• <feature 3>
• <security note, if any>

**Self-hosting:** <one line — "`guided-setup.sh --upgrade X.Y.Z` (or bump `BREEZE_VERSION` + pull on tag-form installs), N idempotent migrations, no new env vars" OR the specific action needed>
<⚠️ one line if there's a breaking/behavior-default change>

📝 Full notes: https://github.com/LanternOps/breeze/releases/tag/vX.Y.Z
```

Tone: operator-to-operator, not corporate. Lead with what people can do now. If a release is mostly internal/security, say so plainly ("mostly hardening + reliability — recommended upgrade").

## Step 6 — Roll it out

The tag built images and registered binaries; it does **not** deploy prod or update the agent fleet on its own. The intent **is** for cutting a release to flow straight into rollout — but **always confirm with Todd before touching prod**, then proceed. Three parts, per region: **back up the DB → deploy the droplet → (later) promote the agent fleet**.

**Read `internal/ops/release-infra.md` now** (main checkout: `~/breeze/internal/ops/release-infra.md`) for the real droplet IPs (`$EU_IP`/`$US_IP`), firewall IDs (`$EU_FW`/`$US_FW`), and jump-host name (`$JUMP`) used below.

### Prerequisite — get SSH access to the droplets

Two access paths, in order of preference:

1. **Jump host** `ssh -J $JUMP root@<ip>` — works when the bastion is up **and you can complete its browser re-auth**; the bastion's SSH is posture-gated with an interactive check, so **headless sessions usually cannot use it** — if it prompts with a login URL or hangs, don't burn time, fall back to (2).
2. **Direct SSH + a temporary DO firewall exception.** Each droplet sits behind a DO cloud firewall that only allows port 22 from an allowlisted IP. Add your current IP, deploy, then **remove it afterward** to restore posture:

```bash
MYIP=$(curl -s4 ifconfig.me)
for fw in $EU_FW $US_FW; do
  doctl compute firewall add-rules $fw --inbound-rules "protocol:tcp,ports:22,address:$MYIP/32"
done
# ... do the rollout via  ssh root@$EU_IP  /  ssh root@$US_IP ...
# then restore:
for fw in $EU_FW $US_FW; do
  doctl compute firewall remove-rules $fw --inbound-rules "protocol:tcp,ports:22,address:$MYIP/32"
done
```

`add-rules`/`remove-rules` are additive/surgical — they won't touch the existing allowlisted IP. Remove your exception once both regions are healthy and promoted (an RMM's prod SSH shouldn't stay open to a residential IP). If the fleet-promote step is deferred pending Todd's go-ahead, remove the exception anyway and re-add it when he approves — it's two commands.

### Back up the managed DB FIRST — before each region's deploy

Migrations auto-run the moment the new API boots (`autoMigrate`), so the only safe rollback artifact is a snapshot taken **immediately before** deploying that region. Prod uses **DigitalOcean managed Postgres** (no `breeze-postgres` container on the droplet) — back it up with the on-demand encrypted off-region wrapper, which sources `backup.env`, `pg_dump -Fc`s the live DB, gpg-encrypts it, and uploads off-region:

```bash
# On the region's droplet (do this right before you deploy that region):
ssh root@<droplet> "bash /opt/breeze-backup/run-backup.sh"
```

- The entrypoint is **`/opt/breeze-backup/run-backup.sh`** (the same cron wrapper) — it `set -a; . backup.env` then runs the encrypted-offsite script. Don't call the inner script directly (it needs the env sourced); dumps are timestamp-named (`db_YYYYMMDD_HHMMSS.dump`).
- **`defaultdb` gotcha:** the backup must target the live `breeze` DB via the `DATABASE_URL` in `/opt/breeze/.env`, **not** `doctl databases connection` (that points at an empty `defaultdb`, ~4K dump). `backup.env` is already configured for the right DB — just run the wrapper.
- **Confirm the dump is real** before deploying: the log prints `Database backup complete: …/db_*.dump (NNM)` and ends with `Off-region encrypted backup complete`. Real sizes are tens-to-hundreds of MB — per-region baselines are in `internal/ops/release-infra.md`. A ~4K dump means you hit the `defaultdb` trap; stop and fix `backup.env`.
- This is *belt-and-suspenders* on top of DO's own daily managed backups + PITR, but the explicit pre-deploy dump is the artifact you actually want when a migration goes wrong — don't skip it.
- The backup GPG passphrases live only on the droplets and must already be vaulted offline; without them a backup is unrecoverable. (Full infra specifics: `internal/ops/off-region-backup-runbook.md`, gitignored.)

Do this **per region, in the night-first order below** — back up region 1, deploy region 1, verify, then back up region 2, deploy region 2.

Two parts after backup: deploy the droplets, then promote the agent fleet.

### Region order — deploy the *sleeping* region first

EU and US are separate droplets/DBs. **Deploy whichever region is in the middle of its local night first**, verify it, then do the other. The point is to let any bad upgrade surface against the smallest live audience.

- Roughly: deploy **EU first** during US daytime/evening (EU is overnight); deploy **US first** during EU daytime (US is overnight). Check the current UTC hour and pick the region whose local clock is deepest into the small hours.
- Per region: **back up its DB → deploy → confirm `/health` 200 and spot-check** → only then move to the other region.

**Auto-detect the night region** (DST-aware — EU droplet ≈ Amsterdam time, US ≈ NYC time). Run this and deploy the region it prints first:

```bash
# Picks whichever region's local time is closest to 3 AM (deepest into the night).
eu=$(TZ="Europe/Amsterdam" date +%H); us=$(TZ="America/New_York" date +%H)
# circular distance from 03:00, in hours
d(){ h=$((10#$1)); a=$(( (h-3+24)%24 )); b=$(( (3-h+24)%24 )); echo $(( a<b ? a : b )); }
if [ "$(d "$eu")" -le "$(d "$us")" ]; then
  echo "Deploy EU first (Amsterdam ${eu}:00, NYC ${us}:00), then US"
else
  echo "Deploy US first (NYC ${us}:00, Amsterdam ${eu}:00), then EU"
fi
```

Per droplet:
```bash
ssh root@<droplet> "cd /opt/breeze && \
  cp .env .env.bak-pre-$NEW && \
  sed -i 's/^BREEZE_VERSION=.*/BREEZE_VERSION=0.X.Y/' .env && \
  docker compose pull api web portal && \
  docker compose up -d binaries-init api web portal && \
  docker image prune -af --filter 'until=168h' && \
  docker builder prune -af"
# then verify:
curl -sf https://<region>.2breeze.app/health     # 200 = healthy; check "version" in the JSON
# and confirm migrations applied cleanly:
ssh root@<droplet> "docker logs breeze-api 2>&1 | grep -aE 'auto-migrate' | tail -5"
# expect "[auto-migrate] Applied N migration(s)" and the unprivileged app-user line
# the two prune lines keep stale release images from filling the root disk (US hit 100% twice in Sept 2026)
```

**Then assert version parity across EVERY first-party container — `/health` does NOT cover this.**

`/health` is served by the API, so it reports the new version even when a sibling container was never rolled. Don't eyeball the service list; enumerate what is actually running and compare each tag to `BREEZE_VERSION`:

```bash
ssh root@<droplet> "cd /opt/breeze && set -a && . ./.env && set +a && \
  docker ps -a --format '{{.Names}}\t{{.Image}}' | grep 'ghcr.io/lanternops/breeze/' | \
  while IFS=\$'\t' read -r n i; do t=\${i##*:}; \
    [ \"\$t\" = \"\$BREEZE_VERSION\" ] && echo \"OK    \$n \$t\" || echo \"SKEW  \$n \$t (expected \$BREEZE_VERSION)\"; done"
# every line must be OK. Any SKEW = that service was not rolled — pull/up it and re-check.
# Today that is api, web, portal, binaries-init. It self-updates as services are added,
# which is the point: the hand-maintained list above is what failed before.
```

**Why this check exists.** The deploy line names services explicitly (rather than a bare `docker compose pull && up -d`) for two real reasons: `billing` is built from a local `breeze-billing:local` image that has no registry to pull from, and a bare `up -d` would bounce `caddy`/`redis`/`tunnel` unnecessarily. But that makes the service list a **hand-maintained list that goes stale the moment a new first-party service is added** — and nothing else catches it. `portal` was added in v0.94.0, never made it into the deploy line, and silently sat on `0.94.0` through five releases while `/health` cheerfully reported `0.98.1`; a portal fix shipped in v0.97.0 was invisible in production for 11 days until a customer-facing proposal link surfaced it. Watchtower is **not** a safety net here: it runs with `WATCHTOWER_LABEL_ENABLE=true` and no service carries the enable label, so it updates nothing. The parity check is the backstop that does not depend on anyone remembering to update a list.

### Rolling back is NOT clean if a migration failed mid-set — the partial-migration trap

`autoMigrate` applies each migration file in its own transaction and records success per file. If migration #7 of 10 fails, files #1–6 have **already committed**. Rolling `BREEZE_VERSION` back to `$PREV` restores the old *code* but the DB is now on a **newer-than-$PREV schema**, and the old code can choke on it. This bit us on v0.95.0: the auth-epochs migration committed a `NOT NULL`-no-default column (`refresh_token_families.absolute_expires_at`) before a *later* migration failed; after rolling back, the old version could not `INSERT` a refresh-token family (the column it doesn't know about is `NOT NULL`), so **new logins silently failed while `/health` stayed green**.

So after any rollback that followed a mid-set failure:
1. **`/health` is not proof of recovery.** Test the real write paths — especially minting a refresh-token family (login) — against the actual DB (`INSERT ... ` in a `BEGIN; ... ROLLBACK;`).
2. **Scan for forward columns the old code can't satisfy:** `NOT NULL` columns with no default, or new CHECK constraints, added by the committed-but-newer migrations on tables the old version writes (`users`, `refresh_token_families`, `oauth_*`, `devices`, `audit_logs`).
3. **Un-break minimally:** `ALTER COLUMN <col> SET DEFAULT <sane value>` (keeps `NOT NULL`, lets the old code insert) is the lightest fix; it's inert on the new version (which sets the value explicitly). Note the divergence and **drop the DEFAULT once you're forward again** so the schema matches a fresh install.
4. **The clean alternative is to roll *forward*** to a fixed build rather than back — completing the migration set makes code and schema match. Prefer this when the fix is understood; the pre-deploy dump is the fallback if it isn't.

### Promote the agent fleet — via a DB row change

Prod sets `AGENT_AUTO_PROMOTE=false`, so deploying registers the new binaries in `agent_versions` (`is_latest` stays on the prior version) and devices don't update until you promote. **Promote with a direct DB row change** — this is the method precisely because it **does not require an MFA session**. The API `/promote` endpoint is gated on a platform-admin MFA session (which we can't drive headlessly), so editing `agent_versions` directly is the authorized, MFA-free path. Confirm with Todd first, then run it per region (US and EU have **separate DBs — promote each**).

The droplets have `psql` installed; run against the region's live DB:
```bash
ssh root@<droplet> 'set -a; . /opt/breeze/.env; set +a; psql "$DATABASE_URL" -f /tmp/promote.sql'
```

The promotion is **slot-aware**: it's per `(component, platform, architecture)` and must only demote slots the target version actually covers — a naive `SET is_latest=false WHERE is_latest=true` strands any platform/arch the new version lacks. Correct SQL, single transaction:

```sql
BEGIN;
UPDATE agent_versions a SET is_latest=false
  WHERE a.is_latest=true
    AND EXISTS (SELECT 1 FROM agent_versions t
                WHERE t.version='0.X.Y'
                  AND t.component=a.component
                  AND t.platform=a.platform
                  AND t.architecture=a.architecture);
UPDATE agent_versions SET is_latest=true WHERE version='0.X.Y';
COMMIT;
```

Caveats:
- **Omitting `component`** in the SQL above promotes everything (agent + helper + watchdog + user-helper). Narrow with `AND component='agent'` etc. if doing a partial roll.
- The DB path **skips the `audit_logs` row** the API endpoint would write. The table is hash-chained — do **not** hand-forge an audit row (a bad row corrupts the chain). Just note the audit gap; re-running the real `/promote` endpoint later is idempotent on the version rows and writes the audit entry.
- Verify after: `select component,platform,architecture,version,is_latest from agent_versions where is_latest and edition='<BINARY_EDITION of this droplet>' order by 1,2,3;` — expect the new version across all covered slots, and no slot left with zero `is_latest`.
  - **Hosted:** agent/backup/watchdog on linux amd64+arm64, macos amd64+arm64, windows amd64; user-helper windows/amd64; helper windows/amd64, macos amd64+arm64, linux amd64.
  - **Self-host:** the same set plus whatever the public release ships.
  - **Hosted specifically:** expect `helper` rows for windows/amd64, macos/amd64, macos/arm64, linux/amd64. Zero helper rows means `BINARY_SOURCE=local` sync did not see `/data/binaries/helper` (#6872) — Assist bootstrap is down until it does.

Rolling the fleet is a user-facing change — get Todd's go-ahead before promoting, and don't promote until both regions are deployed and healthy.

**Security releases:** publish the GHSA advisory **after** rollout completes, never before (fix → quiet release → roll out → then publish). Internal cadence docs: `internal/security-disclosure-policy.md` (the decision doc), `internal/security-advisory-queue.md` (what's owed), `internal/security-hardening-tracker.md`.

### Advisory-debt pre-flight — run this FIRST, every release, no exceptions

Not a "security release only" step. Run it before writing the release body, because a pending
advisory changes how *this* release's notes are worded.

```bash
gh api repos/lanternops/breeze/security-advisories \
  --jq '.[] | select(.state=="draft") | "DRAFT \(.ghsa_id) patched=\(.vulnerabilities[0].patched_versions // "?") — \(.summary)"'
```

**GATE — this release does not proceed if a draft advisory's `patched_versions` has already rolled
out.** Publish it first (`internal/security-advisory-queue.md` has the exact commands). Once a fix
ships, it is visible in the diff and the release notes; the advisory is the only thing that turns it
into a *notification*. Holding it protects nobody and leaves self-hosters unaware they're exposed.

This gate exists because the old trailing checkbox ("GHSA published after rollout") **never once
fired**: it sat last, after manual go-ahead-gated steps, so the release session always ended before
rollout finished — and nothing persisted the debt afterward. Two High advisories from v0.69.0 sat
unpublished for 2.5 months and 36 releases. Binding the check to the *next* release works because
releases reliably recur; "after rollout" is a moment nobody revisits.

### Release-notes security sections

Split by urgency — one bullet in the first section carries more signal than a paragraph buried in
forty:

```markdown
### Security — action required
Self-hosted operators on <affected range> should upgrade immediately. <what an attacker could DO>.
(GHSA-xxxx, #NNNN)

### Security — hardening
<routine defense-in-depth, batched, no urgency language>
```

Say what an attacker could **do**, not what the code now does — "could execute arbitrary code as
root on macOS agents with auto-update enabled", not "now verifies the .pkg against the manifest".
Name the affected version range; self-hosters do not know whether they're affected. Full taxonomy
and per-severity clocks: `internal/security-disclosure-policy.md`.

## Quick checklist

- [ ] **Advisory-debt pre-flight run** (`gh api ... select(.state=="draft")`) — any draft whose patched version already rolled out is **published before this release proceeds**
- [ ] `docs/release-notes/next-release-draft.md` read and folded into the body — then cleared
- [ ] `git diff $PREV..HEAD --stat` + migrations reviewed — blast radius understood
- [ ] **Pre-cut gates** (Step 0.2): newest sweep doc's "Before the release cut" list cleared, sweep fix PR merged, live-agent rows run on the lab rigs (not deferred)
- [ ] In-app What's New entry added to `WHATS_NEW_ENTRIES` (`apps/web/src/lib/whatsNew.ts`) and **merged to main BEFORE the tag** — version = bare semver of this release (or deliberately skipped for an internal-only release)
- [ ] Tag pushed (off main for full release / off $PREV for surgical hotfix)
- [ ] **Abuse-asset gate** (Step 0.6): agent-family exes/MSI verified unsigned at the byte level, manifest all `edition=self-host`, checksums spot-checked
- [ ] **Hosted signing lane dispatched and green** (private repo — see `internal/ops/release-infra.md`), with Todd's explicit go; draft stays unpublished until the droplet flip completes
- [ ] **Build finished green + ALL artifacts present** (signed MSI, notarized macOS, manifest+sigs, checksums) — asset count matches prior release; do not publish/roll out before this
- [ ] GitHub Release body written — Summary/Added/Improved/Fixed/Security + **Self-Hosting/Upgrade Notes** (env vars, migrations, behavior changes, breaking changes), appended above auto "What's Changed"
- [ ] `/update-breeze-release-notes` (marketing site) — published via the site's deploy script, not just committed
- [ ] `/update-breeze-docs` (technical docs)
- [ ] Discord announcement drafted, shown to Todd
- [ ] Confirmed rollout with Todd
- [ ] DB backed up per region (encrypted off-region dump) **immediately before that region's deploy** — verified non-trivial dump size (not the `defaultdb` 4K trap)
- [ ] Droplets deployed — **night region first** (back up → deploy → verify), then the other + `/health` green
- [ ] Agent fleet promoted via slot-aware DB row change (both DBs) — only with Todd's go-ahead
- [ ] Firewall exception removed (if the doctl fallback was used)
- [ ] GHSA published *after* rollout (security releases only) — **and its row in `internal/security-advisory-queue.md` moved to Published.** If rollout finishes after this session ends, the queue row is what carries the debt forward; the next release's pre-flight gate will block on it.

## Notes & gotchas

- **Version format:** tag `v0.84.0`; everywhere else `0.84.0` (bare semver).
- **Don't touch `CHANGELOG.md`** — it's abandoned at 0.67.1. GitHub Releases are the changelog.
- **`gh` repo is `lanternops/breeze`** (canonical remote is LanternOps/breeze). Push and PR there.
- **Reference, don't reinvent:** `gh release view <prev-tag>` is the best style guide for the body. Match its section order and tone.
- **Operator language vs marketing language differ:** the GitHub body is for self-hosters/operators (precise about env vars, flags, migrations); the marketing notes (`/update-breeze-release-notes`) are for MSP buyers (benefit-led). Don't copy-paste between them.
- **A release is comms + rollout, not just a tag.** Cutting the release is meant to flow into rollout, but deploy + agent-promote are **confirm-with-Todd-first** and still manual (the tag only built images + registered binaries). Deploy the night region first; promote the fleet via the slot-aware DB row change once both regions are healthy.
- **No infra literals in this file.** Droplet IPs, firewall IDs, bastion names, bucket names, and dump-size baselines belong in `internal/ops/release-infra.md` (gitignored) — this skill is committed to a public repo. Keep it that way when editing.