# Releasing Crewship

Internal runbook for cutting a release. The pipeline is fully automated —
this document explains *what* the steps do, *what* to check, and *how*
to recover when something goes wrong.

## Versioning

Crewship uses [Semantic Versioning](https://semver.org/). Pre-1.0
releases (`0.x.y`) treat **minor bumps as the breaking-change boundary**;
patch releases are backwards-compatible fixes only.

Pre-release suffixes follow semver: `-beta.1`, `-beta.2`, `-rc.1`. The
goreleaser config has `prerelease: auto`, so a tag with a suffix lands
as a GitHub pre-release; a clean `vX.Y.Z` tag becomes a stable release.

## Pre-flight checklist

Before tagging, verify:

- [ ] `main` is green: `go test ./... -count=1 && go vet ./...` and
      `pnpm lint && pnpm build && pnpm test --run`.
- [ ] `gofmt -l internal/ cmd/` is empty for files touched in the
      release window.
- [ ] Fragments are collected: on a release-prep branch run
      `python3 scripts/changelog-collect.py` (`--dry-run` first to preview).
      It moves every `changelog.d/<number>-<slug>.md` to the top of
      `## [Unreleased]`, highest number first, and deletes the fragment
      files; it refuses, changing nothing, on an empty fragment or a missing
      heading. If it stops partway (crash, kill, full disk), just rerun it:
      it records what it collects in `CHANGELOG.md.collect-journal`,
      installs `CHANGELOG.md` durably before deleting any fragment, and the
      rerun finishes from that journal, so no note is lost or added twice.
      If you edited `CHANGELOG.md` in between, it refuses and says what to
      check. Commit the result with
      the version bump below. Fragments
      merged after that commit wait for the next release.
- [ ] `CHANGELOG.md` has a populated `## [X.Y.Z]` section under
      `## [Unreleased]`. Move the entries down, leave `## [Unreleased]`
      empty; `changelog.d/` holds only its `README.md`.
- [ ] `package.json` `version` matches the target tag (without the `v`
      prefix; `0.1.0-beta.1` for tag `v0.1.0-beta.1`).
- [ ] No unresolved major review findings on the release PR; follow [Review before merge](CONTRIBUTING.md#review-before-merge).
- [ ] Goreleaser dry-run passes:
      `goreleaser release --parallelism 2 --snapshot --clean --skip=publish,sign,sbom,homebrew`.
- [ ] The `release` branch is at most a few commits behind `main`
      (`git log --oneline origin/release..origin/main`). The dogfood
      prod VM tracks `release` and a stale branch means you ship from
      old code.

## Execution-authority rollout (#3068)

Before upgrading **each** stage or production instance to the Issue-operation
authority contract, run the new release's read-only doctor against that exact
instance's existing database. A report from a development or demo database is
not evidence for another target. The production owner performs the production
audit and rollout; development agents must not access production.

```bash
DATABASE_URL='file:/absolute/path/to/target/crewship.db' crewship doctor execution-origins --json
```

Review the blocked and uncertain records with the operator before deployment;
the report inventories possible impact, does not authorize work, and does not
certify an upgrade. Unknown legacy queued work requires fresh authenticated
admission. Retain the report privately with the target identity and release SHA.

After deploying the matching backend and sidecar artifacts, recycle each
affected crew's retained container to refresh its bind-mounted sidecar and
credentials. Coordinate a safe idle window first: account for running and
queued work and prevent new dispatch during that window. The command removes
the runtime container; it is not a graceful drain and must not interrupt active
work. Select the target server/workspace in the authenticated CLI, then run:

```bash
crewship crew restart-agents <slug-or-id>
```

The next agent execution recreates the container. Verify an authorized Issue
write before declaring the new authority guarantees active. After a restore,
web-origin recurring templates pause at their next firing until reviewed and
saved/re-enabled in a fresh full authenticated session; see
[restore authorization](docs/guides/backup.mdx#restore).

## Shared dev release build profile

`.env.production` is the single build profile and starts with
`CREWSHIP_BUILD_PROFILE`. On main it is `main`: every experimental artifact
surface is off and the Artifacts nav opens `/pages`. A shared dev release
(a `demo/release-*` branch with a draft PR for CI) changes only this file: set
`CREWSHIP_BUILD_PROFILE=release` and turn on the flags listed in
`RELEASE_FLAGS` in `e2e/build-profile.ts`; the sharing mock stays off.

The profile separates three things that must not be read as one:

| Surface | Production default (`main`) | Shared dev release | Accepted |
|---|---|---|---|
| Artifacts library (which also moves old Pages addresses to `/artifacts/<id>`), chat artifact context and projects, Craft lease, Design | off | on (`RELEASE_FLAGS`) | only on the exact release SHA whose `VALIDACE` records milestones A and B |
| Dashboard (milestone C) | off | off (`UNACCEPTED_FLAGS`) | no |
| Slides and Design System (milestone D) | off | off (`UNACCEPTED_FLAGS`) | no |

A surface moves into `RELEASE_FLAGS` after its milestone is accepted, not to make
it reachable for a demo; the unit test fails if a flag is listed in both, or if
either profile turns on an unaccepted flag. This table covers the build-time
`NEXT_PUBLIC_*` flags. Server-side flags (`crewship feature-flag`, environment
`CREWSHIP_*`) live with each environment and are verified on the deployed slot.

The feature-flag unit tests and the phone browser contract read the profile and
check either build, so CI of the exact release SHA is green without skipping
tests. Anything bound for main (a push to main or a merge-queue ref) must carry
the `main` profile; `lib/__tests__/feature-flags.test.ts` fails otherwise.

## Cutting the tag

```bash
# 1. Confirm you're on main and synced
git checkout main
git pull --ff-only origin main

# 2. Tag (signed if you have GPG configured, plain otherwise)
git tag -a v0.1.0-beta.1 -m "v0.1.0-beta.1: first public beta"
git push origin v0.1.0-beta.1
```

The push triggers `.github/workflows/release.yml`. The tagged commit must
have successful CI, Security and CodeQL main-push runs and belong to main's
history. The workflow builds a multi-platform image candidate, scans it,
boots both platforms and signs its digest. It then builds the final binary
assets once with GoReleaser publishing disabled, verifies checksums and
signatures are present, scans the artifacts and boots the Linux archive.

Only these verified assets are uploaded to a draft GitHub Release, which is
published after upload completes. Image aliases and the atomic stable
Homebrew update follow publication. Homebrew has a separate retryable job.
Package signing credentials are required; a missing key stops publication.
See [the CI/CD runbook](docs/runbooks/ci-cd-implementation-2026-09-11.md)
for exact identity checks and rollout requirements.

## After the release lands

- [ ] Install the binary from the release artifact and run
      `crewship version` — confirm the tag, commit SHA, and build date
      look right.
- [ ] `brew tap crewship-ai/tap && brew install crewship` on a clean
      Mac and run `crewship version` — same check, plus the formula
      published cleanly.
- [ ] Promote the GitHub Release notes (auto-generated from the
      changelog block + commit history) — re-edit if anything reads
      poorly.
- [ ] Bump `## [Unreleased]` in `CHANGELOG.md` to the next planned
      version's heading, push the change.
- [ ] Sync `release` branch from `main` so the dogfood prod VM rolls
      forward: `git push origin main:release`.
- [ ] Announce: short post in the discussion forum + Discord/community
      channel; mention any breaking changes prominently.

## Recovery

**Tag pushed too early / wrong commit.** Delete the remote tag, fix the
issue, retag the right commit:

```bash
git push --delete origin v0.1.0-beta.1
git tag -d v0.1.0-beta.1
# Fix things, then retag from the corrected SHA
```

If goreleaser already produced a GitHub Release for the bad tag, delete
it via the UI before retagging — otherwise the next run errors on the
existing draft.

**Goreleaser fails mid-flight.** Re-run the workflow from the Actions
tab. The build is reproducible (`-trimpath`, fixed `GOFLAGS`), so a
re-run from the same commit produces identical bytes; cosign keyless
signing is the only step that's not perfectly idempotent (it generates
a fresh certificate against the same identity), but that's fine.

**Homebrew formula push fails (403 / permissions).** Check that the
`HOMEBREW_TAP_TOKEN` repo secret has not expired. Goreleaser writes
the formula to a separate repo, so the main release is unaffected —
re-run only the Homebrew step manually if needed.

**Binary doesn't run on a user's machine.** Common causes:

- *macOS Gatekeeper* — binaries are signed but not notarized in v0.1;
  users must `xattr -d com.apple.quarantine ./crewship` after download.
  Documented in the Quick Start.
- *Linux missing glibc* — Crewship is built `CGO_ENABLED=0` so a static
  binary works on any modern distro. If the user is on
  Alpine + musl, the static build still runs.

## Hotfix workflow

When a P0 / data-loss bug lands in a released version, ship a patch
without waiting for the next planned release.

```bash
# 1. Branch from the existing release tag (NOT main — main may have
#    incompatible changes that you don't want in the hotfix).
git checkout -b hotfix/v0.1.1 v0.1.0

# 2. Cherry-pick the fix from main, or write the minimal fix directly
#    on this branch. Keep scope tight — a hotfix is "patch the bug",
#    not "while we're here, also...".
git cherry-pick <sha-of-fix-on-main>

# 3. Tag and push. The release workflow fires off this tag exactly
#    like a normal release.
git tag -a v0.1.1 -m "v0.1.1: hotfix for <one-line summary>"
git push origin hotfix/v0.1.1 v0.1.1
```

Criteria for a hotfix (otherwise wait for the next regular release):

- Data loss, data corruption, or silent data divergence.
- Auth/authorisation bypass.
- Security vulnerability (CVE-grade or equivalent).
- Server fails to start on a version's own DB after a clean install.

After shipping, forward-port the fix to `main` if it wasn't cherry-picked
from there to begin with — otherwise the next regular release ships
the bug again.

**If the hotfix itself breaks something:** retag `v0.1.2` with a
fix-of-the-hotfix. Don't try to "untag" v0.1.1 — once a tag is in
the public release feed, customers have already pulled it. The fix-
forward path is always cheaper than the rollback path.

## Distribution channels

Three concurrent channels serve different audiences:

| Channel | Trigger | Docker tag | Binary | Use case |
|---|---|---|---|---|
| **stable** | clean semver tag (`v0.1.0`) | `:vX.Y.Z`, `:vX.Y`, `:latest` | GitHub Release, Homebrew | Production / default `brew install crewship` |
| **beta** | pre-release tag (`v0.1.0-beta.1`) | `:vX.Y.Z-beta.N` | GitHub Pre-release | Opt-in beta testers (`brew install crewship@0.1.0-beta.1` or `docker pull :vX.Y.Z-beta.N`) |
| **nightly** | successful verified main CI + daily schedule | `:nightly`, `:main-<sha>` | Immutable `nightly-YYYYMMDD-rN` GH pre-release | Internal CI, brave testers wanting trunk |

The `:latest` Docker tag only moves on clean semver tags — pre-releases
must never overwrite `:latest`, or `docker pull crewship` would silently
hand beta to users expecting stable. See `.github/workflows/release.yml`
for the gating logic.

## Migration safety

Migrations are the highest-risk part of every release — they touch
production data and a bad one is hard to undo. Guardrails:

1. **`migration-lint` CI workflow** runs on every PR touching
   `internal/database/migrate.go`. Enforces append-only ordering
   (versions strictly increase, no rename of a version already in
   `main`). The Go test counterpart (`migrate_lint_test.go`) catches
   the same class of mistake locally.

2. **Auto-snapshot before apply** — `database.SnapshotBeforeMigrate`
   takes a `VACUUM INTO` copy of the live DB as
   `<dbpath>.pre-migrate-vN-to-vM-<UTC>.bak` whenever any migration is
   pending. Last 10 snapshots are retained per database; opt out with
   `CREWSHIP_SKIP_MIGRATION_BACKUP=1`.

3. **Forward-only schema changes**. Never `DROP COLUMN` in the same
   release that stops reading it: ship "stop reading" in vX, then
   `DROP COLUMN` in vX+1. The previous-release client must remain
   compatible with the next-release schema for at least one minor
   bump.

4. **Restore-from-backup tests** in `internal/backup/` exercise the
   `restoreBackfill` hook chain — when a customer restores an older
   bundle into a newer schema, every migration between source and
   target gets a chance to populate any new columns.

## Telemetry (crash reporting) — Sentry setup

Crewship ships with Sentry-backed crash reporting on both sides of the
stack:

- **Backend (Go):** wired through `internal/crashreport`, baked into
  the binary at build time via `-ldflags="-X ...DSN=$SENTRY_DSN"`.
- **Frontend (Next.js):** wired through `sentry.client.config.ts` (+
  `.server.ts` / `.edge.ts`), baked into the static export via the
  `NEXT_PUBLIC_SENTRY_DSN` env var that `pnpm build` consumes.

Both streams route to the **same Sentry project** (`your-sentry-org/crewship-backend`)
with a shared DSN secret. The Go events tag `ServerName` with the
anonymous install ID; the JS events read that same install ID from
`GET /api/v1/system/telemetry` so a backend error and a browser error
from the same install group together in the Sentry UI.

The frontend gates `Sentry.init` on the same `app_settings.telemetry_opt_in`
row the Go side reads — when the operator runs `crewship telemetry off`,
the next browser page load fetches `enabled: false` and skips init
entirely. No client-side persistence, no separate browser-side consent
prompt; the CLI is the single control surface. Any fetch failure on
the consent endpoint defaults to NOT initialising (privacy bias).

The runtime default is decided by build flavour
(`crashreport.DefaultOptIn`, keyed on the version stamped at build
time):

- **Pre-release (-beta/-rc) and dev builds: default ENABLED.** There is
  no first-run prompt on `crewship start` — a deterministic default-on
  means a non-interactive deployment (Docker, systemd, CI) doesn't end
  up silently telemetry-less while a release is still baking. On first
  start, if no consent row exists in `app_settings`,
  `crashreport.Init` writes `"1"` and brings the Sentry client up.
- **Stable release versions: default DISABLED, strictly opt-in.** Init
  writes nothing; telemetry stays off until the operator consents via
  the onboarding step (web wizard checkbox / `crewship setup` prompt /
  `--telemetry` flag) or `crewship telemetry on`.
- **Explicit choices are sticky in both directions.** Once a consent
  row exists in `app_settings.telemetry_opt_in` ("1" or "0"), Init
  treats it as the operator's decision and no version default ever
  flips it, including across upgrades.
- **Status visible any time** via `crewship telemetry status` — shows
  enabled/disabled, install ID, and the resolved DSN endpoint host.

### Routing override

Operators who want to route crash data to their own Sentry (or a
self-hosted instance) set the `CREWSHIP_SENTRY_DSN` env var:

```bash
CREWSHIP_SENTRY_DSN=https://<key>@<org>.ingest.sentry.io/<project> crewship start
```

`crashreport.ResolveDSN()` prefers the env value over the ldflag-baked
default. `crewship telemetry status` shows the resolved endpoint host
+ which source the DSN came from. Empty/unset env = vendor default.

### One-time project setup

1. Create a Sentry project (Platform: Go, or "Multi-stack" — the same
   project ingests both the Go SDK and the @sentry/nextjs browser SDK).
   Note its DSN.
2. Add `SENTRY_DSN` to the repo's GitHub Actions secrets. The
   `release.yml` and `nightly.yml` workflows pass it BOTH as a build-arg
   to goreleaser / the Docker image build (Go side) AND as the
   `NEXT_PUBLIC_SENTRY_DSN` env in `pnpm build` (frontend side). Local
   `go build` and PRs from forks leave the DSN empty, so they ship
   telemetry-disabled — same defense-in-depth applies to the JS bundle.
3. **Configure server-side data-scrubbing rules in the Sentry UI.**
   The client-side BeforeSend hook in `sentry_adapter.go` scrubs
   request headers, query strings, request bodies, the User field,
   and several context maps. It cannot reliably scrub free-form
   strings that *we* generate, e.g.
   `fmt.Errorf("auth failed for %s", userEmail)`. The only sound
   defense for that class of leak is regex-based scrubbing at the
   Sentry server.

   Project Settings → Security & Privacy → Data Scrubbing → add:
   - `@email-pattern`     — emails in messages, breadcrumbs, exception values
   - `@password-pattern`  — Sentry built-in
   - `@creditcard-pattern` — Sentry built-in
   - Custom rule: `[Mask] [Message] [^Bearer\s+\S+]` — bearer tokens
   - Custom rule: `[Mask] [Message] [^sk-[A-Za-z0-9]{20,}]` — OpenAI/Anthropic-style keys

   These rules run inside Sentry before the event is persisted; if
   the regex matches, the matched substring is replaced with
   `[Filtered]`. Verify by raising a test error containing the
   pattern and checking the resulting event in the UI.

### What gets sent

Stack traces, exception messages (subject to server-side scrubbing
above), Crewship version + commit, OS name, an anonymous install ID
(random 32-hex, generated on first opt-in, stable across
opt-out/opt-in cycles).

### What is never sent

Workspace data, credential values, request bodies, Authorization
or Cookie headers, query-string secrets, environment variables, the
user's hostname (`ServerName` is overridden with the anonymous
install ID), the Go module list, or any of the runtime/device/culture
contexts that sentry-go's default integrations would normally attach.
See `internal/crashreport/sentry_adapter.go::scrubEvent` for the
backend filter and `crashreport_test.go::TestScrubEvent_DropsLeakyContexts`
for the pinning test.

The **frontend mirrors the same scrubbing posture** in
`sentry.client.config.ts::scrubEvent`: device/runtime/culture contexts
dropped, User cleared, Breadcrumbs.data cleared, Modules cleared. The
pinning test lives in `lib/__tests__/sentry-scrub.test.ts` and runs in
the standard `pnpm test` flow — a future @sentry/nextjs upgrade that
silently un-scrubs one of these fields fails CI rather than leaking
quietly into production events.

## Branch protection

Changes to `main` go through a pull request and the merge queue. Follow
[Review before merge](CONTRIBUTING.md#review-before-merge),
[Review scaled by risk](CONTRIBUTING.md#review-scaled-by-risk) and
[Merge through the queue](CONTRIBUTING.md#merge-through-the-queue) for the
required aggregate checks, review evidence and exact-head submission procedure.
A formal approval count or a green PR run does not replace that procedure or
the queue's checks.

See the [CI/CD runbook](docs/runbooks/ci-cd-implementation-2026-09-11.md) for
exact-SHA release verification and the authorized repository-policy workflow.
When auditing live protection, inspect the branch's
[effective rules](https://docs.github.com/en/rest/repos/rules#get-rules-for-a-branch).
A missing classic branch-protection API response alone does not mean `main`
is unprotected: rulesets also enforce branch policy.

## Cadence

Target cadence in pre-1.0:

- **Beta minor (`v0.x.0-beta.1`)** — every 4–6 weeks while the API
  surface is still moving.
- **Beta patch (`v0.x.y-beta.N`)** — within 1 week of finding a P0/P1.
- **RC (`v0.x.0-rc.1`)** — once no open P0/P1 for 7 days.
- **Stable (`v0.x.0`)** — promote RC after 7-day soak.

## Stability tiers (per-feature, in addition to the version)

Inside a release, every documentation page carries one of five labels. The
label is stored as `stability` frontmatter and rendered beside the page title
through Mintlify's `tag` metadata:

- **stable** — frozen surface; only backwards-compatible changes in
  patches.
- **early** — available and supported, but the contract may shift in minor
  releases.
- **experimental** — research preview; do not build production
  workflows on it.
- **deprecated** — still available for compatibility, with a documented
  replacement and removal path.
- **roadmap** — planned or documented for orientation, but not part of the
  shipped product contract.

`early` replaces the former per-feature label `beta`; `beta` remains a valid
SemVer release-channel name, but is not a feature-stability label. The Crew
Journal, Paymaster, Lookout, Harbormaster, and Backup ship as **stable** in
v0.1.0-beta.1. Episodic memory and Consolidate ship as **experimental**
(packages exist, auto-wiring lands in v0.2). The full matrix and the policy for
pages that cover more than one feature are in
[docs/production-checklist.mdx](docs/production-checklist.mdx#stability-matrix).