# Phlix Release Process

## Versioning strategy

Phlix follows [Semantic Versioning (SemVer)](https://semver.org/):

- **MAJOR** version: Incompatible API changes between server and hub
- **MINOR** version: New backward-compatible functionality
- **PATCH** version: Backward-compatible bug fixes

Example: `v1.2.3` where:
- `1` = major version
- `2` = minor version
- `3` = patch version

## Server-hub compatibility

> ⚠ **The matrix previously printed here ("server and hub must have
> matching major versions") was never real and is falsified by the live
> estate:** the shipping pair is server `1.2.3`
> (`src/Common/Version.php::STRING`) × hub `0.5.0` (hub `VERSION` marker),
> and they interoperate. The two products do not share a version space and
> are not released in lockstep, so matching majors is neither required nor
> meaningful until a coordinated 1.0 (pending the owner's
> release-authority decision, same caveat class as the tag table below).

What actually governs interop today:

1. **The shared wire contract.** Both repos vendor the generated contract
   artifacts of the pinned `@phlix/contracts` release — the error-code
   registry (`tests/Fixtures/Contracts/error-codes.json`, asserted by
   `tests/Unit/Contracts/ErrorCodesContractTest.php` on the server; the
   hub carries its own mirror) — and each repo machine-checks its
   `openapi.yaml` against its live router. A contract-breaking change
   requires coordinated releases on both sides, each proven green against
   the pinned contract version in its own CI.
2. **Per-release-pair acceptance.** Until both products reach a coordinated
   1.0, compatibility is asserted by the pair that was CI-tested together at
   release time — not by a version formula.
3. **Version advertisement, not version enforcement.** The server reports
   `Version::STRING` on `/health` and `/system/info` (and via the published
   `VERSION` marker to the update check). There is **no** compatibility
   endpoint — the formerly advertised `/api/v1/compatibility` never existed
   in this repo's router or `openapi.yaml`, and the hub enforces no
   version gate on relay registration. The only runtime version floor that
   *is* enforced is plugin → server: `PluginLoader` validates each
   manifest's `phlix_min_server_version` against `Version::STRING`.
   `composer.json`'s `extra.phlix.minHubVersion` is a declared floor with
   no read sites — a statement of intent, not an enforcement point.

## Release channels

### Stable
- Production-ready releases
- Minimum 2 weeks of beta testing
- Semantic version tags (e.g., `v1.2.3`)

### Beta
- Pre-release testing
- Tagged with `-beta.N` suffix (e.g., `v1.3.0-beta.1`)
- May contain breaking changes

### Nightly (dev)
- Continuous builds from `master` by the `Docker Build & Push` workflow
  (runs on every push to `master`): three legs publish mutable `latest`,
  `intel` and `nvidia` tags plus deterministic immutable
  `<full-sha>-<variant>` tags.
- No stability guarantees.
- ⚠ `nightly-YYYYMMDD` image tags are **NOT-YET-PUBLISHED** — no nightly
  workflow exists and the registry serves none (S483DOCTRUTHX9P2,
  measured against `ghcr.io/v2/detain/phlix-server/tags/list` 2026-09-11).
  The `master`-push regime above is the real continuous-build channel;
  date-stamped nightly tagging awaits the owner's release-authority
  decision and should not be relied on until it ships.

## Release schedule

| Release Type | Frequency | Example |
|--------------|-----------|---------|
| Patch | As needed | v1.2.1 released 2 weeks after v1.2.0 |
| Minor | Monthly | v1.3.0 on 3rd Thursday |
| Major | 6-12 months | v2.0.0 with breaking changes |

## Pre-release checklist

- [ ] All tests pass on `master`
- [ ] Coverage >= the `MIN_COVERAGE` floor declared in `.github/workflows/phpunit.yml` (the live gate; AGENTS.md — never set a coverage demand above measured coverage or every run turns red, making the checklist false-by-construction)
- [ ] PHPStan level 9 clean
- [ ] PHPCS PSR-12 clean
- [ ] Changelog updated
- [ ] Version bumped with `./scripts/release.sh` (see below — it bumps every source at once)
- [ ] Docker images built and pushed
- [ ] Draft release created and reviewed
- [ ] Release notes reviewed

## Release steps

### 1. Start from current master

```bash
git checkout master
git pull
```

Releases land as direct commits on `master`. (This page previously prescribed
a release branch here; no release branch exists in the repo's release history
— v1.2.3 is the direct master commit `c17bb9ec` — and `scripts/release.sh`
itself closes by printing `git push && git push origin v$NEW_VERSION`. The
branch+PR ceremony was fiction and has been corrected to the practiced flow.)

### 2. Update version

**Never edit the version by hand.** Run:

```bash
./scripts/release.sh [patch|minor|major] [--dry-run]
```

`src/Common/Version.php::STRING` is the **authoritative** version source — it is
what the API reports, what `PluginLoader` enforces `phlix_min_server_version`
against, and what the core-update check compares the published marker to.
The script reads it and writes the bumped value to every mirror in one commit:

| Source | Role |
|---|---|
| `src/Common/Version.php` `public const STRING` | authoritative |
| `VERSION` (repo root) | the update marker every deployed server polls |
| `k8s/helm/phlix/Chart.yaml` `version:` + `appVersion:` | Helm chart |

`composer.json` deliberately carries **no** `version` field, and the script will
never add one: the `composer-validate` CI job runs
`composer validate --strict --no-check-publish`, which exits non-zero when the
field is present (that was the decision in `f5375f7a`).

The script refuses to release from a tree where those sources disagree, and
`tests/Unit/Server/Updates/VersionSourcesAgreeTest.php` +
`tests/Unit/Scripts/ReleaseScriptTest.php` keep any drift red in CI.

`extra.phlix.minHubVersion` in `composer.json` is a compatibility floor, not a
version of this package — bump it only when the server actually starts
requiring a newer hub.

### 3. Update changelog

Generate changelog:
```bash
git log --pretty=format:"- %s" v1.1.0..HEAD
```

### 4. Commit and push on master

```bash
git add .
git commit -m "Release v1.2.0: <one-line summary>"
git push
```

### 5. Tag and release

After pushing `master`:
```bash
git tag v1.2.0
git push origin v1.2.0
```

GitHub Actions will:
1. `Release` workflow (trigger `v*.*.*`), job "Verify version sources match
   tag": strip any prerelease suffix from the tag, then assert the
   **authoritative trio that `scripts/release.sh` writes** all match the
   resulting MAJOR.MINOR.PATCH base — `src/Common/Version.php::STRING`, the
   `VERSION` marker, and `k8s/helm/phlix/Chart.yaml`'s `version:` and
   `appVersion:` — and that `composer.json` carries **no** `version` field
   (the `f5375f7a` law above; the assert is law-enforcing, not
   law-contradicting). It then lints and packages **both** charts and
   creates the GitHub release ("Phlix Server <tag>"; `alpha`/`beta` in the
   tag name marks it a prerelease) with the chart `.tgz` files attached as
   **release assets**.
   * The **hub** chart (`k8s/helm/phlix-hub/Chart.yaml`) is packaged but
     deliberately **not** asserted against the server tag: per "Server-Hub
     Compatibility" the two products share no version space, and nothing in
     `scripts/release.sh` bumps it. The previous wording here claimed the
     workflow "asserts both Helm charts' `appVersion`"; that claim — plus
     the matching assert — is exactly why **every** tagged `Release` run in
     this repo's history (v1.2.0 … v1.2.3, attempts included) failed at
     "Assert Chart.yaml appVersion matches tag": the hub chart sat, and
     still sits, at `appVersion: "1.0.0"` under a stale comment that no
     tooling ever honored. A green end-to-end `Release` run requires a real
     tag push, which is owner release authority — the step logic is
     verified by dry-run replay, not by a published run (same disclosure
     precedent as the windows WIN-05 upload leg).
   * It likewise no longer asserts `composer.json`'s `version` equals the
     tag. That assert and this document's own no-`version`-field law were
     mutually exclusive: the check could only "pass" on a tagged tree that
     violated `composer validate --strict` — which is what the v1.2.3 tag
     tree did.
2. `Docker Build & Push` workflow (trigger `v*`): rebuild and publish the
   three runtime image legs with mutable `latest`/`intel`/`nvidia` tags
   plus the immutable `<full-sha>-<variant>` tags for that commit

**Not yet true (pending the owner's release-authority decision):**
no workflow pushes GHCR tags named `v1.2.0` or `v1.2` (the registry serves
zero semver image tags — see the tag table below), and nothing "pushes" a
Helm chart to any repository — charts ship only as release assets. Steps 3–4
as previously worded here described an intended flow that has never run.

## Docker image tagging

Published today (current regime — measured against the live registry, see the
recipe below):

| Tag | Description | Example |
|-----|-------------|---------|
| `<full-sha>-latest` / `-intel` / `-nvidia` | **Deterministic immutable tag per commit and leg** — the form to pin | `ghcr.io/detain/phlix-server:<full-sha>-latest` |
| `latest` / `intel` / `nvidia` | Mutable convenience tags, repushed by every `master` (and `v*`) build — may be stale or move under you | `ghcr.io/detain/phlix-server:latest` |
| `buildcache-latest` / `-intel` / `-nvidia` | BuildKit registry cache, not runnable app images | — |

**Not-yet-published** (no workflow produces them; the registry serves zero of
these; re-check with the recipe below before trusting any change here):

| Tag | Status |
|-----|--------|
| `v1.2.3` (semver image tags) | NOT-YET-PUBLISHED — pending the owner's release-authority decision |
| `v1.2` (minor alias) | NOT-YET-PUBLISHED — same |
| `nightly-YYYYMMDD` | NOT-YET-PUBLISHED — no nightly workflow exists |

To find the tags that actually exist right now (anonymous, no login):

```bash
curl -s "https://ghcr.io/token?scope=repository:detain/phlix-server:pull" \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])' >/tmp/ghcr.token
curl -s -H "Authorization: Bearer $(cat /tmp/ghcr.token)" \
  "https://ghcr.io/v2/detain/phlix-server/tags/list" | python3 -m json.tool
```

Never bake a `<full-sha>` into docs or manifests by hand — query the list and
pick the commit you want.

## Feature compatibility and API versioning

### Feature compatibility

> ⚠ **The per-feature "1.x.0+ / 1.x.0+" floor table that used to sit here
> never matched shipped versions:** pairing, relay tunnels, delegated auth
> and shared libraries all went live while the hub was still on its 0.x
> line (hub `0.5.0` today) against a 1.x server. There is no per-feature
> version floor recorded in either repo. What these features actually
> negotiate is the pinned `@phlix/contracts` wire contract and the
> documented relay envelope — see "Server-hub compatibility" above.

### API versioning

- Server API: `/api/v1/*`
- Hub API: `/api/v1/*`
- Breaking changes increment major version

## Rollback procedures

### Docker image rollback

Rollback targets the immutable `<full-sha>-<variant>` tag of the previous
known-good commit — precise because the current regime publishes one per build
(never `latest`: it is mutable and may already be the broken build). Find the
current tag list with the anonymous recipe in "Docker image tagging" above,
pick the previous run's full SHA, then:

```bash
# Rollback to the previous good build's immutable tag
PREV_SHA=<full-sha-of-last-good-master-build>   # from tags/list, see above
docker pull ghcr.io/detain/phlix-server:${PREV_SHA}-latest

# Update deployment
kubectl set image deployment/phlix phlix=ghcr.io/detain/phlix-server:${PREV_SHA}-latest
```

(`-intel` / `-nvidia` for the hardware-accelerated legs. The Helm chart has
**no default `image.tag`** — an unset value fails rendering via `required`
with this exact immutable form named in the error; the chart does not
mechanically reject a hand-set mutable `latest`, but pinning one defeats
rollback, so don't.)

### Database migration rollback

**There is no automated migration rollback.** The commands this section used
to list — `php scripts/run-migrations.php rollback` and
`php scripts/run-migrations.php migrate:down <version>` — exist nowhere in the
codebase: `scripts/run-migrations.php` parses no arguments at all, so typing
them during an incident ran the **forward** migration anyway — the opposite of
what was asked. `bin/phlix migrate` has no `down`/rollback command either, and
`MigrationRunner` has no reverse path. Stray words passed to
`scripts/run-migrations.php` are silently ignored, not guarded against.
`bin/phlix` behaves the opposite way: it runs on symfony/console (pinned
v7.4.21), so a stray extra word after `migrate` errors non-zero — `No
arguments expected for "migrate" command, got "rollback".` — and an unknown
`migrate:down` errors as an undefined command. Loud, but no rollback either.

Real recovery, in the order an incident should reach for it:

1. **Roll back the image and leave the schema alone.** The
   `schema_migrations` ledger and all applied DDL live in the database, not in
   the image, so a rolled-back image re-runs only its own (older) migration
   files — every one of them is already in the ledger and is skipped. Recent
   migrations have been additive, dead-index drops (e.g.
   `migrations/119_drop_redundant_dead_indexes.sql`), or self-contained
   rebuilds whose own headers analyze the old-code impact (e.g.
   `migrations/120_collation_unicode_ci_conversion.sql`, a ten-table collation
   `CONVERT TO CHARACTER SET` rebuild); read the release's migration SQL to
   confirm before relying on this.
2. **Restore from a backup.** Take one before the release with
   `php bin/phlix backup:create --label=pre-v1.2.3` and find its id with
   `php bin/phlix backup:list`. Restore is admin HTTP API only —
   `POST /api/v1/admin/backup/{id}/restore`, behind `AdminMiddleware` (valid
   admin JWT); there is no CLI restore command. Each archive carries a
   `mysqldump --single-transaction` of all tables, every `*.php` under
   `config/` (recursive), and — when those directories exist — the
   `data/` tree and SSL certs; the stored SHA-256 checksum is verified before
   the import runs. The restore itself re-imports the database dump and the
   config files **only**: `BackupManager::restore()` never touches `data/` or SSL
   files from the archive, so extract those manually if you need them. Honest
   semantics: the dump replaces the tables that existed at backup time
   (mysqldump's per-table `DROP TABLE IF EXISTS` + `CREATE TABLE`) but does not
   drop tables created afterwards, the `schema_migrations` ledger travels
   inside the dump — so once newer code boots, post-backup migrations replay
   FORWARD against the restored schema — and the `backups` history table
   itself is rolled back too (later archives survive on disk/S3 but vanish
   from the list). Pair the restore with the image rollback so the replay
   matches the code.
3. **Remediate forward.** The supported fix for a migration that left the
   schema in a bad shape is a NEW numbered migration that repairs it — the
   ledger records filenames and applies each once, so history is never
   rewritten, only followed up.
4. **Manual reversal, where a migration documents one.** Some migrations ship
   a `## Reversibility` header with hand-written `ALTER` recipes and a
   data-shape safety analysis — `migrations/100_user_item_data_profile_id.sql`
   states in those words that its reverse is *"deliberately NOT automated"*.
   These are best-effort, human-audited escape hatches to run only against a
   verified backup, not a command surface.

## Security releases

For critical security issues:
1. Release patch version within 48 hours
2. No advance notice
3. No beta period
4. Immediate Docker image push
5. Security advisory published

## Hotfix process

For critical bugs in production:

1. Commit the fix directly on `master` (it is always the releasable HEAD)
   and test it.
2. Bump the patch version with `./scripts/release.sh patch`.
3. Tag and push both:
   ```bash
   git tag v1.2.1
   git push && git push origin v1.2.1
   ```

(Former revisions of this section prescribed a hotfix branch, `merge --no-ff`
and branch deletion — the same fictional ceremony corrected at step 1 above.)

## Release integrity (signing: NOT SHIPPED)

> **Correction first.** This section used to open with "All release tags and
> Docker images are GPG signed for security" and offered operator commands
> (`git tag -s`, `cosign sign/verify ghcr.io/detain/phlix-server:v1.2.0`).
> None of it was true on any axis, checked at `dd9e54a8`:
> * *Tags* — `git verify-tag` answers `error: no signature found` for every
>   release tag this repo has ever published (`v1.0.0`, `v1.2.0`, `v1.2.1`,
>   `v1.2.2`, `v1.2.3`); they are all annotated, all unsigned.
> * *Workflows* — no file under `.github/workflows/` contains a GPG, cosign,
>   sigstore or attestation step, and no job requests `id-token: write`, the
>   OIDC permission keyless (sigstore) signing would require. The signing
>   pipeline does not merely not run — it cannot run from these workflows.
> * *Registry* — the OCI referrers API for a published image's digest
>   (probe recipe below) returns `MANIFEST_UNKNOWN`: zero signature or
>   attestation manifests exist beside any pushed image.
> * *The example itself* — `ghcr.io/detain/phlix-server:v1.2.0` is a tag the
>   registry has never served ("Docker image tagging" above); `cosign
>   verify` as written could not have resolved anything to verify.
> Publishing signed artifacts is a deliberate owner-gated gap,
> not an oversight the docs may pretend away.

What IS true today, and what it is actually worth:

| Mechanism | Where | Trust you get | Trust you do NOT get |
|-----------|-------|---------------|----------------------|
| TLS transport + registry digest verification when pinning by `@sha256:…` | GHCR, every pull | content integrity of the pulled bytes | publisher identity |
| Per-run base-digest equality gate | `docker.yml` "Verify the base image is the one built from this commit" (hard error, empty/mismatched digest) | runtime image was built against the base image published in the same run | anything about who ran the build |
| Immutable `<full-sha>-<variant>` image tags, derived from the commit | docker.yml tag table ("Docker image tagging" above) | the image bytes trace to a specific commit on this repo's master | the commit was never signed; the GHCR write token does not carry `id-token` |
| GitHub release with the packaged Helm `.tgz` files attached | `release.yml` "Create Release" | assets are served by GitHub over TLS, at the release URL | no published checksum file, no detached signature — hash each download yourself if this matters |

Integrity checks that work right now:

```bash
# 1. Pin by digest. Resolve a tag to its digest (or use the tags/list
#    recipe above), then pull the digest form. The pull fails loud on any
#    byte mismatch; this is the only cryptographic verification that
#    exists today — of CONTENT, not of AUTHORSHIP.
TOK=$(curl -s "https://ghcr.io/token?scope=repository:detain/phlix-server:pull" \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])')
DIG=$(curl -s -D - -o /dev/null -H "Authorization: Bearer $TOK" \
  -H "Accept: application/vnd.oci.image.index.v1+json" \
  "https://ghcr.io/v2/detain/phlix-server/manifests/<full-sha>-latest" \
  | tr -d '\r' | awk -F': ' '/^docker-content-digest/{print $2}')
docker pull "ghcr.io/detain/phlix-server@$DIG"

# 2. Re-derive a manifest digest locally: GHCR returns the uncompressed
#    manifest (docker-content-digest == sha256 of the body), so
#    GET the manifest and sha256sum the bytes; the two must agree.
# 3. Query whether ANY signature/attestation now exists (a future
#    cosign/sigstore implementation would make this answer):
#    expect "manifest unknown" until it does.
curl -s -H "Authorization: Bearer $TOK" \
  "https://ghcr.io/v2/detain/phlix-server/referrers/$DIG"
```

Do not advertise tags or images as "signed" until the workflows themselves
change. The prerequisites, whichever route the owner picks:

* **Signed tags** — the tagging operator (owner) needs a configured
  GPG identity, `git tag -s` at release time, and consumers need the public
  key out of band. Retagging published versions is history rewriting; the
  cure is forward-only (new tags signed, old ones stay as they are).
* **Keyless cosign (recommended shape)** — add `id-token: write` to the
  docker.yml push jobs' permissions and a `cosign sign` (or buildx
  provenance + attestation) step after each push; no secret to manage,
  signatures recorded in the registry's referrers API, verified with
  `cosign verify --certificate-identity-regexp …`.
* **BYOK cosign** — a stored keypair as a repo secret plus the signing
  step; more key-management surface, works outside GitHub Actions too.