# Secret rotation Status: **documented** (v1.3.0) Scope: DOMINUS Cloud — Terraform state, cloud-init secrets, PostgreSQL roles Related: [Terraform (Hetzner)](../../deploy/terraform/README.md), [RTO / RPO](rto-rpo.md), [Incident response](incident-response.md), [Launch checklist](../releases/launch-checklist.md) Every credential of a DOMINUS Cloud deployment is a value that two or three places know about at once: the database, a file on a node, and the Terraform configuration that provisioned it. Rotating one means changing all of them in the same window, and the runbook exists because the order in which that happens decides whether a customer sees an error or a scheduled job does. ## 1. What holds what | Secret | Live location | In the state | Rotate with | |---|---|---|---| | `hcloud_token` | `terraform.tfvars` | no — a provider credential is not persisted | console → new token → `terraform.tfvars` → apply | | `hetzner_dns_token` | `terraform.tfvars` | no — a provider credential is not persisted | console → new token → `terraform.tfvars` → apply | | `dominus` password (`db_password`) | PostgreSQL role `dominus`; `/opt/dominus/backup.env` (`PITR_MANIFEST_PASSWORD`) | yes (rendered `user_data`) | `ALTER ROLE` over SSH + edit `backup.env` | | `backup` password (`db_backup_password`) | PostgreSQL role `backup`; `/opt/dominus/backup.env` (`PG_PASSWORD`) | yes (rendered `user_data`) | `ALTER ROLE` over SSH + edit `backup.env` | | `dominus_app` password (`db_app_password`) | PostgreSQL role `dominus_app`; `/opt/dominus/.env` on the app node (`DATABASE_URL`) | yes (rendered `user_data`) | `ALTER ROLE` over SSH + edit the app `.env` + recreate the containers | | `app_env` (`JWT_SECRET`, `STRIPE_SECRET_KEY`, `API_KEYS`, …) | `/opt/dominus/.env` on the app node | yes (the whole map) | edit the app `.env` + recreate the containers | | B2 application key (`b2_backup.application_key`) | `/var/lib/postgresql/.config/rclone/rclone.conf` on the db node | yes | B2 console → new key → `rclone.conf` | | B2 state-bucket key (`backend.hcl`) | the developer's shell / `backend.hcl` | the state it protects | B2 console → new key → `backend.hcl` | | operator SSH public key (`ssh_public_key`) | `~/.ssh/authorized_keys` on both nodes | yes | `terraform apply` (it is a managed resource) | ## 2. The rule that shapes every rotation **The db node's cloud-init is not re-applied.** `hcloud_server.db` carries `lifecycle { ignore_changes = [user_data] }` (`deploy/terraform/modules/db-node/main.tf`): a new password in `terraform.tfvars` renders a new cloud-init, Terraform sees the difference and deliberately ignores it, because recreating a node that holds the production database is not an acceptable way to change a password. The app node carries the same `ignore_changes` for the same reason (its named volumes hold the data). Three consequences, and every step below follows from them: 1. **Rotation is out-of-band.** The database is changed with `ALTER ROLE` over SSH, and the files on the nodes are edited by hand. A `terraform apply` does not rotate anything; it is not even the right tool for the job. 2. **The state is a secret store.** The state carries the rendered `user_data` of both servers — the three database passwords, the B2 application key and the whole `app_env` map, as they were at provisioning — and **every apply rewrites it**. Anyone who can read the state can read those secrets. It is not a plan cache: it is a secret file with an infrastructure-shaped filename. Access to the state bucket is a credential, its backups are secret backups, and its rotation is the rotation of everything inside it. Note the consequence of the `ignore_changes` above: a rotation performed out of band does **not** update the state (the attribute is ignored), so the state keeps the previous values until the node is replaced. They are worthless credentials by then, but they are still secret material sitting in a bucket. The Hetzner API token is not among them — a provider credential is not persisted in the state — but it is in `terraform.tfvars` on the same machine, so the machine holds the whole set either way. 3. **`terraform.tfvars` is half of the rotation.** A value that lives only in the live node is lost at the next node replacement: the new node is provisioned from `terraform.tfvars` and comes up with the old password. Write the new value into `terraform.tfvars` first, rotate the live side, and the two halves stay in step. Two rules that are not specific to any secret: - **One secret at a time.** Rotate, verify, then rotate the next. A rotation that touches three credentials at once has no bisect. - **Do not revoke before the replacement works.** Create the new key, deploy it, prove it, and only then delete the old one. The Stripe and B2 dashboards both allow several keys to be live at the same time; use that. ## 3. The three PostgreSQL roles | Role | Rights | Used by | Password lives in | |---|---|---|---| | `dominus` | owner of the `dominus` database | the PITR manifest writer, operator `psql -U dominus` sessions | `backup.env` → `PITR_MANIFEST_PASSWORD` | | `backup` | `REPLICATION` only — no read access to the data | the daily PITR cron (`pg_basebackup`) | `backup.env` → `PG_PASSWORD` | | `dominus_app` | the RLS-scoped application role | the api, worker and scheduler containers **and the migration step** (`migrate.sh` inherits the api service environment, so the schema is migrated as the app role) | the app node's `.env` → `DATABASE_URL` | The app must never connect as the owner: an owner bypasses `FORCE ROW LEVEL SECURITY` (ADR-0038/0047), and tenant isolation would rest on application filters alone. The three roles therefore hold three different passwords, and the backup role never shares a credential with either of the others — a leak of `/opt/dominus/backup.env` must not open the data. ### The order 1. **`backup`** — no live traffic depends on it; the only consumer is the 04:30 cron, and a mistake is caught by a job, not by a customer. 2. **`dominus`** (owner) — the same cron writes the PITR manifest with it, and every operator's `psql -U dominus` uses it. 3. **`dominus_app`** — the running stack connects with it. This is the only rotation with a customer-visible window, so it goes last, with the whole chain in one command. The order is "least exposed first, live traffic last": the credentials no request touches are rotated while the only witness is a scheduled job, and the one that interrupts traffic is rotated when the operator is watching for it. ### The commands Generate the new value where it is used. Use a hexadecimal or alphanumeric password: the app's `DATABASE_URL` embeds it in a URL, and a password containing `@ : / ? # %` breaks the URL or the shell quoting of every command below. ```bash ssh root@ openssl rand -hex 24 # 48 hex characters, no quoting hazards ``` Change the role. `\password` prompts for the value, never echoes it and never writes it to the shell history or the process list: ```bash sudo -u postgres psql # interactive session psql> \password backup Enter new password for user "backup": psql> \password dominus Enter new password for user "dominus": psql> \quit ``` (`ALTER ROLE backup WITH PASSWORD '…'` is the scripted alternative: it works, but it puts the secret in the shell history and in the process list of the node.) Update the file the cron reads. It is `0600` and postgres-owned, so the edit must run **as postgres** — a `sed -i` performed as root replaces the file and leaves it root-owned, the cron can no longer read it, and the daily backup dies silently: ```bash sudo -u postgres sed -i \ -e 's|^PG_PASSWORD=.*|PG_PASSWORD=|' \ -e 's|^PITR_MANIFEST_PASSWORD=.*|PITR_MANIFEST_PASSWORD=|' \ /opt/dominus/backup.env ``` Prove both credentials with one run of the backup script: it connects as `backup` for `pg_basebackup` and as `dominus` for the manifest insert — and, when the off-host archive is enabled, it exercises the B2 key too. ```bash sudo -u postgres bash -c 'set -a; . /opt/dominus/backup.env; set +a; /opt/dominus/base-backup.sh' tail -n 20 /var/log/dominus/base-backup.log ``` `base-backup: done` means the `backup` credential works; a `base-backup: PITR manifest recorded` line means the owner credential works too. A missing manifest line with a successful backup means the owner password is wrong — the backup still runs, and the pitr-health alert is the only thing that notices. Now the application role, on the db node and then on the app node. The order inside is deliberate: PostgreSQL authenticates at connect time, so the connections already open keep working after `ALTER ROLE` while every *new* connection with the old password fails. The window is the time between the two commands, so it is kept to seconds. ```bash # db node sudo -u postgres psql -c "ALTER ROLE dominus_app WITH PASSWORD '';" # app node, immediately after cd /opt/dominus sed -i 's|^DATABASE_URL=.*|DATABASE_URL=postgres://dominus_app:@10.0.0.3:5432/dominus?sslmode=require|' .env docker compose up -d --force-recreate api worker scheduler curl -fsS http://127.0.0.1:3000/api/health ``` (`docker compose up -d` alone is normally enough — compose recreates a container whose environment changed — but `--force-recreate` removes the doubt, and the health check is the proof that the new password was accepted.) Finally, write all three new values into `terraform.tfvars` (`db_password`, `db_backup_password`, `db_app_password`) and apply, so the next provisioning carries them: ```bash cd deploy/terraform terraform plan && terraform apply ``` The plan is empty for the nodes — `user_data` is ignored, so the apply changes nothing on the machines and the state is rewritten with the `user_data` it already held. A node replacement from here on comes up with the new passwords; until then the state still carries the old ones, which is why the state bucket stays a secret store and not a history of what used to be true. ## 4. The application environment (`app_env`) `app_env` is rendered verbatim into `/opt/dominus/.env` on the app node (`0600`) and consumed by the api, worker and scheduler containers. The same map sits in the state in full. The three classes in it do not have the same blast radius, and the difference decides when a rotation is acceptable: | Key | What the rotation costs | |---|---| | `JWT_SECRET` (and the SSO client secret) | **Every OIDC session dies.** ADR-0062 signs the SSO session cookie with a key derived (HKDF) from the operator's client secret: change the signing material and every open session fails verification at the same instant — a mass logout, with no per-session revocation list to soften it. `JWT_SECRET` belongs to that class of material: rotate it in a window and expect every browser to land on the login page. | | `STRIPE_SECRET_KEY` | **A restart, and nothing else.** The key is read per request, so a rotated key only needs the containers to pick up the new value: update `/opt/dominus/.env`, `docker compose up -d --force-recreate api worker scheduler`. No session is affected and no customer is logged out. Do not recreate the api in the middle of a checkout — a payment in flight fails. | | `API_KEYS` | **The clients, immediately.** The key is compared verbatim on each request, so the old key stops working the moment it is changed — no restart, no session invalidation. Rotate it when the client (a script, a partner integration) can be switched in the same minute; `API_KEYS` accepts several named keys (`name=key,name2=key2`), so keep the old one alive until the client is switched. **This is the community edition's key (`AUTH_PROVIDER=env`).** In the Cloud edition (`AUTH_PROVIDER=db`) the key material lives in the `api_keys` table, hashed with scrypt, and `API_KEYS` is not read at all: the keys are minted with `dominus keys create --tenant ` and rotated by creating a new key and revoking the old one. | The two names in the first row matter, because the deployment carries both and the application reads only one of them: the SSO session cookie is signed from `AUTH0_CLIENT_SECRET` (ADR-0062 — `.env.example` warns about exactly this), while the API-key path compares `API_KEYS` verbatim (ADR-0017 chose it over JWT bearer tokens, so `JWT_SECRET` is not read by the shipped code at all). The kill-switch for the SSO flow is therefore `AUTH0_CLIENT_SECRET` — check the deployment's `app_env` for it before assuming that a rotation of the other name did the job. The file to edit is the app node's `/opt/dominus/.env`; the same values must be written into the `app_env` map in `terraform.tfvars` and applied, or a re-provisioned node comes up without them. ## 5. The B2 application key Two B2 credentials exist and they are not interchangeable: the PITR bucket key (`b2_backup.application_key` in `terraform.tfvars`, used by the WAL archiver and the base-backup cron) and the state-bucket key (`backend.hcl`). Scope each one to the single bucket it serves, and never use the account master key for either. The PITR key lives in the rclone config in the postgres user's home, so that both the WAL archiver and the backup cron can use it: ``` /var/lib/postgresql/.config/rclone/rclone.conf (0600, postgres-owned) [b2] type = b2 account = key = endpoint = https://s3.eu-central-003.backblazeb2.com ``` Rotate it on the db node, again **as postgres** — a root-owned `rclone.conf` is unreadable by the WAL archiver, and a stalled archiver is a node that silently falls behind on WAL while still looking healthy: ```bash sudo -u postgres sed -i 's|^key = .*|key = |' \ /var/lib/postgresql/.config/rclone/rclone.conf sudo -u postgres rclone lsd b2:/ # proves the credential sudo -u postgres rclone lsd b2:/wal/ # the archive prefix ``` `rclone config update b2 key ` is the alternative, but it rewrites the whole file (and stores the key obscured, in a different format from the one cloud-init wrote) — so the ownership and the mode have to be put back afterwards: `chown postgres:postgres` and `chmod 600` on `/var/lib/postgresql/.config/rclone/rclone.conf`. The endpoint in the config names the region of the **PITR** bucket, which is not necessarily the region of the state bucket (`us-east-005` in `backend.hcl.example`): two buckets, two regions, two endpoints. No restart is needed for the archiver to pick the new key up — rclone is invoked per WAL segment and reads the file every time — but the archiver must be seen to be keeping up: ```bash sudo -u postgres psql -c "SELECT archived_count, last_archived_wal, failed_count FROM pg_stat_archiver;" ``` Then update `b2_backup.application_key` in `terraform.tfvars` and apply, so that a node replacement from here on comes up with the new key (the apply does not touch the node — `user_data` is ignored, and the state keeps the key it already held). The state-bucket key is rotated the same way: create a new B2 application key scoped to the state bucket, put it in `backend.hcl` (or in `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` in the shell), run `terraform init -backend-config=backend.hcl && terraform plan` to prove it can read the state, and only then delete the old key in the B2 console. `backend.hcl` is a secret file and must be gitignored. ## 6. The Hetzner API token `hcloud_token` is the credential that can create and destroy both nodes. It lives in `terraform.tfvars` (not in the state — a provider credential is not persisted), and how it is scoped is the difference between a leak and an incident: - **Project scope.** A Hetzner Cloud API token belongs to exactly one Cloud Project and sees nothing in the others. Put DOMINUS in a project of its own, so a leaked token cannot reach unrelated infrastructure in the same account. - **Permission scope.** A token is either *Read* or *Read & Write*. Terraform needs Read & Write to create servers, firewalls and networks; a *Read* token is enough to review a plan, and it is the right credential for a second pair of eyes that should not be able to change anything. - **Location scope.** The token is needed only where Terraform runs (a plan or an apply) — the operator's machine, or a CI runner that holds it as a secret. It is never needed on a node: the release pipeline reaches the app node with an SSH key, not with the Cloud API. - **Separate the DNS credential.** `hetzner_dns_token` talks to a different API and cannot touch the Cloud API at all. Leave it empty when the A record is managed at an external registrar, and give it only to the process that writes the record. Rotation: ```bash # 1. console: Security → API Tokens → generate a new Read & Write token # in the DOMINUS project # 2. write it into terraform.tfvars terraform plan # a wrong token fails here, before anything changes terraform apply # 3. console: delete the old token ``` The old token is deleted last, after an apply has proved the new one works. The plan is empty (a token is not a resource attribute) — what the plan proves is that the provider authenticated and refreshed the state with the new credential. ## 7. Closing the rotation A rotation is not finished when the new value works; it is finished when the gate says the deployment is still launchable: ```bash # on the app node, with the deployment's environment loaded set -a; . /opt/dominus/.env; set +a npm run launch:check -- --url https://dominus.example ``` `npm run launch:check` is the mechanical half of the [launch checklist](../releases/launch-checklist.md): it refuses to declare a deployment launchable while the legal identity, the metrics token, the backup configuration, the Stripe configuration or the webhook secret are missing or still placeholders. A rotation must not change its verdict — the same exit code as before the rotation, 0 if it was 0. A rotation that turns the gate red is a broken rotation, not a new baseline. The rest of the closing checks: - the health endpoint answers (`curl -fsS http://127.0.0.1:3000/api/health`); - the next PITR run logs a completed base backup, and — if the B2 key was rotated — a completed off-host copy to `b2:/base/`; - `docker compose ps` on the app node shows the containers that were recreated with the new environment; - `terraform.tfvars` and the live nodes agree: the value in the file is the value in use, and the state is where it should be (a remote backend, not a laptop). That last point is the one worth repeating: the Terraform state is rewritten by every apply, so the state is a secret store in practice. Whoever holds the state bucket holds the database passwords, the B2 key and the application secrets as they were provisioned; that is a credential to rotate and to audit, not a file to forget.