# Upgrading a running installation ## The operator's path: `sitesolide upgrade` From 0.3 on, bringing a machine to a new release is two commands, run from the workstation by the person who owns the machine. Read the notes of every release in between first: a release that asks for more than this says so below, under its own heading. 1. **The CLI.** Install the new binary, or `git pull` a checkout and run `bin/test.sh` there. Both read the same `~/.config/sitesolide/config.json`. ```bash curl -fsSL https://github.com/cthiriet/sitesolide/releases/latest/download/install.sh | sh ``` 2. **What would change.** It reads the machine and changes nothing there: ```bash sitesolide upgrade --dry-run ``` Every component is listed `up to date`, `out of date` with what differs and what would run, or `missing`. A component is out of date when what this release would install differs from what is installed, measured on both sides, never assumed from version numbers. 3. **The upgrade.** It redeploys the components found out of date, and only those, one after the other: ```bash sitesolide upgrade ``` Run it again afterwards: every component reads `up to date`, and nothing changes on the machine. Another installation is upgraded with `SITESOLIDE_CONFIG_DIR=` before the command, as for every command. Upgrade the CLI when traffic is lowest: a dashboard, a portal or a Caddyfile redeployed reloads Caddy, which resets the connections being opened at that instant, for about a tenth of a second; a new drop-in restarts it. ### What it compares, and what it runs Each component is checked first, reading only, then redeployed only if the check found a difference, through the very script or `sitesolide deploy` that `sitesolide setup` runs for it. The check is what setup checks for the component (active, enabled, its files present), plus: | Component | Out of date when | Redeployed by | |---|---|---| | Caddy's drop-in | `override.conf` differs from the release's | setup's own step: the drop-in installed, `daemon-reload`, `systemctl restart caddy` | | backups | `backup.js` or one of its three units differs, or restic, the repository or its key is missing | `bin/deploy-backup.sh install`; the timer stays as it is | | egress proxy | `egress.js` or its unit differs | `bin/deploy-egress.sh` | | steward | `steward.js`, its unit or the portal relay's two units differ, or it started before the egress proxy's unit was last laid, or before the backups' folder existed | `bin/deploy-steward.sh` | | dashboard | `sitesolide deploy --dry-run --compare` finds a difference: a file to send or delete, its manifest, a missing unit, its Caddy block | `sitesolide deploy` in `dashboard/` | | collector | one of its two units differs | `bin/deploy-collector.sh` | | gatekeeper | `gatekeeper.js` or one of its templates differs, or the single template from before is still there | `bin/deploy-gatekeeper.sh` | | installer | `installer.js`, its template or its environment file differs | `bin/deploy-installer.sh` | | the Caddyfile | `/etc/caddy/Caddyfile` differs from the release's | `bin/deploy-caddy.sh`, which validates, reloads, verifies and restores on failure | | shared service | the release in service differs from `api/`, file by file, or its unit does | `bin/deploy-api.sh` | | portal | as the dashboard | `sitesolide deploy` in `portal/` | | monitor | `monitor.js`, its unit or its timer differs | `bin/deploy-monitor.sh` | A component built by its script, the steward or the monitor for instance, is measured by the script itself: `--fingerprint` builds it exactly as it would to install it and prints the SHA-256 of every file it would lay, which the check compares with `sha256sum` on the machine, the measure the script verifies after installing. The same sources built by the same Bun give the same bytes; a binary and a checkout built with another Bun may differ, and then upgrade redeploys what they build, once. The order is the one this page has always given: the backup install before the steward, whose unit opens the backups' folder only if it exists when the steward starts; the egress proxy before the steward too, which starts again after it so that it may write the connectors' folder; the root components that only listen before the dashboard, which they accept old and new; the collector, the gatekeeper and the installer after the dashboard; the Caddyfile, the shared service, the portal and the monitor last. ### What it never does - **Install a component.** One the machine does not carry is reported `missing`, and left alone: `sitesolide setup`, run again for that machine, installs it, the backups, the installer and the egress proxy of a machine set up with `--minimal` for instance. - **Touch a secret.** No file of `/etc/sitesolide` is read, no password is drawn nor rotated: those are setup's first install and the dashboard's. - **Force.** A unit or a Caddy block someone edited on the machine stays as `sitesolide deploy` leaves it: a unit is reported and kept, a block stops the component with the differing lines. `sitesolide deploy --force` in that component's folder of a checkout replaces it, on your decision. - **Stop or start Caddy** but through `bin/deploy-caddy.sh` and `systemctl`; never `caddy stop` nor `caddy start`. ### When it stops A failure stops at its component and says which, the end of what its script printed, and the command that shows more, such as `ssh deploy@203.0.113.10 'sudo journalctl -u sitesolide-steward -n 50'`. The components before it are upgraded, the ones after it untouched. Fix the cause, then run `sitesolide upgrade` again: the components already up to date are skipped, and it resumes at the one that failed. ### What stays by hand - **Each site.** A site is redeployed by `sitesolide deploy` in its folder, when you next deploy it; upgrade only brings the platform's own components. The units a release generates reach the apps already deployed without their code through `sitesolide refresh --all`, see [commands.md](commands.md#refreshing-the-apps-units). - **The loopback rule.** Setup lays it; a release that changes it says so in its notes, and `bin/deploy-loopback.sh close` lays it again, from a checkout or from the kit the binary unpacked into `~/.cache/sitesolide/-/`. It changes what every service may reach, and keeps its own safety net. A machine upgrades one release at a time: from 0.1, follow [From 0.1 to 0.2](#from-01-to-02) first, then [From 0.2 to 0.3](#from-02-to-03), then `sitesolide upgrade` for every release after, reading the notes below for each, [Access: one registry](#access-one-registry) first, then [Anyone with the code, from the dashboard](#anyone-with-the-code-from-the-dashboard). ## Every app filters its system calls **An app could make any system call its account was allowed.** The platform's own services ran under systemd's `@system-service` filter; the units generated for projects did not, so a compromised app reached every call of the kernel, the ones that trace a process, mount or load a module included, and any flaw of the kernel behind them. Every generated service unit now carries `SystemCallArchitectures=native`, `SystemCallFilter=@system-service` and `SystemCallErrorNumber=EPERM`: a call outside the set fails with `EPERM`, the process goes on. **What `sitesolide upgrade` brings**: the installer, with the new generator, so that a deployment by token writes the same unit as one by SSH; and the dashboard and the portal, deployed as projects, whose units take the filter then. Every other app takes it at its next deployment. A unit 0.6 or an earlier release wrote is the generator's own and is replaced without `--force`; one edited by hand stays as it is, and says so. **Every app at once, without sending its code**: `sitesolide refresh --all`, after the upgrade, gives every app on the machine the filter, and the port rules too to an app last deployed before 0.6.0. It writes each project's units again from the manifest its last deployment left on the machine, restarts its services and checks them as deploy does, one project at a time, and stops at the first that does not hold. `sitesolide refresh --all --dry-run` first lists the units it would write; see [commands.md](commands.md#refreshing-the-apps-units). **What changes for you.** Nothing for a server in Bun, Node, Python or Go. Each was deployed under the filter on a Debian 13 machine on x86_64, Bun 1.3.14 and 1.4.3, Node 20, Python 3.13 and Go 1.24, serving, writing its data, starting a child process, a thread or a worker. One call was refused: Node asks for memory protection keys as it starts (`pkey_alloc`), which systemd's default action would have answered by killing it, and which `EPERM` turns into a Node that goes on without them. A program that needs a call outside the set, a profiler that traces, gets `EPERM` and says so in its own error; give it that call in a drop-in, see [Isolation](concepts.md#isolation). **Going back.** A drop-in holding `[Service]` and an empty `SystemCallFilter=` lifts the filter from one unit. The previous release's `deploy` reads a unit with the filter as edited by hand and leaves it, unless run with `--force`. ## One project holds its share of the egress proxy **One project could take the egress proxy down for every other.** The connectors' port counted no connections, read an answer as fast as its upstream sent it whether the app read it or not, and read a body as fast as the app sent it whatever the upstream took: thousands of idle connections from one project, one large answer read slowly, or 32 bodies to a slow upstream, ran the proxy out of descriptors or of memory, and every project's egress and connectors with it. The connectors' port now runs on the same raw listener as the CONNECT port, counts its connections with it, and streams both ways under the same byte budgets. A review of the rewrite closed what it brought or kept: a body in one-byte chunks cost the heap twenty times what the budgets counted, an upstream connection given up on stayed open, a `TRACE` went up with the credential the proxy adds and came back echoed, a `GET` or a `HEAD` carried a body an upstream could read as a second request, and a header byte past ASCII was sent twice encoded, on both ports. **What `sitesolide upgrade` brings**: the egress proxy alone, the steward restarted after it as always. Nothing else moves, and nothing on the machine changes form: the connectors, the grants and the audit are read as they were. **What changes for you.** Nothing, unless an app calls a connector: - **A connector call is one connection**, answered `Connection: close`, with a TLS connection of its own upstream, whose certificate is checked against the connector's host name. A project's connections count against its 128 on both ports together. A call is refused 503 past its project's 32 under way, counted until the last byte of its answer and no longer its headers, past the proxy's 256, or while its project or the proxy holds too many bytes nobody has read; an answer whose head does not come within 30 s is a 504, no longer a 502. - **An answer reaches the app as the upstream sent it**, compressed only if the app's own client asked with `Accept-Encoding`, as Bun's `fetch` does and decodes. - **Refused**: HTTP/1.0, 505; a request with both `Content-Length` and `Transfer-Encoding`, 400; `TRACE`, `TRACK` and `CONNECT`, 405. A `GET` or a `HEAD` goes up without a body. **Going back.** `sitesolide upgrade` with the previous binary puts the previous proxy back; nothing it reads changed. ## What a project, a token or a cloned folder could reach A review of the whole platform found ways past its boundaries, and this release closes them. None took more than what the platform hands out on purpose: an app deployed on the machine, a token, a repository someone cloned and deployed over SSH, a page on a sibling subdomain. **Every app could read every preview code.** The gatekeeper wrote the preview locks' fragment, `/etc/caddy/locks/verrous.caddy`, readable by every account, with every code in force in the clear, and a project's service sees `/etc/caddy`. The fragment is now root's and Caddy's group's, `0640`. Any code drawn before this release may have been read by any project: draw each again, below. **An app could listen where others are sent.** The loopback rule judges who connects to a port, never who listens on it. An app could wait on the portal's port while the portal restarted, during an upgrade, and answer every sign-in check as it liked; on the egress proxy's ports, it read every project's outgoing calls. Every generated unit now lets a server listen on its own port alone among Caddy's admin API, 2019, the range 3000 to 3099 and the proxy's 3128 and 3129, and a worker or a scheduled service on none of them (`SocketBindAllow=`, `SocketBindDeny=`); a service's backup command, the manifest's `install` and the extraction of a token's archive too. The slug `public` is refused as well: a request for `.`, an empty first label, landed in the folder a project of that name would have, its manifest and its code served past its portal. **A cloned repository deployed over SSH could reach the machine's files.** `deploy` now refuses, before anything is pushed: - **a secret file that is not the project's.** The unit reads every declared file as root: `dashboard-backup.env` handed the code the backups' bucket keys and passphrase, `cms.env` the secrets of cms. A project declares `.env` and `-` files alone, never the machine's own nor one a more specific site deployed there holds, and a file already on the machine must belong to `site-`; - **a `publicDir` that is a link out of the repository**, which sent `~/.ssh` and served it to anyone; - **a link among the public files that leads out of them**, which Caddy follows on the machine: `robots.txt -> /etc/caddy/cloudflare.env` served the zone's DNS token. `sitesolide remove` deleted every secret file the folder's manifest named, `dashboard.env` included; it now deletes the project's own alone, and says which it kept. **A token could do more than its scope said.** Its deployment could switch the project's domain on, which `sitesolide domain --activate` does only after checking the DNS: the machine's switch now stands. It could create a project under the name of one removed and inherit what the machine still keeps under that name, the snapshots its creator then restores as Admin and the connector grants: no token creates such a name now, `reserved`. Its refused changes of access went to the journal unbounded, and pushed out every change of secrets made in the hour; its accepted ones could fill the access log, which then lets nobody more in for 180 days but the owner over SSH. Both are bounded now. An expired token still counted toward the 200 live ones, so that tokens a second long stopped everyone minting. And the CLI sent its token to whatever address `SITESOLIDE_API` named: it now sends the token it keeps to the dashboard `login` checked it against, and nowhere else. **Smaller ones.** A WebSocket opened from a sibling subdomain reached a restricted site with its visitor's cookie: the portal now wants the site's Origin on a handshake, a GET though it is. Signing in with password access cleared the failures counted against the owner's password; a sign-out minted a ticket for anyone who posted, which signed out of the portal whoever opened it. The dashboard could be framed by a page on another subdomain, read a sign-in body of any size, and the collector, root, followed a link at the place of the dashboard's previous snapshot. The settings file of the backups, written from the *Secrets* page, could name the program root runs as restic. A connector's request body sent in chunks went past its 10 MiB, and a call whose app hung up kept its answer streaming into the proxy. A value with a Unicode noncharacter made systemd drop every variable of its file; a secret file created under the name of an earlier one kept that one's previous version; a project deployed over SSH as `dashboard-backup` listed the backups' keys in its own *Secrets*; a token's label could carry a terminal control. Each is closed, and each component's README says how. **What `sitesolide upgrade` brings**, in its usual order: the backups, whose settings file no longer moves root's programs or folders, the zone read after it, and whose service backup commands listen on none of the machine's ports; the egress proxy, which counts a connector's body as it streams and lets an upstream go when its caller hangs up; the steward, which keeps from every token the name of a removed project the machine still keeps snapshots or grants of, bounds a token's changes of access, counts live tokens alone, refuses the labels and the values above, creates a secret file with no previous version, keeps the machine's files to the dashboard, and reads the portal's answer within a bound; the dashboard, with its sign-in body bounded, its pages refusing to be framed, the collector's new reading, and its own unit with the bind lines; the gatekeeper, which writes the fragment `0640` to Caddy's group, and whose script closes the one in place; the installer, which keeps the machine's domain switch and writes the bind lines, its runs of `install` and of the extraction listening on none of those ports; the shared service, whose only change is a comment of `src/locks.ts`; the portal, with the WebSocket's Origin, the sign-out ticket, and the failures it keeps until the owner signs in or a quiet day passes; the monitor, whose unit reads the zone after `dashboard-monitor.env`, so that a `SITESOLIDE_ZONE` written from *Secrets* never points its probes elsewhere. Caddy's drop-in, the collector's units, the Caddyfile and the loopback rule do not move. No file on the machine changes form: `team.json`, the access registry and its log, the secret files and the backups' index are read as they stand. **Then by hand, once**, after `sitesolide upgrade` has run to its end: 1. **Check the preview locks' fragment.** The gatekeeper's script, which upgrade runs, gives it to Caddy's group and closes it; a machine that never opened a site with a code has none. ```bash ssh 'sudo stat -c "%U:%G %a" /etc/caddy/locks/verrous.caddy' # -> root:caddy 640 ``` Anything else, `644` from an upgrade stopped before the gatekeeper, is closed by hand, the group first, so that Caddy never meets the file closed to it. Nothing reloads, nothing served changes: ```bash ssh 'sudo chgrp caddy /etc/caddy/locks/verrous.caddy && sudo chmod 0640 /etc/caddy/locks/verrous.caddy' ``` 2. **Draw a new code for every site opened with a code.** Every project could read the old ones. The sites whose general access is *Anyone with the code*: ```bash ssh "grep -lE '\"lock\": *true' /srv/sites/*/sitesolide.json" ``` For each, *New code* in its *Access* section, under the unlock, or `sitesolide lock --new-code` in its folder; then send the new link to whoever should still have it. The old link stops opening the site at once. 3. **Deploy every app again**, so that its units gain the bind lines: `sitesolide deploy` in each project's folder, over your SSH, or a deployment with its token. The dashboard and the portal are done, upgrade having just deployed them. A static site has no unit. A unit the previous release generated is recognised as the generator's own and replaced without `--force`. Check one: ```bash ssh 'systemctl cat | grep -c SocketBindDeny' # -> 3, and as many in each . of the project ``` Deploy them after upgrade, not before: the installer from before reads a unit with the new lines as edited by hand, and stops that project's token deployments on `edited-by-hand` until the installer is upgraded. 4. **Check that systemd enforces those lines.** It does through BPF, and a systemd built without it ignores them without a word, the units starting as before: ```bash ssh 'systemctl --version | grep -o +BPF_FRAMEWORK' # -> +BPF_FRAMEWORK; nothing printed, the lines hold nothing on that machine ``` 5. **Check that no project is named `public`**, a name now refused: ```bash ssh 'test -e /srv/sites/public && echo "a project named public" || echo none' ``` One that is there serves its manifest and its code at `.` until it is removed. This release's CLI refuses its manifest, the `remove` command included: run `sitesolide remove --confirm public` in its folder with the previous binary, then deploy it under another slug. **Worth a look**, reading only: the links among the deployed sites' public files, `ssh 'sudo find /srv/sites/*/public -type l -exec ls -l {} +'`. One that leads out of its folder, which the next `deploy` of that site refuses, had Caddy serve its target until now: change what it pointed at, then deploy the site with the link replaced. And the names in *Sites > dashboard > Secrets > dashboard-backup.env*: the bucket's `BACKUP_S3_*`, `BACKUP_ENCRYPTION_PASSPHRASE`, the retention, the reserve and the child's time, nothing else. A name you did not write there was written by whoever held the dashboard's unlock. **What changes for you.** - **A changed manifest no longer needs `--force` for its block.** A block in service that is exactly what the manifest deposited on the machine generates is the generator's own, and `deploy` replaces it, as the installer already did for a token: `upgrade`, which never forces, would otherwise stop at the dashboard, whose manifest gains the headers that refuse framing. A block edited by hand on the machine still stops `deploy` until `--force`. - **Over your SSH, `deploy` may stop on a manifest that deployed before**, saying which file or link, with nothing pushed: `secrets: a file that is not this project's`, `publicDir leads outside the repository`, or `carries links that lead out of publicDir`. Rename a secret file after the project, `.env` or `-`, and move its variables there from *Secrets*; replace a link with the file it points to. A manifest in your sites repository keeps a `publicDir` that leads anywhere, as its `source` already did. `deploy` also reads, before anything is pushed, who owns each declared file on the machine, through `sudo`, never its content. - **`remove` keeps what is not the project's**, and prints each with its reason, `kept: dashboard.env belongs to the machine, root's, and no project reads it`. A file it kept that really was the removed project's, made by hand under another name, you remove yourself. - **A token no longer creates a name a removed project left things under.** The refusal, `reserved`, counts the snapshots and the connector grants left, never what they hold. Deploy it yourself over SSH, which is how a removed project comes back with its data; or withdraw the grants on the *Connectors* page and forget the snapshots by hand ([dashboard/src/backup/README.md](../dashboard/src/backup/README.md#a-name-a-removed-project-left-behind)), and a token may create it again. - **Each app's units change at its next deployment**, without `--force`. A bind refused fails with `EPERM`: a server that opened a second port of the machine's, 2019, 3000 to 3099 or 3128 and 3129, or UDP on its own port, and a worker, a scheduled service or a backup command that listened inside that range. Declare a service of its own, with its own `port`, for what listens there. A port outside those stays free. A unit edited by hand on the machine stays as it is, without the lines, until `--force`. - **A token's deployment keeps the machine's domain switch.** A domain it declares, a new name or alias, stays inactive until `sitesolide domain --activate` in the project's folder over your SSH. - **The slug `public` is refused**, to the owner as to a token. - **A token fills half of the access log at most.** Past half, what it would let in, Can open given or a project created, is refused `not-available` until older rows age out; its removals still work, and people and you are not held back. Its refused changes of access count among its 60 refusals an hour, then `too-many-attempts`. - **Live means neither revoked nor expired**, for the 200 tokens of the machine as for the ten of a person. - **A label with a control character is refused**, C1 included. A label minted before that carries one is printed by the CLI with `?` in its place, when its person leaves. - **A value with a Unicode noncharacter is refused**, U+FDD0 to U+FDEF and the last two code points of every plane, from the dashboard, a token or the CLI: systemd would read none of the file's variables. - **A secret file created from the dashboard or by a token has no previous version**: one kept under its name, from an earlier file, is dropped first. - **A WebSocket on a restricted site needs the site's Origin.** Every browser sends it; a client of your own outside a browser sends it too, or is refused 403. - **A site's sign-in failures** are no longer cleared by password access signing in, only by the owner's own success, or forgotten after 24 hours with none. - **A sign-out with a provider** ends the portal's session of the person who signed out alone, and only the owner's ends whatever session that browser holds; password access is signed out of the site and sent home. A sign-out link opened in another browser ends nobody's session there. - **The backups' settings file** sets the bucket, the retention and the reserve. A `BACKUP_RESTIC`, `SYSTEMCTL`, `SITES_DIR` or any other variable that moved a program or a folder is ignored under systemd, and so is a `SITESOLIDE_ZONE` there. - **The dashboard refuses to be framed**, `Content-Security-Policy: frame-ancestors 'none'` and `X-Frame-Options: DENY`, and a sign-in body over 8 KiB, 413. - **A connector's request body over 10 MiB** is refused 413 as it streams, `body too large` in the proxy's audit, whether it announced its length or came in chunks. **Whoever works with a token installs the new CLI** too, from the same `install.sh` as step 1 above: the token's binding to its dashboard and the prompt that no longer echoes it are the CLI's. `sitesolide login` now writes the address it checked the token against beside it in the vault, `team-token-api`. A vault filled by an earlier `login` holds none, and its token goes to the `api` of the `config.json` beside the vault's folder, as before. A `SITESOLIDE_API` or a `config.json` naming another dashboard is refused `unauthenticated` with the vault's token: `sitesolide login --url` again, or `SITESOLIDE_TOKEN` beside it, as in a sandbox. A vault kept elsewhere, by `vault` or `SITESOLIDE_VAULT`, needs `sitesolide login` once more. **Going back.** `sitesolide upgrade` with the previous binary. The previous gatekeeper writes the fragment `0644` again at its next change of a site's general access or code, every code readable by every project once more. A unit with the bind lines reads as edited by hand to the previous CLI, which keeps it, and to the previous installer, which stops that project's token deployments on `edited-by-hand`: `sitesolide deploy --force` in its folder, with the previous CLI, writes it without them. `team-token-api` stays in the vault, unread. The registry, its log and `team.json` stay as they are. ## Secrets with a token A token may now manage secrets. Minted with *Manage secrets*, it lists its projects' secret files and variable names, and sets and removes the variables of `.env`, never reading one back; any token that deploys a project also restarts it. From a workstation with a token, `sitesolide secrets`, `secrets set`, `secrets remove` and `sitesolide restart` do it, see [access.md](access.md#secrets-with-a-token). **What `sitesolide upgrade` brings**, in its usual order: the steward, which judges the new routes and keeps each token's hourly bounds; the dashboard, with the routes, *Manage secrets* in the *New token* dialog, and *Token activity* reading the steward's journal; the installer, whose refusal of a missing secret now names `sitesolide secrets set`. The backups, the egress proxy, the collector, the gatekeeper, the shared service, the portal, the monitor, the Caddyfile and Caddy's drop-in do not move. No file on the machine changes form: `team.json` keeps its version, and a token minted before reads as without the permission. **A dashboard ahead of its steward**, deployed alone or with the steward's step failed, answers the new routes `not-available`, saying to run `sitesolide upgrade`, and refuses to mint a token with *Manage secrets*, the dialog saying the server's steward predates the permission. Running upgrade to the end mends both. **What changes for you.** One thing right away: **every token already minted restarts the services of the projects it deploys**, a person's own while they are Developer or Admin there, 30 times an hour at most, each restart in the journal under the token. No permission is ticked for it: a token that deploys code already restarts the service at each deployment. Revoke a token whose holder should not restart. Secrets wait for someone to mint a token with *Manage secrets*: every token already minted, yours and everyone's, lists and manages none. Then: - whoever holds that token sets variables without an unlock: ticking the box is that decision. Mint it for the projects that need it, with an expiry; - an owner's token that does not deploy public sites changes the secrets of the Restricted projects it could deploy as they stand alone; - the *Tokens* page's *Token activity* lists, beside deployments, the secrets a token set or removed and the restarts it made, and so does each project's *Activity*; - over your SSH, `sitesolide secrets` and `sitesolide restart` point to the dashboard, as `secrets` did. **Whoever works with a token installs the new CLI** too, from the same `install.sh` as step 1 above: the commands are the CLI's, and the binary they have predates them. A previous CLI refuses them before anything is sent: `sitesolide secrets` as a command that needs the owner's SSH, and `sitesolide restart` as a command it does not know, a usage error. An agent without the CLI calls the routes over HTTPS ([access.md](access.md#over-https-without-the-cli)). **Going back.** `sitesolide upgrade` with the previous binary. The previous steward reads `team.json` as it stands, the new key ignored, and knows no secrets route, which the dashboard answers `not-available`. The values tokens set stay in the files, and their lines in the journal. ## A token's project reads its own secret files alone The installer now refuses, before anything is written, a deployment by token whose project would read or take a secret file that is not its own: `secret-taken`, naming the file, never what it holds. That is a file kept for root (`dashboard-backup.env`, `dashboard-monitor.env`), a declared file already in `/etc/sitesolide` that is not a plain file of `site-`, and a file another site holds that a new project would take by its name, `cms-smtp` taking cms's `cms-smtp.env`. The steward refuses a token creating a project named after a platform project and a dash, `dashboard-*`, `portal-*`, `api-*`, `analytics-*`, `landing-*`, `www-*`, as `reserved`; a project of such a name already on the machine stays deployable. **What `sitesolide upgrade` brings**: the steward, which refuses the new names; the dashboard, whose folder carries the sources of both, with nothing new for it; the installer, which judges the files. The backups, the egress proxy, the collector, the gatekeeper, the shared service, the portal, the monitor, the Caddyfile and Caddy's drop-in do not move. Nothing changes form on the machine. **What changes for you.** A token's deployment that read a file of another project, or of the machine, stops at its next run, and says which file: move the variable into the project's own `.env` from *Secrets*, or, for a file created by hand, give it to `site-` (`sudo chown site-:site-`) if it is the project's. Over your SSH, nothing changes. **Going back.** `sitesolide upgrade` with the previous binary: the previous installer reads the declared files whoever owns them, as before. ## Workers and scheduled services A project can run work that answers no request. Under `services`, an internal service without a `port` is a worker, which runs for good and listens on nothing; a service with a `schedule`, a systemd calendar expression, is started by a timer at the times it names and runs to its end. `cpu` caps a service's processor. A project of such services alone, with no `publicDir`, serves no page and its address says so. See [docs/manifest.md](manifest.md#services). **What `sitesolide upgrade` brings**: the components that read a manifest, rebuilt: the backups, the egress proxy, the steward, the dashboard, the gatekeeper, the installer and the monitor. The Caddyfile, Caddy's drop-in, the collector, the shared service, the portal and the loopback rule do not move. The units and blocks of the projects already deployed are generated as before, line for line: none reads as edited by hand, none needs `--force`. **What changes for you.** - **Upgrade before deploying the first project with a `schedule`.** The previous monitor and dashboard read a scheduled service between two runs, `inactive` by design, as a service down. The new monitor reports a failed run as a warning and a timer that stopped as an outage. - **`deploy` watches the services it restarted for ten seconds**, a server or a worker, and fails when one is no longer active or was restarted by systemd meanwhile. A project that fell at once after its restart, and was restarted every two seconds behind a successful deployment, now fails its deployment: `sitesolide logs` says why. The wait runs beside the Caddy step and the check over HTTPS. - **`deploy` asks systemd on the machine** whether it reads each schedule, before anything is written. - **`sitesolide logs` reads the journal through `sudo -n`.** On a machine `sitesolide setup` installed, it printed "No entries" for every project: the deployment account is in no group that reads the system journal. It needs the account's sudo without a password, which every deployment already does. - **A token deployment no longer chooses a port for an internal service that names none**: it is a worker. One the machine already gave a port to keeps it. - **A project whose services are all internal, with no `publicDir`**, was refused, and now deploys as a project that serves no page. - **Removing a project** clears the failed state of its units, which systemd otherwise keeps in `systemctl --failed` once their files are gone. **Going back.** Remove first, with this release's CLI, every project that declares a worker, a `schedule` or `cpu`: the previous release refuses those keys, and would read a scheduled service between two runs as a service down. Then `sitesolide upgrade` with the previous binary. ## Whose token, from a menu that says who each person is *Whose token*, in the *New token* dialog, is a menu drawn by the page rather than the browser's: each person of People over a line saying where they deploy and whether they may create projects. A Viewer everywhere is no longer offered, since the steward mints them no token; with nobody to offer, the menu is left out. **What `sitesolide upgrade` brings**: the dashboard alone. **What changes for you.** Nothing beyond the dialog. **Going back.** `sitesolide upgrade` with the previous binary. ## A token's projects are ticked, not typed The *New token* dialog lists the projects on the server to tick, where the owner used to type slugs from memory; creating projects is one more row of that list, *New projects*. A permission a chosen project cannot be deployed without is ticked for you and says which project needs it: public sites for a project that is not Restricted, or that exempts paths from the portal, and a domain for one that declares it. A token with nothing to deploy, no project and no *New projects*, is refused by the form. **What `sitesolide upgrade` brings**: the dashboard alone. The tokens already minted are untouched, and the steward judges a new one as before. **What changes for you.** Nothing beyond the dialog. **Going back.** Deploy the previous release's dashboard: `sitesolide upgrade` with the previous binary. The tokens minted with this dialog stay as they are. ## The machine's own state is backed up Every run now takes, first, a snapshot named `machine`: `/etc/sitesolide`, every project's secret files; `/var/lib/sitesolide-steward`, the access registry, the previous versions of the secrets and the steward's log; and `/etc/caddy`, the preview codes, the zone's settings, the DNS token and the domains table. It is read back, kept, copied to the bucket and checked like a project's snapshot. Until now a lost machine took all of it along: the data came back from the bucket, and every key had to be minted again. **What `sitesolide upgrade` brings**: the backup component, which takes it; the steward and the dashboard, which never take `machine` for a site. Check after the next run, or at once with `ssh you@your-machine 'sudo systemctl start sitesolide-backup.service'`: `sudo cat /var/lib/sitesolide-backup/last-run.json` names `machine` first, with a snapshot. Its files are put back by hand, never from a page: [dashboard/src/backup/README.md](../dashboard/src/backup/README.md#the-machines-own-state). **What changes for you.** - `deploy` refuses the slug `machine`. A project deployed under that slug before this release is no longer a site to the steward nor to the backups: deploy it under another slug before upgrading, and remove the old one. - The monitor's backup line counts `machine` among the projects, and names it when its snapshot fails. - The secrets now sit in the bucket too, encrypted with `BACKUP_ENCRYPTION_PASSPHRASE`: whoever holds that passphrase and read access to the bucket holds every secret of the machine, as they already held every project's data. **Going back.** Forget the snapshots of `machine` first, in the server's repository and in the bucket's: the previous steward takes any deployed folder for a site, `machine` included, and a project deployed under that slug with the previous CLI would list them and could restore every secret into its own data. ```bash ssh you@your-machine sudo -i export RESTIC_REPOSITORY=/var/backups/sitesolide-restic RESTIC_PASSWORD_FILE=/var/backups/sitesolide-restic.key restic forget --prune $(restic snapshots --path /machine.tar --json | python3 -c 'import json, sys; print(" ".join(s["id"] for s in json.load(sys.stdin)))') ``` Then the same with the bucket's repository, as [Restoring by hand](../dashboard/src/backup/README.md#restoring-by-hand-when-the-dashboard-does-not-answer) opens it. The previous component takes no new snapshot of `machine`. ## A site closes on every address Restricted and Anyone with the code now close a site on its own domain too, not only on its preview: the dashboard and `sitesolide lock` used to refuse a site that served its own domain, since the domain would have stayed open. The domain's other names, its aliases and the `www` the table adds to a second level domain, now send to the domain with a permanent redirect. They used to fall into Caddy's nameless block, which serves `public/` and never wakes an app: an app on its own domain answered its `www` without its service, every `/api/*` there a 404. **What `sitesolide upgrade` brings**, in its usual order: the steward and the dashboard, which offer all three accesses to a site on its own domain and name both addresses before a change; the gatekeeper, which checks the domain beside the preview after every change, and writes an app's block anew when the one in service is an earlier release's; the Caddyfile, whose nameless block lets a static site's code close its domain; and the portal, which judges a domain's visitors on its site's people. Two changes reach sites at once, and nothing else does until a site is deployed or its access changed: - the nameless block no longer serves an app's folder: a name the table carries for an app and its block does not claim answers 404 instead of the app's `public/`, which skipped its portal. The `www` of an app deployed before this release is one such name, until the app is deployed again; - the preview locks in service keep matching their previews alone; the gatekeeper writes all of them in the new form at its next change of any site's code. **Then each app site that serves its own domain**, when traffic allows: `sitesolide deploy` in its folder. It says the block "was written by an earlier release, the current one replaces it", without `--force`. Check: `curl -sI https://www./` answers `301` to `https:///`, and `https:///` answers as before. Until then its `www` keeps falling into the nameless block; restricting it or giving it a code from the dashboard writes the new block in the same change. **What changes for you.** - **A site that opens with a code and serves its own domain** stays open there until its stanza is written again: `sitesolide lock` in its folder, or *New code* in its Access section, after the deployment above. Either rewrites every site's stanza at once. The dashboard no longer calls such a code in place. - `sitesolide domain --activate` on a restricted site, or one that opens with a code, no longer refuses the restricted one, and refuses either while what Caddy serves would not close the domain: an app's block other than the one its manifest generates (deploy it first), a Caddyfile from before (upgrade first), or a code's stanza from before (`sitesolide lock` first). - A deployment refuses a domain name, an alias or their `www` that another project declares, active or not, and an alias in the served zone. - The link that carries a site's code leads to its own domain once it serves one. - `bin/lock.sh state`, which `sitesolide lock --status` runs, measures the domain with the code too. - An app reading `X-Sitesolide-*` behind the portal receives them on its domain as on its preview. **Going back.** A portal from before refuses the announcement of a domain's block, so a restricted site on its own domain is closed to everyone there, its preview unaffected: make it public before going back, or upgrade again. The previous gatekeeper refuses to change the access of a site whose block is this release's, as one it did not write: deploy such a site with the previous release first. The redirect of a domain's other names stays until the site is deployed with the previous release. ## Access: one registry Who may do what on a project now lives in one place, the steward's access registry, and the portal reads a projection of it. `sitesolide upgrade` is all it takes; read this first, for what changes for the sites and for you. **Before you run it.** - **An app that reads `X-Sitesolide-Role`** must be updated, and deployed, before the portal is: the header now carries the role, `admin` for the owner's password and the admin emails, `visitor`, `viewer`, `developer` or `admin` for someone signed in with a company account, `visitor` for password access, which carries no `X-Sitesolide-User`. Its old values, `member` and `guest`, are gone. An app that let `guest` read only now tests `role === "visitor" && user === null`, or better, the role it means. - **Everyone with a role above Can open on a project opens its site** when it is restricted: a Developer of a restricted site no longer needs Can open besides. Read each restricted site's people with access afterwards, `sitesolide share` in its folder. - `sitesolide members` is gone: `sitesolide share --role ` from a project's folder gives a role, `sitesolide people` lists everyone and `sitesolide people --may-create` gives the right to create projects. `sitesolide share --domain` and `--only-admins` are gone too: a domain is written `@acme.com`, and general access, public or restricted, is the dashboard's. **What changes in the dashboard.** - **One place per question.** In a site, one *Access* section replaces its *Access*, *Sharing*, *Guests* and *Members*: its general access first, Public, Restricted or Anyone with the code, then its people with access, each with a role, and *Add people*. At the machine level, *People*, the owner's, replaces *Members*: everyone across projects, their roles, who may create projects, the domains; *Tokens* replaces *Team*, each token saying who made it. The old addresses, bookmarks included, lead to the new pages. - **Password access waits for the unlock** when it is given from the dashboard, as a role above Can open does: the owner's password, or an Admin's forced sign-in, since it lets in someone from outside the company. So does a whole domain while `OIDC_ALLOWED_DOMAINS` is empty. Can open for a company account or one of the listed domains, lowering and removing still never wait, nor does restricting a site, and the owner's `sitesolide share` over SSH asks for nothing. - **Someone who can only open sites is told so** when they try the dashboard, `can-open-only`, apart from someone on no list, `no-role`. - **Activity** lists a change of general access as `access.general`, where it said `door.update`: a filter or an export that looked for the old name looks for the new one, or for `access.` to find every change of access. The rows written before the registry still read, in today's words. - **Changes of access are kept 180 days, and bounded.** The steward writes every accepted change of access to a file of its own, `access-log.jsonl` beside its journal, seeded from the journal and topped up at each start with what it lacks; refusals stay in the journal. Its rows are never pushed out before 180 days: once it holds 20,000 of them, or 12 MB, a change that lets more people in is refused, `log-full`, until older ones age out, while removing and lowering still work, and you still change access from your workstation with `sitesolide share` and `sitesolide people`. Such changes are also counted, 120 an hour for each person, each token and you in the dashboard, 2,000 for you over SSH, `too-many-changes` past it. - **Every token from before becomes someone's**, once, at the steward's first start. A token whose email, what you typed when creating it and shown as its label, is the address of a person who signs in to the dashboard, a role above Can open or the create right, becomes that person's, made by you: it is narrowed to their roles from then on, and revoked when they leave, removed or lowered so that they keep no role above Can open and no create right. Every other token becomes your own, its scope unchanged. To keep a shared CI token yours when its email is a person's: after upgrading, on the *Tokens* page, make a token with *Whose token* set to *Mine* and the same projects, put it in the CI, then revoke the one that became the person's. **What `sitesolide upgrade` runs, in this order.** 1. **The steward.** At its first start it makes the registry, once, from `members.json` and the portal's database, then writes the portal's projection: [migration.md](migration.md#access-the-registry-made-from-the-stores-before-it) says what it carries over, what it sets aside and why. The running portal, the old one, keeps deciding from its own tables, which still say the same. `bin/deploy-steward.sh` ends on `access registry 600 root:root, its projection for the portal 640 root:site-portal`. A store that does not read leaves no registry, and is tried again later; when it is the portal's database, a table past 16 MB (`oversized-table`) among others, `sitesolide people --migrate-without-portal` carries the rest over without it, and the tokens are made someone's at once. 2. **The dashboard.** Its access API speaks to the steward, and its pages are the ones above: a site's Access, People and Tokens. 3. **The installer**, which reads the registry when it starts. 4. **The portal**, last: from its first request it decides from the projection, and leaves the mark that keeps it from ever reading its old tables again. Run them together, as `sitesolide upgrade` does: a change of access made in the old dashboard between the steward's step and the dashboard's would go to the portal's old tables, which nothing reads any more. **Check.** ```bash sitesolide people cd && sitesolide share ssh deploy@203.0.113.10 'sudo curl -s http://127.0.0.1:3026/admin/access' ``` `people` lists everyone with their roles; `share` lists the site's people with access and says nothing of the portal; the portal answers `"reading": "steward"`. A password the portal handed out before the upgrade still opens its site, as password access, and a cookie set before it is still valid. In the dashboard, a site's *Access* lists the same people as `share`, and *People* the same as `people`. **Going back** is the previous commit of the steward, the dashboard, the installer and the portal, which read the old stores as they stood at the migration: every change of access made since is lost for them. See [migration.md](migration.md#going-back). **Deleting `access.json` is no way to start over.** The steward makes it again at the next request from the old stores, as they stood at the migration, so every change of access made since is gone; within the next 30 seconds its sweep revokes the live tokens of everyone the rebuilt registry gives no rights. A deletion made by mistake is undone by putting back a copy of the file, root `0600`: `ssh deploy@203.0.113.10 'sudo install -m 600 -o root -g root /var/lib/sitesolide-steward/access.json'`. Everyone's roles and access come back with it; the changes made to the rebuilt registry in between are lost, and the tokens the sweep revoked stay revoked, to be minted again. The machine's backups hold the steward's state from 0.4.2 on, an hour old at most; make a copy yourself all the same, `sudo cp -p` of the file, before you edit it by hand. ## Anyone with the code, from the dashboard A site's general access is now chosen among its three from its *Access* section, Anyone with the code included: the owner and the site's Admins open a site with a code there, read the code, draw a new one, and go back to Public or Restricted, a switch between Restricted and the code in one transaction. `sitesolide lock` and `unlock` no longer change the machine themselves: they ask the steward on its owner socket, which launches the same gatekeeper. One path changes a site's general access, whoever asks. **What `sitesolide upgrade` brings**, in its usual order: the steward and its `POST /general` routes, the dashboard, then the gatekeeper and its four templates, `-on@`, `-off@`, `-code@` and `-renew@`. `bin/deploy-gatekeeper.sh` lays what they open for writing and cannot create, when missing: `/etc/caddy/locks/`, an empty `/etc/caddy/locks-codes.json` for the deployment account in `0600`, and `/srv/garde/`. A machine that has locked a preview before carries them already, and they are left as they are, codes in force included. The Caddyfile changes by a comment only, which upgrade redeploys through `bin/deploy-caddy.sh` all the same. **Until the gatekeeper step has run**, `sitesolide lock` on that machine says to run `sitesolide upgrade`, and the dashboard's *Use a code* is refused by a gatekeeper that does not know it: nothing changes, and running upgrade to the end mends both. **What changes for you.** - `bin/lock.sh enable`, `code` and `disable` are gone; `bin/lock.sh state`, which `sitesolide lock --status` runs, stays, and reads what the machine's manifest asks for rather than the repository's. - The next `sitesolide deploy` writes into `sitesolide.json` the general access the machine carries, `lock` included, as it already did for `portal`; `sitesolide lock` writes it at once. - `sitesolide lock` on a site that serves its own domain is refused, as the dashboard refuses it: the code would close the preview and leave the domain open. Switch the site back to its preview first, `sitesolide domain --deactivate`. `LOCK_DESPITE_DOMAIN` is gone with the script that read it. The release after this one closes the domain too, see *A site closes on every address*. - A Viewer or a Developer of a site that opens with a code reads that it opens with one, no longer the code itself: its Admins and the owner hold it. **Going back.** The previous release's `bin/lock.sh` writes the same files the gatekeeper does, `sitesolide.json`, the codes file and `verrous.caddy`: a workstation still on it keeps working, without the dashboard knowing what it did until the next snapshot. Going back to the previous gatekeeper leaves the two new templates installed, which nothing launches once the previous steward is back. ## Backups stored by restic The backup component stores its snapshots with restic: one repository on the server, `/var/backups/sitesolide-restic`, and, with a bucket, a second one in it, `-restic`, filled by `restic copy`. An hour now stores what changed since the hour before, compressed, rather than a whole archive; the bucket is read back with stock restic on any machine, the passphrase alone. What runs around it does not change: the copy as the project, the backup commands, the bounds, the status file, the restore and its rollback. See [dashboard/src/backup/README.md](../dashboard/src/backup/README.md). **What `sitesolide upgrade` brings**, in its usual order: the backup install first, which now installs Debian's `restic` package if it is missing (and refuses one older than 0.18.0), draws the repository's key and initialises the repository; then the steward, which reads the snapshots from the index the run writes instead of the archives' folder, and the dashboard, whose *Backups* section shows what each snapshot added and the last daily check; the monitor last, which learns to warn on a failed check. The run's and the restore's units go from 256M to 512M of memory, restic running beside Bun. **Then a run by hand**, which upgrade does not start: ```bash ssh you@your-machine 'sudo systemctl start sitesolide-backup.service; sudo cat /var/lib/sitesolide-backup/last-run.json' ``` It imports the archives of the version before into the repository, each proved byte for byte, and writes the index the new steward reads; until it runs, within the hour otherwise, the *Backups* section lists no snapshot, the archives still on the disk. With a bucket, it also creates the bucket's repository with the server's chunker parameters, copies every snapshot there, and imports the objects the server no longer has. The archives and the objects are deleted seven days after their copy was verified. **What changes for you.** - Snapshots are named `-