# Deploy to a server One host, Docker Compose and a reverse proxy remain the shipped single-server path. For clusters, use the maintained [Kubernetes Helm chart](kubernetes.md), with external data services and a blocking migration Job. OpenShift has a candidate configuration but is not yet validated. There are no one-click PaaS quickstarts. Past one host - a managed Postgres, several API hosts, several deployments - is [Scale beyond one host](scaling.md). !!! tip "Read the [production checklist](configuration.md#production-checklist) first" Nine settings ship with defaults that are fine on a laptop and wrong on a host somebody else can reach. `scripts/server-init.sh` below generates all nine, so the checklist is what you check afterwards rather than what you type. ## What you need | | | |---|---| | **A host** | 4 vCPU and 8 GB of RAM runs it. See [sizing](#sizing-the-host) | | **Container engine** | Engine 24+ with the Compose plugin (2.24 or later), and your user in the `docker` group. Nothing is built on the host: the images are pulled from GHCR. Alternatively, use the [manual Podman path](#podman-support). | | **Two hostnames** | one for the site, one for the API — see [why two](#why-two-hostnames) | | **A reverse proxy** | [Traefik](#option-a-traefik) or [Nginx](#option-b-nginx). It terminates TLS | | **No provider key** | chat models and embeddings are both keyed per organization, in the product's vault. The environment holds no credential for either — see [Data protection](data-protection.md#nothing-leaves-by-default) | The host also needs ports 80 and 443 open, and nothing else. Postgres and Redis are published on no interface at all, and the worker serves nothing. ### Why two hostnames The browser talks to both. Most calls go through the frontend's own server-side routes, but the chat WebSocket connects to the API directly, so the API needs a name a browser can resolve and a certificate of its own. `app.example.com` and `api.example.com` is the shape. They can be any two names; what they must not be is one name with a path prefix, because the API's cookies and the site's are scoped to the host. ### Podman support **Podman is supported through the manual deployment path with Docker Compose as its provider.** The runtime checks used Podman 6.1.4 and Docker Compose v2.40.3 on Fedora CoreOS 44 (a macOS Podman machine, ARM64, cgroups v2, SELinux enforcing). The shipped file needs Compose 2.24 or later. **Do not use `podman-compose` 1.6.0.** It started the stack and an agent answered, but a controlled failed migration still started the dependent app. Docker Compose correctly refused that start. The providers' version numbers are independent; `podman-compose` 1.6.0 is not Docker Compose 1.6.0. The quickstart remains Docker-only and directs detected Podman to this manual path, including with `--no-sandbox`. From the configured repository root, choose the actual Docker Compose executable (a standalone binary or CLI plugin): ```bash export PODMAN_COMPOSE_PROVIDER=/absolute/path/to/docker-compose podman compose version podman compose -f docker-compose.yml up -d ``` Check that `podman compose version` identifies Docker Compose. Using `-f` explicitly keeps the clone's source-build override out of this published-image installation. Follow [Install](install.md) to bootstrap the organization and model after the services start. **Optional sandbox.** `pydantic-ai-backend` 0.2.39 worked against both rootful and rootless Podman sockets. It does not need a new backend or release. Use `docker-compose-podman.yml` to adapt the socket and workspace mounts; prepare their permissions before enabling the profile. On native Linux with rootless Podman: ```bash systemctl --user enable --now podman.socket export SANDBOX_DOCKER_SOCKET="$XDG_RUNTIME_DIR/podman/podman.sock" export SANDBOX_WORKSPACE_ROOT="$HOME/.local/share/agenticos/sandbox-workspaces" mkdir -p "$SANDBOX_WORKSPACE_ROOT" podman unshare chown 10001:10001 "$SANDBOX_WORKSPACE_ROOT" export DOCKER_GID=0 ``` For rootful Linux, enable the system `podman.socket` instead, set `SANDBOX_DOCKER_SOCKET=/run/podman/podman.sock`, and prepare the workspace with `sudo install -d -o 10001 -g 10001 "$SANDBOX_WORKSPACE_ROOT"`. Run Compose against that rootful engine. Its socket group is normally 0; a customized rootful socket needs its actual group in `DOCKER_GID`. On a rootless macOS Podman machine, both bind sources belong to the Linux VM, not to macOS. Provision them inside that VM: ```bash export SANDBOX_DOCKER_SOCKET="$(podman machine ssh 'printf "%s/podman/podman.sock" "$XDG_RUNTIME_DIR"')" export SANDBOX_WORKSPACE_ROOT=/tmp/agenticos-sandbox-workspaces podman machine ssh 'mkdir -p /tmp/agenticos-sandbox-workspaces && podman unshare chown 10001:10001 /tmp/agenticos-sandbox-workspaces' export DOCKER_GID=0 ``` Keep the deployment's existing keys. Set a nonempty `SANDBOXD_TOKEN` and `SANDBOX_URL=http://sandboxd:8080` in the env file the stack reads, as described under [sandbox configuration](sandbox.md). Then start with both files: ```bash podman compose -f docker-compose.yml -f docker-compose-podman.yml \ --profile sandbox up -d ``` The override disables SELinux labeling only for `sandboxd`, which already holds the engine socket. It shares the workspace label with `:z` because sibling sandboxes also mount it. Sandbox containers retain their own labels and `no-new-privileges`; do not relabel the socket or disable SELinux globally. Rootless socket ownership maps to group 0 inside the user namespace. Use a persistent workspace path on a server: `/tmp` is only a local example. Persist the socket, workspace and group settings in the operator's environment for subsequent Compose commands. A host `DOCKER_HOST` does not change the socket mounted inside `sandboxd`. Verified paths include an HTTP agent run, persisted completion, workspace files, nonzero exits, timeouts, enforced memory/CPU/PID limits and runtime image builds. The shipped workbench image also read/wrote XLSX, rendered a plot and ran Node and LibreOffice. The model was a deterministic OpenAI-compatible test server; the desktop runtime and every Linux distribution were not tested. See the [recorded checks](https://github.com/vstorm-co/agenticos/blob/main/docs/audits/podman-2026-11-03.md) for versions, scope and the migration failure reproduction. ## Sizing the host Measured on an idle deployment, not estimated: | | at rest | ceiling | |---|---|---| | `app` (2 uvicorn workers) | ~1.0 GB | 2.5 GB at the default 4 workers | | `db` | ~1.3 GB with the tuning below | 2 GB | | `worker` (one ingest child) | 387 MiB | 2.5 GB (`WORKER_MEMORY_LIMIT`) | | `frontend` | ~300 MB | 1 GB | | `redis` | 9 MiB | 512 MB | The number that decides the host is **`UVICORN_WORKERS`**. Each worker is a separate process that imports the whole application — 460 MiB, spawned rather than forked, so nothing is shared. Four of them are 1.9 GB before a request arrives. Two workers suit a team of ten and still leave one serving while [the watchdog](configuration.md#a-worker-whose-event-loop-has-stopped-turning) replaces a wedged sibling. One worker is the setting to avoid: a blocked event loop is then the whole deployment, until it kills itself. A host short of CPU shows a different symptom: `Child process [N] died` lines in the `app` log with no watchdog verdict above them, while every page is slow. Those are workers uvicorn's supervisor killed for answering its ping late, not workers that failed. It waits [`UVICORN_WORKER_HEALTHCHECK_TIMEOUT`](configuration.md#a-worker-whose-event-loop-has-stopped-turning) seconds, 60 by default. Look at the host's CPU steal before you raise it. Two settings keep a slow deployment diagnosable: - **Every log is capped.** Each service in the production compose files writes at most five files of 20 MB. Docker's default keeps a container's log until the container is removed, and nothing on the host rotates it. - **Postgres records what each statement costs.** The `db` service preloads `pg_stat_statements` and a migration creates the extension. Where the database role may not create it, the migration logs a notice and goes on. `agenticos cmd doctor` prints a `query statistics` line either way. !!! note "The database is tuned against its own limit" `docker-compose-prod.yml` runs Postgres with `shared_buffers=512MB` against a 2 GB limit, and gives it 512 MB of `/dev/shm` — Docker's default is 64 MB, which a parallel scan over a collection's vectors exhausts, reporting `could not resize shared memory segment`. Move the limit and move the tuning with it; they are written next to each other for that reason. It also sets `max_connections=200`, because the pools can open more than Postgres' stock 100: two pools of 15 per uvicorn worker, so 120 at the default four, and 30 more for the background worker. Raise it with `UVICORN_WORKERS` or `DB_POOL_SIZE`, or Postgres refuses connections with "too many clients" before a pool can queue them or shed the load. ## Point the names at the host Two A records, before anything else. Let's Encrypt proves you control a name by fetching a file over HTTP from wherever it resolves, so a certificate is not issued until this is true and propagated. ``` app.example.com A 203.0.113.10 api.example.com A 203.0.113.10 ``` !!! warning "A wildcard does not do this for you" Where `*.example.com` already points somewhere — a marketing site, usually — both names resolve to it. A record for the specific name beats a wildcard, so the fix is to add the two above rather than to remove the wildcard. Check from somewhere that is not the host, because the host may have its own answer: ```bash dig +short app.example.com api.example.com ``` ## Get it onto the host ```bash sudo install -d -o "$USER" -g "$USER" /opt/agenticos git clone https://github.com/vstorm-co/agenticos.git /opt/agenticos cd /opt/agenticos bash scripts/server-init.sh ``` `server-init.sh` writes `backend/.env`: it generates the five secrets, asks for the two hostnames and an address for Let's Encrypt, and derives the public URLs and the CORS origin from what you gave it. It asks for no provider key: those live in each organization's vault. It refuses to overwrite an existing file. The clone is where the compose files and that env file live; no code runs from it. What runs is the two images the repository publishes. ### The images | | | |---|---| | `ghcr.io/vstorm-co/agenticos-backend` | The API, the worker and the migrations - one image, three commands | | `ghcr.io/vstorm-co/agenticos-frontend` | The console | Both are built for `linux/amd64` and `linux/arm64` by `.github/workflows/images.yml`. A release (`v0.0.380`) publishes `0.0.380` and moves `latest`; every commit on `main` publishes `edge` and `sha-`. The compose files read the tag from `AGENTICOS_VERSION` in `backend/.env` and default to `latest`. Three rules the workflow keeps, each worth knowing before relying on a tag: - **Only a commit on `main` is ever published.** A `v*` tag pushed from a branch, or a run dispatched on one, is refused before anything is built - so `latest` cannot get past the pull-request boundary. - **A release on a commit `main` already built is not rebuilt.** Its `sha-` manifest gets the version and `latest` as extra names, so the digests a host pinned to are exactly the ones the release names. - **A commit with no images can be given them.** Run the workflow by hand with its `sha` input - `gh workflow run images.yml --ref main -f sha=` - and it publishes that commit's `sha-` tag and nothing that moves. That is the path for a commit older than the workflow, and for one whose run was lost. !!! warning "Pin a release on a host you care about" `AGENTICOS_VERSION=0.0.380` in `backend/.env`, so that `make prod` on a bad day pulls what ran yesterday rather than whatever was released this morning. `scripts/deploy.sh` pins for you - to the `sha-` tag of the commit it deploys - for exactly as long as the deploy runs. Pin it from that release's checkout: `git checkout v0.0.380` beside the variable. The sandbox service's image is pinned in the compose file, at the version the backend of the same commit speaks, so an older backend run from a newer checkout meets a newer `sandboxd` - and `latest` from an old checkout is the same mismatch the other way round. `scripts/deploy.sh` checks out the commit it deploys, so a scripted deploy or rollback keeps the two together. Both packages pull anonymously. If a pull answers `unauthorized`, the package has been made private or a stale `docker login ghcr.io` is in the way; neither is something a host can fix by itself. !!! danger "`backend/.env` holds the key that unwraps every stored credential" `VAULT_MASTER_KEY` is what makes an organization's provider keys, bot tokens and MCP credentials readable. Losing it does not lock you out of the product; it makes every secret in it unrecoverable. Back the file up somewhere a lost disk cannot take with it, and rotate with [`agenticos cmd vault-rotate`](secrets.md#operations) rather than by editing. Two optional things it does not ask about, both in that file: `SMTP_*`, without which invitations and password resets cannot be sent, and `LOGFIRE_TOKEN`, which is where traces of agent runs go. ## Choose a reverse proxy Something has to terminate TLS and route the two names. Both options below reach the same containers; pick on whether you already run one. !!! note "Leave compression to the application" The API gzips its own answers, the console's `/api/*` proxy compresses again for the browser whatever the API compressed, and Next compresses its pages. So the proxy needs no `gzip on`, and it must not blank the client's `Accept-Encoding`: that only turns the compression off. A response already carrying `Content-Encoding` passes through Nginx and Traefik unchanged. ### Option A: Traefik The shorter path, and the one to pick on a host that already has Traefik: the containers carry labels, Traefik discovers them, asks for the certificate and renews it. Nothing to reload and no second config file to keep in step. If Traefik is not there yet, the repository ships one: `traefik/traefik.yml` and `docker-compose-traefik.yml`, which is an entrypoint on 443 with a Let's Encrypt resolver and 80 redirecting to it. ```bash docker network create traefik_webgateway docker compose --env-file backend/.env -f docker-compose-traefik.yml up -d ``` Where Traefik is **already** running, leave those alone and point `TRAEFIK_NETWORK` at the network it watches. The overlays read that name, so nothing about the existing proxy has to change. Then bring the stack up with `PROXY=traefik`, which adds the two overlay files that carry the labels: ```bash make prod PROXY=traefik make prod-frontend PROXY=traefik ``` `server-init.sh` has already written `PROXY=traefik` into `backend/.env`, which is where `scripts/deploy.sh` reads it from — so later deploys keep the proxy this host was set up with rather than the one a script assumed. !!! info "`exposedByDefault: false` is doing real work" It is the one setting in `traefik/traefik.yml` worth reading before you run it. Only `app` and `frontend` carry `traefik.enable=true`, so Postgres, Redis, the worker and the sandbox daemon are reachable from nothing outside the host — and that is a property of *not being labelled*, so it survives somebody adding a service without thinking about the proxy. !!! warning "Some request paths and query strings are credentials" These path segments open something to whoever reads them: - a public link's key, in `/a/`, `/api/v1/public/artifacts/` and `/api/v1/artifact-embed/`, until the link is rotated; - a signed content address, `/api/v1/artifact-content/`, for its lifetime; - a shared conversation's token, in `/shared/` and `/api/v1/conversations/shared/`; - a workflow form's token, in `/f/` and `/api/v1/public/workflow-forms/`; - a webhook test's token, in `/api/v1/workflow-webhook-tests/` and `/api/v1/workflows//webhook-test/`; - a channel link's token, in `/link/` and `/api/v1/me/channel-link/`; - an invitation's token, in `/invitations/`, `/api/v1/invitations/` and `/api/v1/invitations//accept`; - a workflow run's resume token, in `/api/v1/workflow-resume//`. So do these query parameters: - `visitor` and `token` on the agent widget's socket, `/api/v1/embed//ws`: a hosted page's key, which resumes the visitor's conversation, and the customer's signed visitor token; - `code` and `state` on a sign-in provider's return, `/api/v1/oauth//callback`, and `code` on the console's `/auth/callback`; - `invitation_handle` on `/api/v1/oauth//login` and `/api/v1/auth/kerberos/login`, which redeems the invitation it stands for; - `token` on the console's `/reset-password`, `/auth/magic-link` and `/auth/email-change`, the links that a password reset, a sign-in link and an email change send. A request made from one of those pages can carry its address in the `Referer` header. The API replaces the API segments and the values of those parameters with `[REDACTED]` in its own access log, in uvicorn's line for each accepted WebSocket, in every log record and in Logfire spans, the console output, the endpoint's arguments and the copy Logfire makes when a span starts included. Other query parameters, such as `q`, `skip`, `limit` and ids, are kept. Traefik can keep, drop or redact a whole field but cannot rewrite part of one, so `traefik/traefik.yml` drops `RequestPath`, which includes the query string, from Traefik's access log. Nginx can rewrite part of a value, but `nginx/nginx.conf` drops the path, the query string and the `Referer` the same way, with the `agenticos_no_path` log format each server block names, so the list of secret paths is kept in one place. Both logs still have the client, the status, the size and the duration. Nginx's error log writes the request line and the `Referer` of a failed request, and no setting changes that, so keep the error log on the host. If you need the paths in the access log, remove that field rule or log format and handle the log as a secret: keep it on the host and do not ship it to a log service. ### Option B: Nginx For a host where Nginx already terminates TLS, or where the proxy is not in Docker at all. The stack publishes both ports on `127.0.0.1` and Nginx reaches them there: ```bash make prod make prod-frontend ``` `nginx/nginx.conf` is the template. Two substitutions before it serves anything: the `server_name` in each block is `${DOMAIN:-localhost}`, and Nginx does not expand that — put the two hostnames in by hand. Certificates are yours to obtain and renew, and so is the `Strict-Transport-Security` header, which the backend deliberately leaves to whatever terminates TLS. The template's access log leaves out the request path, the query string and the `Referer`; *Some request paths and query strings are credentials*, above, says why and what the error log still writes. !!! warning "`BIND_HOST` is a security setting, not a convenience" The loopback default is what makes the auth rate limit mean anything. `RATE_LIMIT_TRUST_FORWARDED_FOR` tells the API to count an attempt against the address the proxy forwards, so whatever can reach the API *past* the proxy chooses the address its attempts are counted against. Set `BIND_HOST=0.0.0.0` only for a proxy on a different machine, and firewall the port to it. !!! warning "The frontend is a compose project of its own" Compose names a project after the directory, so both stacks were `agenticos` — and bringing the frontend up then reported the five backend containers as **orphans**, with compose's own suggestion to run the command again with `--remove-orphans`. Taking that advice stops the API, the database, Redis and the worker. The `make` targets and `scripts/deploy.sh` pass `-p agenticos-frontend`, so the warning is gone. The compose files fix no container names either - each project names its own, so two stacks on one host cannot take each other's containers over, and `deploy.sh` waits on the `app` and `frontend` services rather than on a name. A deployment that predates both fixes is recreated under the new names on its next `up`; nothing has to be removed by hand. The two projects still meet on a network with a fixed name - `agenticos_edge` in production, `agenticos_backend` on the dev server - because the frontend joins it as an external network. A host running **two** AgenticOS stacks sets `AGENTICOS_EDGE_NETWORK` and `AGENTICOS_DATA_NETWORK` (or `AGENTICOS_NETWORK`) apart in each stack's `backend/.env`; otherwise both stacks' `db`, `redis` and `app` resolve on one bridge, and a request can reach the neighbour's database. ## Start it, and create the first account `make prod` pulls the images, starts the stack and runs the migrations - the last as a `migrate` service the API waits on, so a `docker compose up -d` by hand on the same files does the same. The first pull is about 2 GB; a later one is the layers that changed. Then create an organization, an owner and a working agent: ```bash docker compose --env-file backend/.env -f docker-compose-prod.yml \ exec -T app agenticos cmd bootstrap \ --email you@example.com --password 'a real password' \ --org 'Your Company' --provider anthropic --api-key sk-ant-... ``` The provider key here is what the two starting agents run on. Without it they are created and cannot answer; every other provider is added in the product, per organization, from the vault. !!! tip "Check it from outside, not from the host" ```bash curl -fsS https://api.example.com/api/v1/health curl -fsSo /dev/null -w '%{http_code}\n' https://app.example.com ``` A stack that is healthy on the host and unreachable from the internet is DNS, the firewall or the certificate — three things a health check inside the host cannot see. Then, once, by hand: sign in, invite somebody (which proves `SMTP_*`), and send AgenticOS a message (which proves the provider key and the WebSocket). Each exercises a path nothing else here checks. !!! info "The security headers come from the backend, so any proxy is covered" A Content-Security-Policy, `X-Frame-Options: DENY`, `X-Content-Type-Options: nosniff`, `Referrer-Policy` and `Permissions-Policy` are set on every response — including the 500 for an unhandled exception, which is built outside the middleware stack and stamps them itself. The interactive API docs drop the CSP only, because Swagger loads assets a strict policy forbids. **HSTS is deliberately left to the proxy**, which is where TLS terminates. A proxy that sets its own CSP should be at least as strict as this one. ### Services on your own network A request a member configures - a workflow HTTP step, a Git, website or S3 source, an A2A agent or an MCP connection - is refused when its host resolves to a private address, so a wiki, a Git server or an MCP server inside your network is out of reach by default. To open one, list it in `OUTBOUND_PRIVATE_ALLOWLIST` in `backend/.env`, as a host, a `.suffix`, an address or a CIDR range, and restart the API and the worker. See the setting's row in [the configuration reference](configuration.md#project-settings) for the forms, what is refused whatever you list and what every account can then reach. List only services that check their own credential. `agenticos cmd doctor` reports the list as its `private network` line. It fails when a listed range covers the database or Redis, and warns about a range wider than /16 (IPv4) or /48 (IPv6). ### Turning the sandbox on The service that runs an agent's code is behind a compose profile, because it is the one container holding the Docker socket and mounting that on a shared host should be a decision rather than a default. Three things, once: ```bash make sandbox-token # writes SANDBOXD_TOKEN to backend/.env sudo mkdir -p /var/lib/agenticos/sandbox-workspaces sudo chown 10001:10001 /var/lib/agenticos/sandbox-workspaces ``` Then a deploy brings it up: `scripts/deploy.sh` passes `--profile sandbox` when `SANDBOXD_TOKEN` in `backend/.env` has a value, so the host itself says whether it runs one. Like `make` and quickstart, it exports `DOCKER_GID` into the sandbox's `group_add`: the socket's group on Linux (following symlinks), or `0` on Docker Desktop, where the mounted socket belongs to the VM. An explicit `DOCKER_GID` in the environment or on the `make` command line takes precedence. Nothing else in `.env` is needed: the backend reaches the daemon through a sandbox *connection* somebody creates in the console, and `http://sandboxd:8080` is recognised as this deployment's own. !!! warning "A profile compose is not told about is a service compose stops" `up -d` on the same project without `--profile sandbox` does not leave the sandbox alone — it stops it. So a host that started it by hand had it taken away by its next deploy, with an agent's code execution failing for reasons nowhere near the deploy that caused it (#1506). That is why the script reads the token rather than taking a flag. ## Deploying a change ### Native memory and routine continuity { #native-memory-upgrade } Before this upgrade, take the [database, media and vault-key backups](#backups) and coordinate the API and worker rollout. Use the deployment's normal migration step; do not copy rows or create replacement schedules manually. Before applying `0285_member_proactivity.py`, stop API/sweep producers and drain active proactive jobs, then stop workers. The migration refuses while a watch or nightly model job is running and backfills queued work with a shared per-member concurrency key. Update API and workers together so new jobs use the same key; a watch and a nightly look for one person must not independently spend the same remaining allowance. - `0248_drop_memory_mem0.py` refuses the upgrade while `memory_mem0` remains in any stored draft or published spec. Before retrying, inventory every endpoint and vault credential mapping from the backup, erase the external memories and verify completion at each provider. Only then remove those bindings explicitly from `agents.draft_spec` and `agent_versions.spec`. An export alone is not an erasure. Keep the cleanup evidence, review unused vault keys and revoke retired credentials. The migration makes no remote request and never assumes a purge succeeded. If an earlier development build already stripped the bindings, recover the inventory from its pre-upgrade backup and perform the same purge; native person/agent erasure cannot reach that external data. - `0251_primary_conversations.py` and `0258_primary_channel_history.py` establish primary private chats. An existing mapping wins. Only demonstrably private, linked channel history is adopted once; project/group and ambiguous histories remain separate. Do not rerun the backfill after somebody clears a primary chat. - Existing routine IDs, schedules and logs remain in place. The delivery migration `0255_trigger_delivery.py` gives old routines `run_log`; new routines default to a primary chat or their trusted source conversation. Review the destination, acting person, end and budget under **Routines**, rather than creating copies. - `0259_trigger_version_pin.py` supports immutable version pins. A pin and a named environment are exclusive. Clear a pin before deleting its version. A parked run retains its admitted version; `0260_routine_run_origin.py` retains its origin so deleting a routine does not release its authorization guard. - `0261_watch_attempt_status.py` starts explicit attempt tracking without guessing older success/failure history. An unavailable or incomplete check does not advance the success timestamp. Review existing watches and their account access: historical rows do not prove how consent was originally given. Choose a model explicitly before enabling memory maintenance; the daily briefing keeps the existing organization model/budget settings. With `0262_watch_agent_sources.py`, watches use their selected agent and retain approved sources on runs. Existing watches require the owner to select an agent and approve its sources before running again; IDs and findings are preserved. Check [governance](governance.md#proactive-work) and the [channel setup](channels.md#sms), including `PUBLIC_BASE_URL` and expired SMS identity links. In a test organization, verify one private chat, one routine's Activity/delivery, pause/expiry and one unavailable-source watch before enabling a wider rollout. ### The retry-history boundary in 1.25.0 { #retry-history-upgrade } Version 1.25.0 adopts Pydantic AI 2.61's `retried` tool outcomes and separate `retry-feedback` messages. Existing histories and run checkpoints load without a database migration or republishing agents. Histories containing these parts need the new reader: an older backend cannot replay these message shapes. Upgrade all API and worker processes together, draining or stopping the old processes before the new ones accept work. Do not mix older and newer backend replicas against the same database. After either new retry form has been written, a code-only rollback across this boundary is unsupported; retaining the new reader or restoring a pre-upgrade database snapshot is required. ### By hand ```bash remote=$(ssh you@your-host 'mktemp -t agenticos-deploy.XXXXXX') ssh you@your-host "cat > $remote" < scripts/deploy.sh ssh you@your-host "trap 'rm -f $remote' EXIT; bash $remote " ``` `scripts/deploy.sh` fetches that commit, waits for the images CI published for it (`sha-`, usually already there), pulls them, restarts, and waits for both containers to report healthy before it returns non-zero or not. It takes a **commit** rather than a branch, so what is deployed is what was reviewed, not whatever `main` has moved to since - and what runs is byte-for-byte what CI built, on a host that never needs the toolchain. !!! warning "Copy it to the host, then run it — do not pipe it into `bash -s`" Under `bash -s` the script is the shell's own standard input, and the first command in it that reads stdin consumes the rest. `docker compose exec` forwards stdin to the container even with `-T`, so the migration ate everything below itself, bash reached EOF, and the deploy exited **0** having never built the frontend or waited for any container. The site was down and the deploy was green ([#1488](https://github.com/vstorm-co/agenticos/issues/1488)). Two connections instead of one is what stops the procedure being able to truncate itself. It is not zero-downtime. Compose recreates the containers whose image changed, so the site is unavailable for the few seconds that takes. ### Release image retention Before pulling, the script removes old, unused release images from only `ghcr.io/vstorm-co/agenticos-backend`, `agenticos-frontend` and `agenticos-browser`. It keeps two additional unused image IDs per repository, newest first; several SHA or version tags on one ID count as one image. Only `sha-` and numeric version tags are eligible. All aliases of a removable ID are removed together, without `--force`. `--no-prune` also prevents implicit deletion of untagged parent images. Every running or stopped container on the host protects its image, including other Compose projects. A target image, any mutable/manual tag, or a tag or digest in another repository also protects the entire ID. Sandbox/runtime images, dangling images and application volumes are outside this cleanup. These protections can keep more images than the unused-image allowance. The hash of the deployment directory separates pins for stacks sharing a host. Local `deploy--current` and `deploy--rollback` tags preserve each component's last healthy image and its predecessor after containers are replaced. On first use, `deploy--current` is seeded from this stack's healthy containers, not the checkout; inconsistent healthy image IDs refuse the deploy. Pins advance only after the new services pass health checks. Failed deploys retain the recorded pins, and redeploying the same image does not replace its rollback. Continue to roll back by an approved commit SHA; these local tags are retention markers, not registry releases. The script requires at least **5 GiB free** on Docker's storage filesystem before each pull and after optional runtime preparation. It checks `DockerRootDir` and the default `/var/lib/containerd` store when present. This is a capacity floor, not a reservation or an image-size estimate; monitor custom containerd stores separately. Both backend and frontend images are pulled before either service is restarted. An insufficient-capacity refusal reports the filesystem and available space: free space outside the protections or expand storage, then retry the approved deployment. Never use a blanket `docker image prune -a` to implement this policy. On Docker Desktop, the storage directory is inside its Linux VM. The script measures VM free space through a temporary, read-only, network-isolated container using an already-cached backend image, preferably the last healthy one. The probe cannot pull images and is removed automatically. It also checks the host filesystem containing the deployment directory; monitor a custom VM backing disk's host filesystem separately. This update procedure requires a cached backend image; complete the normal initial installation first. ### From GitHub, with an approval `.github/workflows/deploy.yml` offers each published stable release for deployment and waits for somebody to approve it. That gate is **a repository setting, not a step in the file** — without it, the workflow deploys each published stable release unattended. Set it up once: 1. **Settings → Environments → New environment**, named `production`. 2. Tick **Required reviewers** and add whoever may approve. This is the gate. 3. Add the environment's variables: `SITE_URL`, `API_URL`, and `APP_DIR` if the checkout is not at `/opt/agenticos`. 4. Add the secrets below. !!! warning "Cancel a deploy you do not intend to approve" Every run shares the `deploy-production` concurrency group, and a run sitting at the approval gate holds it. It does not expire on its own — GitHub cancels an unactioned one after 30 days — so until somebody approves or cancels it, later releases queue behind a decision nobody is going to make, and the server keeps running whatever was deployed last. So a deploy you have decided against is cancelled, not left. One left waiting on a superseded commit blocked three later runs here before anybody noticed the queue rather than the runs. `deploy-queue.yml` is what happens when it is left anyway: every six hours it cancels a Deploy run that has waited more than twelve for an approval, and opens an issue naming the run and how far the latest stable release has advanced from the last successful deploy. It approves nothing and deploys nothing — it drains the queue so the next published release reaches the gate, and says that it did. Run it by hand with `gh workflow run deploy-queue.yml`. !!! danger "Approving from the environment's queue approves the **oldest** run" Which is the stale one. A late approval given there deploys the commit from whenever the queue jammed, not the commit that was just pushed — that is how a fortnight-old commit reached this server. Approve from the run page of the commit you mean, or start a fresh one with `gh workflow run deploy.yml --ref main -f ref=v1.34.3` and approve that. | Secret | What | |---|---| | `DEPLOY_HOST` | The host's address | | `DEPLOY_USER` | The account the checkout belongs to | | `DEPLOY_SSH_KEY` | A private key whose public half is in that account's `authorized_keys` | | `DEPLOY_KNOWN_HOSTS` | `ssh-keyscan your-host`, run from somewhere you trust | Generate the key for this and nothing else: ```bash ssh-keygen -t ed25519 -N '' -C 'github-actions-deploy' -f deploy_key ssh-copy-id -f -i deploy_key.pub you@your-host ssh-keyscan your-host # → DEPLOY_KNOWN_HOSTS cat deploy_key # → DEPLOY_SSH_KEY, then delete it locally ``` !!! note "The host key is a secret rather than a `ssh-keyscan` at deploy time" Scanning at deploy time trusts whatever answers on that address, which is the thing a host key exists to prevent. Scan once, from somewhere you trust, and store the answer. A run then appears with **Review deployments**; approving it starts the job. `workflow_dispatch` runs the same job against a ref you name, which is how a rollback is done, and it passes through the same approval. ## Backups One volume matters, and it is not obvious which: | Volume | Holds | Backup | |---|---|---| | `postgres_data` | everything — agents, conversations, sealed credentials | **yes** | | `media_data` | uploaded files, before ingestion | yes | The cache has no volume at all. Rate-limit buckets, channel dedupe claims and membership answers all carry a TTL and all rebuild themselves, so Valkey runs with `--save ''` and starts empty after a restart. A host that ran an earlier release still carries an `agenticos_redis_data` volume that nothing mounts any more; `docker volume rm agenticos_redis_data` removes it. ```bash docker compose --env-file backend/.env -f docker-compose-prod.yml exec -T db \ sh -c 'pg_dump -U "$POSTGRES_USER" -Fc "$POSTGRES_DB"' > "agenticos-$(date +%F).dump" ``` The identifiers come from the container's own environment rather than being written out, because both are settings: a deployment that changed either would otherwise get an empty file and an error nobody reads on the way past. !!! danger "A database backup without `backend/.env` is not a backup" The credentials in it are sealed with `VAULT_MASTER_KEY`. Restored next to a different key, every provider key, bot token and MCP credential in the dump is unreadable — and the product will tell you so one refusal at a time. ## Rolling back | | How | |---|---| | **Code** | Deploy the previous commit: `workflow_dispatch` with its sha, or `scripts/deploy.sh`. The images are still in the registry, so this is a pull, not a build. A commit with no `sha-` images - older than `images.yml`, or its run lost - is published first with `gh workflow run images.yml --ref main -f sha=`; the deploy names that command when it gives up waiting | | **Schema** | `agenticos db downgrade --revision=-1`, then deploy the code that matches | | **Data** | `pg_restore` the dump, then check the migration the code expects | Rolling code back **across a migration is a decision, not a command**. The old code meets a schema it has never seen; whether that works depends on the migration. Read it before assuming. ## Recap - **One host, Compose, a proxy in front.** Seven containers - one of them runs the migrations and exits - two of them stateful, every one pulled - nothing is built on the host. Pin `AGENTICOS_VERSION`. - **`UVICORN_WORKERS` decides what the host costs.** 460 MiB per worker, nothing shared. Two for a team, four for real traffic. - **DNS before everything.** No certificate is issued until the names resolve to the host. - **The approval is a repository setting**, not a line in the workflow. Without required reviewers on the `production` environment, every merge deploys itself. - **Back up `postgres_data` and `backend/.env` together.** Either without the other is not a restore.