# Deployment

## Building

```bash
make build   # produces dist/cloudzilla (single binary, embedded templates + CSS)
```

The binaries report `git describe --tags --always --dirty` (e.g. `v0.4.0-41-g8ab8c6e4`) as their version in the page
footer and `cz-admin --version`. Override it with `make build VERSION=v0.4.0`.

`make build` also produces `dist/cz`. `cz-admin` is the operator tool and runs on the server host; `cz` is the separate
remote client that developers run on their own machines, so the server image and archive leave it out. See
[cli](./cli.md).

No Node.js, npm, or Bun needed at runtime. Set config via `config.yaml` or environment variables.

## Docker

Cloudzilla ships a multi-stage `Dockerfile` and `docker-compose.yml`.

### Image build stages

| Stage     | Base                 | Purpose                                                                                                                     |
| --------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `builder` | `golang:1.27-alpine` | Downloads Tailwind CLI (arch-aware musl build), compiles CSS, builds both Go binaries with `CGO_ENABLED=0 -ldflags="-s -w"` |
| runtime   | `alpine:3.24`        | Copies binaries; installs `ca-certificates tzdata postgresql18-client`; exposes 8080/2222; `HEALTHCHECK` on `/healthz`      |

### Persistent volume (`/data`)

All mutable state lives under `/data` inside the container, mounted as a named Docker volume:

| What             | Path                        |
| ---------------- | --------------------------- |
| Git repositories | `/data/git-repos/`          |
| Uploaded files   | `/data/storage/`            |
| SSH host key     | `/data/cloudzilla_host_key` |

`/data/storage` holds avatars when `storage.backend` is `local`, the default. Back it up with the database. To keep
uploads in S3 or an S3-compatible bucket instead, set `CZ_STORAGE_BACKEND=s3` and the `CZ_STORAGE_S3_*` variables; see
[storage](./storage.md) for AWS and R2 examples and the backup notes.

### PostgreSQL volume (`postgres_data`)

The compose `postgres` service runs `postgres:18-alpine` with `postgres_data` mounted at `/var/lib/postgresql`; the 18+
image keeps its cluster in the versioned subdirectory `18/docker`. Don't mount the volume at `/var/lib/postgresql/data`
— the image refuses to start with a mount there.

#### Upgrading from PostgreSQL 17

Postgres 18 can't open a volume created by the old `postgres:17-alpine` service; the container exits with
`Error: in 18+, these Docker images are configured to store database data in a format ...`. Dump the database before
switching compose files, then restore it into a fresh volume:

```bash
# 1. Still on the Postgres 17 compose file (already switched? check out the old docker-compose.yml for this step)
docker compose exec -T postgres pg_dump -U cloudzilla -Fc cloudzilla > cloudzilla.dump
docker compose down

# 2. Remove only the Postgres volume. Not `down -v`: that also deletes cloudzilla_data (git repos, SSH host key)
docker volume ls --filter name=postgres_data
docker volume rm <project>_postgres_data

# 3. On the Postgres 18 compose file
docker compose up -d --wait postgres
docker compose exec -T postgres pg_restore -U cloudzilla -d cloudzilla < cloudzilla.dump
docker compose up -d
```

Keep `cloudzilla.dump` until you've checked the restored instance.

### Key environment variables (Viper `CZ_` prefix)

```
CZ_DATABASE_DRIVER=postgres
CZ_DATABASE_DSN=postgres://cloudzilla:cloudzilla@postgres:5432/cloudzilla?sslmode=disable
CZ_GIT_REPOS_ROOT=/data/git-repos
CZ_GIT_SSH_HOST_KEY=/data/cloudzilla_host_key
CZ_STORAGE_LOCAL_ROOT=/data/storage
CZ_AUTH_JWT_SECRET=<strong secret>
CZ_SECURITY_SECRET_KEY=<at least 32 bytes>
CZ_SERVER_PORT=8080
CZ_GIT_SSH_PORT=2222
```

No `config.yaml` file is needed at runtime when env vars are set.

### Docker make targets

```bash
make docker-build   # docker build --build-arg VERSION=$(VERSION) -t cloudzilla-app:latest .
make docker-run     # docker compose up -d
make docker-down    # docker compose down
```

`make docker-build` stamps the image with the same version as `make build`. A plain `docker build` reports `dev` unless
you pass `--build-arg VERSION=<tag>`. Release images carry their tag.

### First-run bootstrap

```bash
make docker-run
docker exec -it cloudzilla-cloudzilla-1 /app/cz-admin migrate
```

Then open `http://localhost:8080` — the first request redirects to `/setup` where you create the superadmin account via
the web wizard.

The SSH host key is auto-generated into the named volume on first boot — no manual `ssh-keygen` step needed.

### Health checks

Two unauthenticated endpoints on the main HTTP port answer probes. They're served ahead of all middleware, so they write
no request log line, set no cookie, skip the `/setup` redirect and count against no rate limit. Both accept `GET` and
`HEAD`, and send `Cache-Control: no-store`.

| Endpoint   | Answers                                                                                         | Use it for                                               |
| ---------- | ----------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| `/healthz` | Liveness: `200 ok` whenever the process serves HTTP. Touches neither the database nor the disk. | Docker `HEALTHCHECK`, Kubernetes `livenessProbe`         |
| `/readyz`  | Readiness: `200` when every check passes, otherwise `503`.                                      | Load-balancer health checks, Kubernetes `readinessProbe` |

`/readyz` returns JSON such as `{"status":"fail","checks":{"database":"ok","migrations":"fail","storage":"ok"}}`. Each
check is `ok`, `fail` or `skipped`, and all of them share a 3-second deadline:

- `database`: the server can ping Postgres.
- `migrations`: every migration embedded in the binary is recorded in `schema_migrations`. The server never migrates, so
  this fails after an upgrade until `cz-admin migrate` runs. It's `skipped` while the database check fails.
- `storage`: a temporary `.readyz-*` file can be created, written and removed in `git.repos_root`. This catches a
  missing volume, a read-only mount and wrong ownership, but not a full disk.

Give a Kubernetes `readinessProbe` a `timeoutSeconds` of at least 4: the default of 1 turns a slow database into a probe
timeout instead of a 503.

The body never says why a check failed; the server logs the cause at `WARN` with `msg="readiness check failed"`.
Readiness doesn't depend on setup having completed, on the SSH listener, or on SMTP or OAuth.

The image's `HEALTHCHECK` probes `/healthz`, not `/readyz`. Orchestrators restart unhealthy containers, and a restart
fixes neither a database outage nor a pending migration; and on first boot readiness fails by design until
`cz-admin migrate` runs. It requests `http://127.0.0.1:${CZ_SERVER_PORT:-8080}/healthz` with any proxy disabled, so it
reads the port only from the `CZ_SERVER_PORT` environment variable, not from `config.yaml`, and it needs `server.host`
to accept loopback connections (the default `0.0.0.0` does). Check it with
`docker inspect --format '{{.State.Health.Status}}' <container>`.

### Metrics

Cloudzilla exposes Prometheus metrics on a separate listener, off by default. Set `metrics.listen_addr`
(`CZ_METRICS_LISTEN_ADDR`) to start it; it serves only `GET /metrics`. The main port never serves metrics: there,
`/metrics` is an ordinary `/{owner}` path.

**The metrics port has no authentication. Don't publish it, and don't proxy it.** Bind it to loopback
(`127.0.0.1:9090`), or to a private network that only Prometheus can reach. Metric labels carry route patterns, never
user or repository names.

Under Docker Compose, set the address and leave the port out of `ports:`; Prometheus on the same Compose network reaches
it as `cloudzilla:9090`:

```yaml
cloudzilla:
  ports:
    - "8080:8080"
    - "2222:2222"
    # no "9090:9090" here
  environment:
    CZ_METRICS_LISTEN_ADDR: ":9090"
```

A Prometheus `scrape_config`:

```yaml
scrape_configs:
  - job_name: cloudzilla
    static_configs:
      - targets: ["cloudzilla:9090"]
```

| Metric                                     | Type      | Labels                                                                                           | Meaning                                                                                                                                                      |
| ------------------------------------------ | --------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `cloudzilla_build_info`                    | gauge     | `version`                                                                                        | Always 1                                                                                                                                                     |
| `cloudzilla_http_requests_total`           | counter   | `method`, `route`, `code`                                                                        | Requests by chi route pattern (`unmatched` when routing found none); a method outside the standard verbs is `other`. `/healthz` and `/readyz` aren't counted |
| `cloudzilla_http_request_duration_seconds` | histogram | `method`, `route`                                                                                | Request duration, buckets from 5 ms to 60 s                                                                                                                  |
| `cloudzilla_http_requests_in_flight`       | gauge     |                                                                                                  | Requests being served                                                                                                                                        |
| `cloudzilla_git_operations_total`          | counter   | `transport` (`http`, `ssh`), `service` (`upload-pack`, `receive-pack`), `result` (`ok`, `error`) | Transport exchanges. One HTTP fetch can take several `upload-pack` POSTs, so this isn't a count of fetches                                                   |
| `cloudzilla_git_bytes_total`               | counter   | `transport`, `service`                                                                           | Bytes received (`receive-pack`) or sent (`upload-pack`)                                                                                                      |
| `cloudzilla_import_jobs`                   | gauge     | `state` (`queued`, `running`)                                                                    | Imports in memory, read at scrape time                                                                                                                       |
| `cloudzilla_imports_total`                 | counter   | `result` (`succeeded`, `failed`)                                                                 | Finished imports                                                                                                                                             |
| `cloudzilla_webhook_deliveries_total`      | counter   | `attempt` (`first`, `retry`), `result` (`success`, `failure`)                                    | Delivery attempts; a non-2xx response is a failure                                                                                                           |
| `cloudzilla_webhook_retries_due`           | gauge     |                                                                                                  | Deliveries due for retry at the last 60-second retry tick                                                                                                    |
| `go_sql_*`                                 | various   | `db_name="cloudzilla"`                                                                           | Connection pool: open, in-use and idle connections, wait count and wait time                                                                                 |
| `go_*`, `process_*`                        | various   |                                                                                                  | Go runtime and process stats (`process_*` on Linux only)                                                                                                     |

### Backup and restore

`cz-admin backup` writes one uncompressed tar holding the database (`pg_dump --format=custom`), every repository under
`git.repos_root`, the local storage root (avatars, when `storage.backend` is `local`) and the SSH host key.
`cz-admin restore` rebuilds an empty instance from it. Both run `pg_dump` / `pg_restore` from `PATH`; the Docker image
includes `postgresql18-client`. On a binary install, add your OS's client package. Its major version must be at least
the server's: `backup` fails, naming the version needed, when the client is missing or older. `--pg-dump` and
`--pg-restore` point at a specific binary.

**The archive holds password hashes, webhook and TOTP secrets, every private repository and the SSH host key's private
half.** Store it encrypted and mode 0600 (the file is created that way).

What a backup doesn't cover:

- **Configuration.** Keep `config.yaml` and your `CZ_*` variables yourself. Restore with the same `auth.jwt_secret`, or
  everyone is signed out. Restore with the same `security.secret_key`, or stored credentials such as mirror tokens
  become unreadable.
- **An S3 bucket.** With `storage.backend: s3` the bucket isn't copied; the manifest records its name, and `restore`
  prints a warning. Use bucket versioning or replication.

#### Hot backup

The server can keep running. `pg_dump` takes one consistent snapshot first. Then each repository is copied refs first
(`HEAD`, `config`, `packed-refs`, `refs/`) and objects second. Git writes objects before it moves a ref, so every ref in
the copy resolves, and so does every commit the database names, even when a push lands mid-copy. **Don't run
`cz-admin gc` while a backup runs:** it could prune an object the copy still needs.

A repository created, renamed, transferred or deleted during the copy can disagree with its database row. `restore`
lists those as warnings. For a copy that is consistent by construction, stop the server first:

```bash
docker compose stop cloudzilla
docker compose run --rm -T --no-deps --entrypoint /app/cz-admin cloudzilla backup --output - \
  | zstd -q -o "/backups/cloudzilla-$(date +%F).tar.zst"
docker compose start cloudzilla
```

#### Nightly backup

`--output -` writes the archive to stdout, so it can leave the container compressed and encrypted without touching a
volume. This crontab line keeps one archive per day:

```cron
0 3 * * * umask 077; cd /srv/cloudzilla && docker compose exec -T cloudzilla /app/cz-admin backup --output - | zstd -q | age -r age1yourpublickey... -o /backups/cloudzilla-$(date +\%F).tar.zst.age
```

The summary prints to stderr. Test a restore from your newest archive now and then; an untested backup is a hope.

#### Restore runbook

Restore only runs into a new, empty database and an empty (or missing) `git.repos_root`. It checks the whole archive
first and refuses, writing nothing, when:

- the database already has a `schema_migrations` table or any other table;
- `git.repos_root` is not empty;
- the archive's format version is unknown, an entry is a symlink, device or hard link, or an entry path is absolute or
  contains `..`;
- the backup holds a migration this binary doesn't have (restore with the release that took it, or a newer one);
- a different SSH host key is already in place (the server generates one on first boot). Remove it, or pass
  `--replace-host-key`.

1. Start a fresh Postgres 18 with an empty volume, and leave the Cloudzilla volume empty too. Don't start the
   `cloudzilla` service yet: its first boot would create a host key.
2. Set the same `auth.jwt_secret` and `security.secret_key` as the old instance.
3. Restore:

   ```bash
   docker compose up -d --wait postgres
   age -d -i key.txt /backups/cloudzilla-2026-10-08.tar.zst.age | zstd -dc \
     | docker compose run --rm -T --no-deps --entrypoint /app/cz-admin cloudzilla restore --input -
   ```

4. Read the report. Warnings name repository rows without a directory, and directories without a row. Migrations newer
   than the backup are applied, so a backup from an older release restores into a newer one.
5. `docker compose up -d`.

If extraction or migration fails after the database was restored, the error says so. Drop the database, empty the
volumes and start again from step 1.

### Behind a reverse proxy

Set `server.trusted_proxies` (`CZ_SERVER_TRUSTED_PROXIES`) to the proxy's IP or CIDR, e.g.
`CZ_SERVER_TRUSTED_PROXIES=172.16.0.0/12` for a Docker network. `X-Forwarded-For` is ignored from any other peer,
because clients can forge it. A trusted proxy must write bare IP addresses into `X-Forwarded-For`: Cloudzilla reads it
right to left and stops at a hop written as `ip:port` or `[v6]`, so those clients share the proxy's budget. Without this
setting, audit-log IPs and the per-IP rate limits see only the proxy's address, so every client shares one budget. The
limits are 10 attempts per 15 minutes on account creation (`/register`, `/register/complete/{token}` and
`/invite/{token}`) and 30 per 15 minutes on password login (`/login`, `/api/auth/login` and `/auth/ldap`); each route
has its own budget, and an IPv6 client is counted per /64.

The global [rate limits](./configuration.md#rate-limits) count anonymous requests per client IP the same way, so a
missing `server.trusted_proxies` puts every anonymous visitor behind the proxy into one budget. When a request carries
`X-Forwarded-For` from a peer that isn't trusted, Cloudzilla logs one warning naming the peer.

### Several instances

Rate-limit counts live in each process. Behind a round-robin load balancer, N instances allow up to N times each budget.

### Upgrading

`healthz` and `readyz` are reserved owner names, because `/healthz` and `/readyz` are the health-check endpoints. An
existing user or organization with either name keeps its repositories at `/{owner}/{repo}`, but its profile page at
`/{owner}` is shadowed. Find them with:

```sql
SELECT id, username FROM users WHERE lower(username) IN ('healthz', 'readyz');
SELECT id, name FROM organizations WHERE lower(name) IN ('healthz', 'readyz');
```

Migration `082_users_email_case_insensitive` refuses to run while two accounts have emails that differ only by case. Its
error names their user IDs; change or merge those accounts, then run `cz-admin migrate` again.

The owner-name rule (see [access-control](./access-control.md)) is checked only when a user or organization is created,
so names from before it may fail it. An owner whose name fails `service.ValidateName` can't create, fork or receive
repositories, and one whose name isn't a single safe path segment (containing `/`, `\`, `*`, `?`, `[` or `]`, or equal
to `.` or `..`) has its repositories refused on the web, the API, SSH and Git smart-HTTP. List every name that fails the
rule with:

```sql
SELECT id, username FROM users WHERE username !~ '^[A-Za-z0-9][A-Za-z0-9_-]{0,38}$';
SELECT id, name FROM organizations WHERE name !~ '^[A-Za-z0-9][A-Za-z0-9_-]{0,38}$';
```

Rename them in the database, update `repositories.owner_name` to match, and move the owner's directory under
`git.repos_root`.

The shared, case-insensitive owner namespace is also checked only on create, so owners from before it may already
collide. A user and an organization with the same name share one directory under `git.repos_root`, and so do owners
whose names differ only by case on a case-insensitive filesystem (the default on macOS and Windows), so their
repositories alias each other. List them with:

```sql
-- a user and an organization with the same name, ignoring case
SELECT u.id AS user_id, u.username, o.id AS org_id, o.name AS org_name
FROM users u JOIN organizations o ON lower(u.username) = lower(o.name);
-- usernames that differ only by case
SELECT lower(username) AS name, array_agg(id ORDER BY id) AS user_ids
FROM users GROUP BY lower(username) HAVING count(*) > 1;
-- organization names that differ only by case
SELECT lower(name) AS name, array_agg(id ORDER BY id) AS org_ids
FROM organizations GROUP BY lower(name) HAVING count(*) > 1;
```

Rename all but one owner in each row in the database, updating `repositories.owner_name` to match. The old directory
also holds the other owner's repositories, so move only the renamed owner's repository directories (`<name>.git`,
`<name>.wiki.git` and their `.deleted.` copies) into its new one.