# AGENTS.md

Docker-based homelab managed with Ansible. Most self-hosted services run in
Proxmox LXC containers on Debian 13 with Docker Compose. Homepage also
has a Talos Kubernetes cluster. Networking is via Tailscale. LXC logging is
centralized via Grafana Alloy to Loki.

## Build / Run Commands

Run recipes from the repository root using [just](https://github.com/casey/just):

```sh
just setup                              # Install Ansible Galaxy collections
just run                                # Run the all-host playbook
just run talos-cluster/gitops-secrets     # Provision Kubernetes external resources
just edit-secrets                       # ansible-vault edit group_vars/all/secrets.yaml
```

From the **repo root**:

```sh
just lint                               # yamllint + ansible-lint + shellcheck
```

### Validation

This checkout has no CI workflows or Git hooks. Run `just lint` locally to
check YAML, Ansible, shell scripts, Kubernetes pod security, backup coverage,
and Renovate configuration. Deployment playbooks are run explicitly with
`just run`; they are not automatically applied by this repository.

Keep all repository work on local `main`: edit, review, and commit there. Create
another branch or worktree only when Dave explicitly asks. Push only on explicit
request.

## Ansible Playbook Conventions

### Playbook structure (canonical form)

Every service playbook follows this exact pattern:

```yaml
---
- name: Install <service-name>
  hosts: <service-name>
  tasks:
    - ansible.builtin.include_tasks:
        file: ../../tasks/install-docker.yml

    - ansible.builtin.include_tasks:
        file: ../../tasks/install-common-packages.yml

    - ansible.builtin.include_tasks:
        file: ../../tasks/install-alloy.yml

    - ansible.builtin.include_tasks:
        file: ../../tasks/install-beszel-agent.yml

    - name: Copy docker-compose.yml
      ansible.builtin.template:
        src: docker-compose.yml
        dest: "~/docker-compose.yml"
        mode: "0640"

    - name: Enable UFW
      community.general.ufw:
        state: enabled
        policy: deny

    - ansible.builtin.include_tasks:
        file: ../../tasks/docker-compose-up.yml
```

### Style rules

- **Indentation**: 2 spaces, always
- **Document start**: playbook `main.yml` files begin with `---`; shared task files and
  docker-compose files do not
- **Play naming**: `Install <lowercase-service-name>` (sentence case, service name lowercase)
- **Task naming**: sentence case, action-oriented (e.g., `Copy docker-compose.yml`,
  `Enable UFW`, `Allow bedrock port`)
- **Module names**: always use Fully Qualified Collection Names (FQCNs):
  `ansible.builtin.template`, `community.general.ufw`, `community.docker.docker_compose_v2`
- **Booleans**: use `true`/`false`, never `yes`/`no`
- **File mode**: always a quoted string: `"0640"`, `"0644"`, `"0755"`. Required
  on `ansible.builtin.file` with `state: directory` too (ansible-lint
  `risky-file-permissions`)
- **Variable references**: `{{ var }}` with spaces inside braces
- **Privilege escalation**: not specified (hosts are accessed as root via Tailscale SSH)
- **Edge hosts** (e.g. `vps`): reached over plain SSH via the operator's
  `~/.ssh/config`, not the tailnet — set `ansible_user: root` in the play,
  allow SSH and required agent ports before enabling UFW, skip Alloy without
  a Loki route, and opt out of CI with `.no-ci`. The VPS runs Beszel in SSH
  mode; restrict its agent port to the hub's verified egress IP.
- **Task directive order**: `name:` first, then module with parameters, then conditionals
- **Hosts value**: must match the playbook directory name and the inventory hostname
- **Use `ansible.builtin.template`** for copying compose files (even if no variables),
  not `ansible.builtin.copy`

### Task ordering within a playbook

1. Include `install-docker.yml`
2. Include `install-common-packages.yml`
3. Include `install-alloy.yml`
4. Include `install-beszel-agent.yml`
5. Copy compose file(s) and any extra config files
6. UFW allow rules for specific ports (if needed)
7. Enable UFW with deny-all policy
8. Include `docker-compose-up.yml` (always last)

Hosts without Docker (e.g. `cinema`) skip `install-docker.yml` and set
`alloy_docker_discovery: false` in the play's `vars:` so Alloy ships journal
logs only. The `vps` play skips `install-alloy.yml` because it has no route to
Loki. It includes `install-beszel-agent.yml` in SSH mode and allows the agent
port only from the hub's configured egress IP.

## Docker Compose Conventions

- **No `version:` key** (it is deprecated)
- **Image registry**: always fully qualified -- `docker.io/`, `ghcr.io/`, `quay.io/`
  - Official images: `docker.io/library/postgres:15-alpine`
  - Third-party: `docker.io/grafana/grafana:12.4.1`
- **Image tags**: always pinned to a specific version, never `latest`
- **`container_name`**: always specified explicitly
- **`restart`**: `always` for primary services, `unless-stopped` for sidecars/helpers
- **Volumes**: short string syntax (`./data:/data`), read-only with `:ro` suffix
- **Environment variables**: prefer list form (`- KEY=value`); map form (`KEY: value`)
  is acceptable but less common. In map form, **quote** Jinja values:
  `APP_SECRET: "{{ umami_app_secret }}"`. Bare `KEY: {{ var }}` is a YAML flow
  mapping, so the file does not parse and the rendered value breaks on a secret
  containing `:` or `#`
- **Networks**: named networks matching the service name, used when a Newt reverse proxy
  sidecar is present
- **Newt sidecar pattern** (for Pangolin reverse proxy tunneling):
  ```yaml
  newt:
    image: docker.io/fosrl/newt:<version>
    container_name: newt-<service-name>
    restart: unless-stopped
    networks:
      - <service-name>
    environment:
      - PANGOLIN_ENDPOINT={{ pangolin_endpoint }}
      - NEWT_ID={{ service_newt_id }}
      - NEWT_SECRET={{ service_newt_secret }}
  ```

## Bash Script Conventions

- Shebang: `#!/usr/bin/env bash`; strict mode: `set -euo pipefail`
- Variables: `UPPER_SNAKE_CASE`, always double-quoted (`"$VAR"`)
- Temp file cleanup: use `trap ... EXIT`

## CI / GitHub Actions Conventions

- Workflow and step names: sentence case; action versions: major-version pinning (`@v6`)
- Runner: `ubuntu-latest`; multi-line shell: `run: |` (literal block scalar)
- Automated commit messages: conventional commits format (`chore: update containers table`)

## Dependency Management

- **Renovate** opens PRs monthly (`renovate.json`). Nothing auto-merges -- merging
  to `main` applies to live hosts, so every update is reviewed and merged by hand
- Container image versions are pinned in each `docker-compose.yml`
- Ansible Galaxy collections: `ansible.posix`, `community.docker`, `community.general`

### Publishing the custom tailnet Caddy image

For `just publish-tailnet-caddy`, read the Caddy base version from
`docker/tailnet-caddy/Dockerfile` and keep the recipe tag's version aligned with
it. After publishing, inspect the exact registry tag and digest; verify the
image's `caddy version`; then update every `homelab-caddy-cloudflare` reference
in the Ansible role and Kubernetes charts to the same `tag@sha256:digest`.
Publishing changes the registry only. Deploy the Ansible role explicitly;
Homepage and Kiwix require separate approved Argo CD syncs after the push.
Confirm the running container
image IDs after each deployment before calling the update live.

## Adding a New Service

1. Add host to `ansible/inventory` under `[homelab]` (alphabetical order)
2. Create `ansible/playbooks/<service>/main.yml` following the canonical structure above
3. Create `ansible/playbooks/<service>/docker-compose.yml` following compose conventions
4. Add an import line in `ansible/playbooks/main.yml`
5. If using secrets, reference via `{{ variable }}` and define in `group_vars/all/secrets.yaml`.
   Non-secret shared values go in `group_vars/all/vars.yaml` (unencrypted)
6. Run `just lint` from the repo root before opening a PR
7. If the host must never run in CI (edge infra like the VPS), add a
   `.no-ci` marker file in the playbook directory with a comment explaining why

## Shared Variables

- `group_vars/all/secrets.yaml` -- ansible-vault encrypted; edit with `just edit-secrets`
- `group_vars/all/vars.yaml` -- plaintext, non-secret values shared by every host
- `alloy_docker_discovery` -- set `false` in a play's `vars:` for hosts without
  Docker, so `tasks/config.alloy.j2` renders a journal-only Alloy config
- `beszel_version` -- host agent `.deb` version in
  `tasks/install-beszel-agent.yml`. The hub image is pinned in
  `kubernetes/apps/beszel/values.production.yaml`; Renovate groups both updates
- `ufw_docker_allowed_sources` -- sources allowed to reach published container
  ports, defaulting to the tailnet only. Override in a play's `vars:` for a
  service that must be reachable from the LAN (see `bedrock`)
- `ufw_docker_public_ports` -- list of `{port, proto}` published container ports
  reachable from anywhere (default `[]`). For edge hosts like `vps` whose
  ports are intentionally public; narrower than widening
  `ufw_docker_allowed_sources`, which would open every published port on the host
- `homelab_lan_subnet` -- the LAN range, for UFW rules

## Firewalling Published Container Ports

`community.general.ufw` with `policy: deny` does **not** cover ports published
by Docker. Docker DNATs in `nat/PREROUTING` and jumps to `DOCKER-USER` at the
top of `FORWARD`, both before ufw's `ufw-user-forward` chain is consulted.

`tasks/install-docker.yml` therefore manages a `DOCKER-USER` block in
`/etc/ufw/after.rules` that allows Docker's own bridge networks plus
`ufw_docker_allowed_sources`, and drops everything else headed for a container
network. Services using `network_mode: host` (adguard-home) are
unaffected -- their traffic hits `INPUT`, where the ufw rules do apply.

A `ufw: rule: allow` task on its own only opens a *host* port. If a service
needs to be reachable from the LAN, it also needs `ufw_docker_allowed_sources`
widened in its play. If a service's ports must be reachable from the public
internet (the `vps` VPS), list them in `ufw_docker_public_ports` instead --
specific ports, any source, without opening every published port on the host.
For a trusted source that needs just one published port, use
`ufw_docker_allowed_source_ports` with `{port, sources}`; widening
`ufw_docker_allowed_sources` opens every published container port.

## Talos Kubernetes Pilot

For Talos VM, Kubernetes workload, Homepage cutover, or backup work, first read
`kubernetes/README.md` and `kubernetes/cluster/README.md`. Manage
infrastructure through `ansible/playbooks/talos-cluster/`; these manual
playbooks are separate from the all-host playbook and CI flows. Pin Kubernetes
container images to both a version and SHA digest.

Use kubectl's default kubeconfig and current context when they resolve to the
intended cluster. Before a cluster mutation, verify the API server and expected
node names; specify `--context` only when selecting a real alternate context.
Get Dave's explicit approval in the current conversation before rebooting any
Kubernetes worker or control-plane node. A runbook step that says a reboot is
required is not approval; explain the target, reason, expected workload impact,
and recovery checks, then wait for approval.
For gated multi-step work, verify each gate against live state. A failed gate
freezes that step: retain evidence, report the result, and leave completed steps
in place unless Dave explicitly authorizes a state reversal in the moment, or
the frozen state is actively harming workloads or data.

The Talos playbooks load Vault variables during syntax checking; set
`ANSIBLE_VAULT_PASSWORD_FILE` to the protected local password helper when
running `just lint` locally. Pass `--vault-password-file` to manual playbooks
too: encrypted group variables load even when the play does not reference a
secret.

- For Kubernetes behavior, use source charts, values, and policy YAML; check
  live resources for deployed state. Keep this file to non-obvious constraints
  instead of repeating configuration.
- When changing Homepage networking, verify a real widget/API call and the
  TrueNAS WebSocket; a dashboard-root request alone does not cover them.
- Talos enforces the cluster-wide Pod Security baseline. A sidecar needing
  `NET_ADMIN`, `NET_RAW`, or `/dev/net/tun` requires a separately reviewed
  policy exception.
- Renovate manages Talos patch pins in `kubernetes/cluster/vars.yaml` within
  the current minor, updating the version, ISO filename, and ISO SHA256
  together. This changes rebuild inputs only; it does not upgrade live nodes.
  For an OS upgrade, follow the Talos runbook, upgrade workers before the
  control plane, verify each node, then reconcile the pins and regenerate
  configs as directed there. Minor upgrades remain manual and must step through
  each minor.
- Headscale's `given_name` controls the MagicDNS label. Homepage now runs on
  Talos; its LXC playbook and LXC rollback path are retired.
- Homepage's local PVCs and Pod are pinned to `k8s-w-1`. Proxmox VM backups
  cover those disks; an off-host etcd snapshot covers cluster state. Read the
  recovery constraints in the cluster README before restoring a VM.
- Before any Talos VM resize, update the node's desired `vcpu`, `memory_mib`,
  or `disk_gib` in `kubernetes/cluster/vars.yaml`; keep `allowed_memory_mib`
  aligned with the current and target memory sizes. Use the guarded resize
  playbook for the resource being changed. Proxmox sizing is not part of the
  generated Talos machine config. Control-plane memory changes require the
  dedicated playbook, its health/backup gates, and explicit reboot approval.
- Render Kubernetes Secrets from Ansible Vault into protected temporary files
  through the apply playbook. Keep secret values out of Git and command output.
- Alloy's Talos DaemonSet reads pod logs through the Kubernetes API with a
  node-local pod selector. It cannot read Talos node service logs. Its Loki
  LAN firewall exception is scoped to TCP 3100 from the Talos node IPs.
- Before adding Loki sources, check Grafana's `df -i /` as well as `df -h /`.
  Loki chunks exhausted the 12 GiB LXC's inodes while 2.1 GiB of blocks
  remained free; writes to Loki's WAL and chunks then failed. Expand its root
  disk through `expand-grafana-rootfs.yml` after confirming an off-host backup.
  Reapplying the Grafana Compose stack restarts Loki; wait for `/ready` to
  recover before checking ingestion.