# AGENTS.md

Concise repository guidance for agentic coding agents. Keep this file short: add detailed runbooks
under `docs/`, and only keep high-signal reminders here.

## Repository Overview

Self-hosted K3s homelab via GitOps. Principles: declarative infra, repeatable automation, small blast radius, version pinning, minimal drift.

**Tech Stack:** K3s, ArgoCD, Helm (template-only), Ansible (bootstrap), External Secrets + Vault, cert-manager, Envoy Gateway (Gateway API), Renovate, ZFS, Cilium (BGP).

**Cluster Access:** This is a GitOps repository. `kubectl --context=grigri`. Metrics: https://prometheus.internal.grigri.cloud

## Repository Map

```
apps/      # Application Helm charts, one release per folder
system/    # Cluster-wide components: ingress, cert-manager, monitoring, identity
platform/  # Supporting operators/services: Vault, external-secrets, git, reloader
bootstrap/ # ArgoCD bootstrap layer
metal/     # Ansible inventory, playbooks, roles for K3s node provisioning
scripts/   # Utility scripts, including deploy-dir.rh for Helm rendering
test/      # Local k3d test helpers
docs/      # MkDocs documentation and runbooks
```

## Build/Lint/Test Commands

```bash
make git-hooks
pre-commit run --all-files
helm dependency build apps/<name>/
helm template --include-crds --namespace <namespace> <release-name> apps/<name>/
helm lint apps/<name>/
```

## Workflows

**Deploying a new application:**
1. Read `docs/conventions/deploying-new-apps.md` — decision trees, patterns, checklist, reference paths (OIDC, Zalando, MinIO)
2. Check existing deployments: `apps/immich/`, `apps/readest/`
3. Run validation commands before committing

## Code Style

- **YAML:** 2-space indent, `---` for multi-doc, yamllint via pre-commit (ignores `templates/`)
- **Helm:** Prefer maintained official/first-party charts; use `app-template` only when none exists. See `docs/conventions/helm.md`
- **Ansible:** `safety` profile. See `docs/conventions/ansible.md`
- **Renovate:** Add hints above image refs — `docs/conventions/helm.md#renovate-integration`
- **Docs:** Troubleshooting in `docs/troubleshooting/`; `AGENTS.md` for links/reminders. See `docs/conventions/documenting-learnings.md`
- **Security:** No hardcoded secrets. ExternalSecret → Vault. Public repo hygiene
- **Personal information:** never commit people's names (household members, children, guests),
  addresses, locations, phone or person details, or which account/token holds which privileges.
  Use roles or rooms instead ("owner", "Office T"). Applies to code, docs, commit messages and PRs
- **Vault CLI:** `export VAULT_ADDR=https://vault.internal.grigri.cloud`; reuses the owner's OIDC login.
  KV v2 mount `secret`: ExternalSecret `remoteRef.key: /<app>/<path>` ↔ `vault kv put secret/<app>/<path>`.
  Never print secret values; load them from a `0600` file and shred it afterwards

## Commit Messages

**Format:** `<scope>: <imperative-description>` — scope is directory or service name. Special scopes: `metal` (Ansible), `docs`, `renovate`.

```
kube-system: Add kata-nvidia coordinator operator
vault: Change to svg logo
monitoring: Update Helm release kube-prometheus-stack to v82.10.1
```

No attribution lines, AI session links or session trailers in commits, PR bodies or comments — the
repos are public (the `no-session-links` commit-msg hook enforces it). Keep GPG signing.

## Deployment Rules

**Commit first, then apply.** Git must contain the change before it is applied anywhere. Never apply
an uncommitted or partially edited tree.

**Owner approval for live changes.** Propose Home Assistant changes made outside git (dashboards,
entity/area registry, integration options) and pushes, then wait for an explicit yes per item.
Analysis and read-only checks need no approval.

### Agents may run

- Read-only inspection: `kubectl get/describe/logs/top`, Prometheus/Grafana/Loki queries,
  `helm template`, `helm lint`, `helm dependency build`, `pre-commit run`
- **Narrowly scoped `metal/` Ansible** for config that is already committed, using explicit
  `--tags` and `--limit`. Example: `cd metal && ANSIBLE_EXTRA_ARGS="-t zfs-config --limit prusik" make prepare`
- **Scoped `k3s` role config apply** for an already-committed, **config-only** change:
  `cd metal && ANSIBLE_EXTRA_ARGS="-t k3s --limit <node>" make cluster`. Re-templates
  `/etc/rancher/k3s/config.yaml` and `k3s.service`, restarts k3s if config changed (idempotent —
  binary install gated on `/usr/local/bin/k3s` absent). Causes ~30–75s API downtime on `prusik`.
  Requires: committed, config-only (no `k3s_version` bump), explicit `--limit`
- ArgoCD sync of an already-committed and pushed change

Report the command and outcome. Prefer narrowest tag/limit. Target by playbook: `prepare.yml`
roles (`prepare`, `setup` incl. `zfs-config`, `swap`, `nvidia-container-runtime`) → `make prepare`;
`cluster.yml` roles (`ntp`, `gvisor`, `k3s`) → `make cluster`. Always `-t <tag> --limit <node>`.

### Never run

- **`kubectl apply/delete/edit/patch`** on cluster resources — ArgoCD `selfHeal: true` with real-time
  watches reverts these almost instantly
- **Node lifecycle / high blast radius:** `make bootstrap`, `make dev`, `make uninstall-k3s`,
  unscoped/all-node `make cluster`, `-t k3s` without `--limit`, k3s version bumps via Ansible
  (changing `k3s_version` or re-downloading binary). These can take down the sole control-plane
  node. Exception: scoped, committed, config-only `-t k3s --limit <node>` — see "Agents may run"
- **Anything data-destroying:** PVC/PV deletion, `zpool destroy`, `zfs destroy`, pool re-creation,
  disk formatting (`prepare_additional_disks_force_format`)
- Untagged or unlimited `make prepare` / `ansible-playbook` runs — always scope them
- `helm install/upgrade` — let ArgoCD do it

When in doubt, or when a change needs a hard prohibition above, stop and hand the user the exact
command.

## Skill Usage

- `debug` skill: live cluster investigation with read-only `kubectl` and Grafana observability only
- Do not copy runbook steps into agent instructions — link to docs instead
- Prefer focused, path-specific docs over broad always-loaded guidance
- Keep durable knowledge in this repo (docs, `AGENTS.md`), not in agent-local memory
- Subagents default to a mid-tier model; use the top model only for genuinely hard tasks, and do not
  spawn agents for work that can be done inline

## Common Pitfalls

Full pitfall index (78 entries grouped by subsystem): `docs/troubleshooting/INDEX.md`

**Pre-deployment rules** (no separate doc — must be known before starting work):

- Forgetting `helm dependency build` after updating `Chart.yaml`
- Missing Renovate hints causes images to not auto-update
- Introducing CRDs without `--include-crds` in helm template
- Not waiting for webhooks (cert-manager, external-secrets) before applying dependent resources
- Forgetting `Prune=false` on PVCs causes data loss on sync
- Comments inside embedded config blocks (strings in `values.yaml` rendered into a ConfigMap, e.g. Alloy, Vector, Loki) are live config: editing them restarts the workload; treat such edits as config changes
- external-dns >= v0.22 reads only `external-dns.kubernetes.io/*`; `external-dns.alpha.kubernetes.io/*` is
  silently ignored (`docs/network/ingress.md#dns-external-dns`)
- Freshly joined node shows `Ready,SchedulingDisabled` for ~1 min: `k3s-agent` system-upgrade Plan
  has `cordon: true`. Don't uncordon by hand — resolves automatically
- Apps with init-container workarounds: log version info, fail loudly (`exit 1`) on mismatch so
  Renovate bumps surface breakage. Check init container logs after version bumps
- ArgoCD `selfHeal: true` reverts `kubectl apply` almost instantly — commit/push first, let ArgoCD
  sync. See `docs/troubleshooting/argocd-gitops-workflow.md`
- Never `kubectl config use-context`/`set-context` (rewrites the owner's kubeconfig default): always
  pass `--context`. The shell is zsh: unquoted variables do not word-split, put multi-step kubectl
  helpers in a bash script

**Subsystem index** (top pitfalls per group — see `docs/troubleshooting/INDEX.md` for full entries):

- **Kubernetes & GitOps** (15): node addition, app-template `tpl` evaluation, topology spread constraints, `eviction-hard` replaces k3s defaults
- **ArgoCD** (6): repo-server probe death-spiral, kustomize OCI auth override, Helm-hook mixing and
  hook-only edits that never converge (`docs/troubleshooting/argocd-helm-hook-provisioning.md`)
- **Monitoring** (9): Grafana sidecar collisions, Vector log loss, Tempo WAL path collision, DiskPressure Loki gaps
- **Networking** (9): Cilium stale BPF datapath, Armbian kernel BPF breakage, nodelocaldns LRP, pfSense alias change keeps old states (`pfsense-alias-change-keeps-old-states.md`), Envoy listener drain (`envoy-gateway-listener-drain.md`), Envoy debugging (`envoy-gateway-debugging.md`)
- **Storage & Backup** (12): prusik RAM-bound latency, ZFS orphan leak, Velero backup chains, CI DinD disk fill
- **Ansible & Infra** (2): cross-host facts in scoped runs, Ubuntu 26.04 compat
- **LLM Infrastructure** (3): Hermes config overwrite, auth failures, persistent CLI tools
- **Desktop & Home** (12): Home Assistant venv/aiohttp/localtuya, localtuya thermostat `hvac_action` needs `heuristic_action`, UI-storage dashboards edited over the websocket API, PVPC negative prices, Shelly CoIoT push, never disable CoIoT on a Gen1 Shelly (breaks HA setup *and* reboot-loops the device when MQTT is on), renaming an HA entity moves its recorder history (use remove-then-promote), Omnibattery `backup_mode` latch (backup threshold 500 W workaround)
- **Applications** (11): Supabase/Zalando bootstrap, GoTrue CORS, Firecrawl nuq schema, Firecrawl
  egress NP probe-timeout rollback (root cause unknown, aggregate denied counters prove nothing)

## Subsystem Docs

- **Cilium:** `docs/conventions/cilium.md` — BGP, TCX, bandwidth limiting
- **Ingress:** `docs/network/ingress.md`: Envoy Gateways, pfSense alias/overrides, DNS, rollback; routes: `docs/conventions/gateway-api-routes.md`
- **Deploying new apps:** `docs/conventions/deploying-new-apps.md` — decision-making, patterns, checklist
- **Kanidm:** `docs/user-guide/kanidm.md` — Kaniop CRDs, OAuth2 clients, group membership rules (person membership manual, never via CRD)
- **Documenting learnings:** `docs/conventions/documenting-learnings.md` — when/how to write troubleshooting docs
- **prusik storage:** `docs/conventions/prusik-fast-storage-tier.md` — ZFS pool sizing, SSD selection, PVC migration. Hardware: `docs/hardware/prusik.md`, `docs/hardware/k8s-amd64-1.md`
- **Adding a node:** `docs/user-guide/add-or-remove-nodes.md` (runbook), `planning/k8s-amd64-1-node-addition.md` (worked example)
- **Topology spread:** `docs/conventions/topology-spread-constraints.md` — `matchLabelKeys`
- **ESPHome devices (flash, logs, rename):** `apps/esphome/config/README.md` ("Agent Runbook"); hardening TODO `planning/esphome-security-hardening-todo.md`
- **Battery optimizer:** app code in `pando85/home-energy-optimizer`; deployment + extraction plan `planning/energy-baseline-production-execution.md`; live control rollout/rollback `docs/deployment/battery-optimizer.md#live-battery-control`

## Licensing

GPLv3 (see LICENSE.md). Generated code must be compatible.