---
name: homelab
description: Operational conventions for the HomeLab repository. Activates for any task that interacts with this repo's OKD or MicroShift clusters, applies manifests, runs Ansible against the fleet, or modifies cluster state. Also indexes other project-local skills.
---

# HomeLab Skill

Operational conventions for this HomeLab repository. Read this in full before
acting on cluster, manifest, or Ansible changes. The companion file
[AGENTS.md](AGENTS.md) expands on directory layout and common tasks.

## Related Skills

- [Kubernetes (KubeShark)](.agents/skills/kubernetes-skill/SKILL.md) —
  production-grade Kubernetes manifest, Helm, and Kustomize guidance with a
  failure-mode workflow and Conditional Reference Retrieval. Required for any
  Kubernetes resource design, review, or refactor. OKD-specific patterns
  (SCCs, Routes, arbitrary UID images) live in its `openshift-patterns` CRR
  reference.

## Cluster Inventory

| Cluster    | Distribution        | Nodes                                                | Storage   | KUBECONFIG               |
| ---------- | ------------------- | ---------------------------------------------------- | --------- | ------------------------ |
| OKD        | OKD 4.22 (SCOS 10)  | 3 control-plane (`server-1/2/3`) + workers + `gpu-1` | Rook-Ceph | `$HOME/.kube/okd`        |
| MicroShift | MicroShift on RPi 5 | Single edge node (PiHole, edge DNS)                  | Local     | `$HOME/.kube/microshift` |

GitOps: ArgoCD reconciles [kubernetes/](kubernetes/) into both clusters via
per-app overlays (`overlays/okd`, `overlays/microshift`). Other active
overlays across the repo: `okd-sandbox`, `okd-unas`, `sandbox`, `sno`,
`microshift-unas`, `old`, `operator` (all OpenShift/OKD-based) and `k3s`
(legacy, the only non-OpenShift target).

## Cluster Authentication

Select the target cluster by exporting `KUBECONFIG` before any `kubectl`,
`oc`, `helm`, `kustomize`, or `argocd` call:

- OKD: `export KUBECONFIG=$HOME/.kube/okd`
- MicroShift: `export KUBECONFIG=$HOME/.kube/microshift`

Both kubeconfigs grant **cluster-admin**. Caution rules:

- Prefer read-only commands (`get`, `describe`, `--dry-run=client`,
  `kustomize build`) when exploring.
- Confirm with the user before any mutating action: `apply`, `delete`,
  `patch`, `scale`, `drain`, `cordon`, `rollout restart`, `evict`.
- Never `oc adm` or `kubectl exec` into a production workload without
  explicit user confirmation.
- Mutations to ArgoCD-managed resources will be reverted on next sync — the
  correct fix is almost always a PR to [kubernetes/](kubernetes/), not a
  live `kubectl` change.

### Debug and ad-hoc pods

Every pod you create by hand (`kubectl run`, `kubectl debug`, throwaway
`kubectl apply`) must satisfy the OpenShift `restricted-v2` SCC and the
`restricted` PodSecurity profile. Most namespaces audit or enforce
`restricted`, so a non-compliant pod triggers the "workload did not match the
PodSecurity restricted profile" alert or is rejected. Required settings:

- Pod level: `runAsNonRoot: true` and `seccompProfile.type: RuntimeDefault`.
- Every container (including init containers):
  `allowPrivilegeEscalation: false` and `capabilities.drop: ["ALL"]`.
- Do **not** set `runAsUser`, `runAsGroup`, `fsGroup`, or `seLinuxOptions`.
  `restricted-v2` assigns them from the namespace UID range, and values
  outside that range are rejected.
- No `privileged`, `hostNetwork`, `hostPID`, `hostIPC`, `hostPath`, host
  ports, or added capabilities. Use an image that runs as an arbitrary
  non-root UID (for example the `toolbox` image from
  [containers/](containers/)), not one that needs root.
- Name the pod `debug-<purpose>`, use `--rm` and `--restart=Never`, and
  delete it when done.

Template (the container name must equal the pod name so `--overrides` merges
into the generated container; set the command inside the override because
`--overrides` drops a trailing `-- sh`):

```bash
kubectl run debug-<purpose> -n <namespace> --rm -it --restart=Never \
  --image=<image> \
  --overrides='{"spec":{"securityContext":{"runAsNonRoot":true,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"debug-<purpose>","image":"<image>","command":["sh"],"stdin":true,"tty":true,"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}'
```

After creating a pod, check `kubectl get pod <name> -o jsonpath='{.metadata.annotations.openshift\.io/scc}'`
returns `restricted-v2`. If it is anything else, delete the pod and fix the
spec. Only use `oc debug node/` when node access is genuinely required, and
confirm with the user first.

## Repository Workflow

### Adding or editing a Kubernetes app

1. Place manifests under `kubernetes/<app>/base/` and per-cluster overlays
   under `kubernetes/<app>/overlays/{okd,microshift}/`.
2. Register the app in [kubernetes/argocd/applications/](kubernetes/argocd/applications/)
   as `<app>.yaml` and add it to the local `kustomization.yaml`.
3. Run `./main.bash kustomize_fix --dir kubernetes/<app>` to apply
   `kustomize edit fix`, prepend the `---` document marker, and
   prettier-format.
4. Validate with `./main.bash test_overlays` (requires `VAULT_ADDR` and
   `VAULT_TOKEN`; runs `argocd-vault-plugin` + `kubeconform -strict` against
   a sibling `../kubernetes-json-schema/` checkout).
5. Follow the [Kubernetes (KubeShark) skill](.agents/skills/kubernetes-skill/SKILL.md)
   for resource design (security context, probes, resources, RBAC,
   NetworkPolicy, rollout strategy).

### Secrets

- **Prefer [External Secrets Operator](kubernetes/external-secrets-operator/)**
  for any new secret: declare an `ExternalSecret` (or `ClusterExternalSecret`)
  that pulls from Vault via the configured `ClusterSecretStore`. This keeps
  the secret reconciled, rotatable, and visible as a first-class Kubernetes
  resource.
- Fall back to **argocd-vault-plugin** placeholders (`<path:secret/data/...#key>`)
  only when ESO cannot express the requirement (e.g., templated values
  injected into a non-Secret field, or bootstrap manifests that must render
  before ESO is available).
- Never commit plaintext secrets, kubeconfigs, tokens, or `.env` files.
  `gitleaks` runs in pre-commit and CI.

### Ansible

- Entry playbooks: `ansible/servers.yaml`, `microshift.yaml`, `desktop.yaml`,
  `laptop.yaml`, `vscode-server.yaml`.
- Inventory: [ansible/inventory](ansible/inventory).
- Standard invocation:
  `ansible-playbook -i ansible/inventory --ask-become-pass --ask-pass ansible/<playbook>.yaml`
  (the `./main.bash ansible` helper runs `servers.yaml`).
- `ansible-lint` must pass; preserve idempotency.

### CI Validation (k8s-gitops-ci)

- Real CI runs via `.tekton/gitops-ci.yaml`'s `gitops-ci` Task, invoking
  `k8s-gitops-ci pipeline --dirs "kubernetes/,tekton/,.tekton/,okd/"
--assume-openshift` from the sibling `../k8s-gitops-ci` repo (see
  `tekton/base/gitops-ci.yaml`).
- **Always pass `--assume-openshift`** when validating locally — it matches
  real CI and tells the sync-options check that OpenShift/OKD built-in API
  groups (`gateway.networking.k8s.io`, `monitoring.coreos.com`, `metal3.io`,
  Routes, SCCs, ...) are guaranteed present at first sync, so those resources
  do **not** need the `SkipDryRunOnMissingResource=true` annotation. Without
  it you get spurious sync-options findings for cluster-native CRDs.
- For local iteration also pass `--disable-checks avp` unless you have
  `VAULT_ADDR`/`VAULT_TOKEN` exported — otherwise AVP placeholder resolution
  fails the overlay build with unrelated "could not replace all placeholders"
  noise.
- **Working on a subset?** Scope validation to just that app/overlay instead
  of a full sweep:
  `k8s-gitops-ci test-all --app kubernetes/<app> --cluster <cluster> --assume-openshift --disable-checks avp`
  (repeatable flags; `--app` alone validates every overlay of that app,
  `--cluster` alone validates every app targeting that cluster).
- Before pushing, confirm the full CI scope still passes:
  `k8s-gitops-ci test-all kubernetes tekton .tekton okd --assume-openshift --disable-checks avp`
  (mirrors real CI; defaults to reading local `test.sh` automatically — no
  PR needed).
- Avoid `test-all .` (full-repo scan) — includes ansible/, machineConfigs/,
  notes/, sandbox/, which are outside CI's actual `--dirs` scope and
  produces irrelevant noise.
- After changing `../k8s-gitops-ci` source, rebuild before testing:
  `cd ../k8s-gitops-ci && task build`.
- Some fixes require changes in **both** repos (e.g., adjusting a check's
  behavior, adding a new exemption mechanism) — check `../k8s-gitops-ci`'s
  own `AGENTS.md`/`docs/` when validation behavior itself needs to change.
- Exemptions: `test.sh` with `export EXEMPTIONS=(...)` at an app root or any
  directory with non-Kubernetes YAML — see `okd/test.sh` for a working
  example and `http://raw.githubusercontent.com/ArthurVardevanyan/k8s-gitops-ci/refs/heads/main/.agents/skills/exemptions/SKILL.md` for the
  full reference.
- ArgoCD sync-options: any resource whose API group isn't guaranteed present
  at first ArgoCD sync needs
  `argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true`. With
  `--assume-openshift` (real CI), OpenShift/OKD built-in API groups count as
  guaranteed-present, so only genuinely third-party CRDs (installed by
  operators this cluster may not have yet) require the annotation.

### OKD provisioning

- [okd/](okd/) holds `agent-config.yaml` and `install-config.yaml` for the
  OKD agent-based installer.
- [machineConfigs/](machineConfigs/) holds Ignition and preseed configs for
  bare-metal OS provisioning.
- Treat both as bootstrap inputs; changes are rare and high-risk — require
  explicit user confirmation.

### Lint and formatting

`pre-commit` is authoritative ([.pre-commit-config.yaml](.pre-commit-config.yaml)):

- `gitleaks`, `shellcheck`, `yamllint`, `markdownlint`, `prettier`,
  `ansible-lint`, `checkov_diff`
- Commit messages must satisfy `conventional-pre-commit`
  (`feat:`, `fix:`, `chore:`, `build(deps):`, `docs:`, ...).

YAML rules: 2-space indent, `---` document marker at top, no line-length
limit. Shell scripts: `set -o errexit -o nounset -o pipefail` and
`shopt -s failglob`; see [main.bash](main.bash).

### Renovate

Pinned versions (shell scripts, pre-commit, manifests, Helm/Kustomize
inputs) are kept current by Renovate. Preserve `# renovate: datasource=...`
comments directly above the pinned version line.

## main.bash Reference

The wrapper script exposes these functions; invoke as `./main.bash <fn>`:

| Function                  | Purpose                                                          |
| ------------------------- | ---------------------------------------------------------------- |
| `ansible`                 | Run `ansible/servers.yaml`                                       |
| `kustomize_fix --all`     | Run `kustomize edit fix` across the whole repo, then prettier    |
| `kustomize_fix --dir <d>` | Same, scoped to one directory                                    |
| `test_overlays`           | Build all OKD overlays through AVP → `kubeconform -strict`       |
| `test_overlays_k3s`       | Same for `overlays/k3s` (legacy clusters)                        |
| `stateful_workload_stop`  | Suspend CronJobs and scale stateful apps to zero for maintenance |

## Out of Scope

- [notes/](notes/) and [sandbox/](sandbox/) are scratch. Read for context;
  do not deploy from them and do not treat their YAML as authoritative.
- `CHANGELOG.md` is auto-maintained; do not hand-edit.
- `img/` holds documentation assets only.