# AGENTS.md

> Collaboration guide for AI coding agents and human contributors. Read this before touching code.

## Project overview

aibox is a lightweight, pure-bash module manager (zero runtime dependencies, compatible with the bash 3.2 that ships with macOS). One-line `curl|bash` install; install / update / uninstall each module on demand. Each "module" is a directory under `tools/<name>/` in the repo, shipping its own `install / uninstall / update / svc` hooks and dispatched uniformly by the `aibox` main CLI.

## Repo layout

```text
bin/aibox            main CLI — GENERATED artifact: edit src/aibox/*.sh, run scripts/bundle.sh
src/aibox/*.sh       the manager's actual sources (concatenated in lexical order into bin/aibox)
scripts/bundle.sh    bundler: write / --check (CI gate) / --print / --out
                     (the single-file shape is a RELEASE constraint, not a source one — this is why)
install.sh           bootstrap (curl|bash install / self-update, idempotent; optional AIBOX_SHA256/AIBOX_VERIFY)
tools/<name>/        module dir: lib.sh + install/uninstall/update/svc.sh + module.yaml
                     shipped: pi-web (macOS launchd service), openmaic (Linux deploy-host ops CLI),
                              windmill (Windmill self-host docker compose ops CLI), clash (Clash subscription proxy pool, mihomo), base (shared PG+Redis)
tools/_shared/       shared module library (common.sh: output helpers + docker.io pool) — single source in the
                     repo; ships into each module cache as _common.sh via module.yaml `includes: [common]`
docs/module-spec.md  the module hook contract (normative; module-system-spec.md is the superseded design draft)
.github/workflows/   CI (release automation + lint quality gate: bash -n / shellcheck / gotcha #1 #8 scans / module-lint / port-conflict / bats tests + coverage probe / macos-bash32 job — the full suite executed under /bin/bash 3.2, the platform this project promises)
```

## Core commands

```text
aibox install <module> [--skip-checks]
aibox uninstall <module>|self [--purge] [--yes]   # --purge: also delete data; self = manager (--purge = cascade full teardown)
aibox update <module>|self|--all [--restart|--no-restart] [--skip-checks]   # self = aibox itself; --all = modules + self
aibox check <module>|self                    # module preflight; self = environment check
aibox status [--available] [<module>]     # overview (versions+endpoints+credentials+ports) / catalog / detail+health
aibox dashboard [--once|--json|--pane P] [--interval N]   # htop-style LIVE view (read-only): modules ·
                                          # containers · upgrades · residue panes; keys Tab/↑↓/d///s/p/+/-/r/?/q
aibox autoclean [<module>...|self] [--apply] [--stop] [--yes]  # residue scan/cleanup (dry-run default)
aibox <module> <action> [args]            # pass-through to module svc.sh
aibox proxy {show|set <url>|unset|on|off|check [url]|env}   # static proxy config (global; see README "Proxy")
aibox --no-proxy <command>                                # bypass the proxy once
aibox clash {set <sub-url>|on|off|status|refresh|select|test|logs|doctor}  # clash pool (mihomo; see tools/clash/README)
```

## Contribution conventions

### Editing the manager (mandatory)

`bin/aibox` is a **generated artifact** — never hand-edit it. Sources live in `src/aibox/*.sh`
(one file per domain, concatenated in lexical order; `00-head.sh` carries the shebang and
`set -euo pipefail`). After any source change run `scripts/bundle.sh` and commit the
regenerated artifact; CI runs `scripts/bundle.sh --check` and fails on drift. Rationale:
the shipped file must stay a single file (curl|bash), but that is a release-shape constraint —
keeping 5k lines in one hand-edited file forced "inline twins" of shared helpers that kept
drifting. `tests/bundle.bats` locks the contract (fragments parse standalone, only the head
fragment has a shebang, artifact == concatenation).

### Test environment (mandatory)

**Never run any aibox functionality on the host.** Not `bats tests/*.bats`, not
`bin/aibox …`, not a module hook, not an extracted helper, not "just this one
function", not even for a read-only check. The development machine is the operator's
live host: a stray install/uninstall/start touches real deployments, and a local run
proves nothing about the environment that matters — the project promises bash 3.2 and
BSD tools, which a Linux/host run silently does not exercise (that mismatch has
produced false "green" more than once, e.g. a `find -quit` that BSD find rejects).

Everything runs inside the pinned container image (`tests/docker/Dockerfile`,
`aibox-test:local`):

```bash
tests/docker/run.sh                  # gates -> fast suite as root AND non-root -> summary
tests/docker/run.sh --fast-only      # the fast suite only
tests/docker/run.sh --integration    # + daemon-backed suites (mounts the docker socket)
tests/docker/run.sh --no-build       # reuse the aibox-test:local image
docker run --rm -v "$PWD":/repo -w /repo aibox-test:local \
  bash -c 'cd /repo && bats tests/<file>.bats'        # a single suite
```

- **bash 3.2 / BSD-tool behaviour** is covered by CI's `macos-bash32` job, which runs
the whole fast suite under `/bin/bash 3.2` — the authoritative runtime gate for the
platform this project promises. Never "approximate" it on the host: reproduce a
Linux-side concern in the container and let CI cover 3.2.
- A CI-only failure is diagnosed from the job log (`gh run view --job <id> --log-failed`)
and reproduced in the container — do **not** re-run the failing test on the host.
- Verifying a real deploy host (its own `aibox`, explicitly requested by the operator)
is a *deployment task step*: it is never a substitute for the suite, and never run from
the development machine against `$AIBOX_HOME`.

### bash coding conventions (mandatory)

1. **`set -euo pipefail`** at the top of every script (for `src/aibox/*.sh`: once, in `00-head.sh`).
2. **Always quote variables as `${VAR}` (braces), never bare `$VAR`** — especially when a variable is immediately followed by a **non-ASCII** character (CJK text, full-width punctuation `，。、；：`). This is pitfall #1 below, the project's #1 footgun.
3. **Declare `local` before assigning** (or on two lines).
4. **Idempotent**: `install.sh` / `update.sh` must be safe to re-run without erroring.
5. **One implementation per helper**: anything both the manager and module hooks need
   lives in `tools/_shared/lib/*.sh` (injected into `bin/aibox` by the bundler);
   `scripts/check-sources.sh` fails on a duplicate definition. `module.yaml` is read
   through the shared readers (`meta_field` / `meta_map_value` / `meta_version`),
   never with a module-local `sed`/`awk`.
6. **No `jq` / `python`**: the registry is shell-sourceable; the main CLI sources it directly, bash 3.2 compatible. (Since v0.4.0 the registry is auto-discovered from `tools/*/module.yaml` via a zero-dependency awk subset parser; yq validates the subset in CI.)
7. **Keep full-width punctuation in Chinese copy** (README.zh.md), but always separate variable boundaries with `${VAR}`.

### Managed vs state files (mandatory)

Shipped files land in a deploy root through `install_managed_file` (never a bare
`cp`): a user-edited copy is KEPT and the new version is written as `<name>.new`.
`.env` and any user-owned path must be declared in `state_files:` — hooks never
overwrite them. Module README action/config tables are GENERATED by
`scripts/gen-docs.sh` between the `BEGIN GENERATED` markers (CI `--check` gate);
edit `module.yaml`, not the table. Spec §File ownership / §Generated README blocks.

### State model (mandatory)

State lives in three layers — **host** (`$AIBOX_HOME/config`, `installed.sh`,
`registry.cache`, `residue.conf`, `profiles/*.conf`), **deployment**
(`apps/<name>[-<profile>]/.env`, `/etc/<name>/<name>.conf`) and **contract**
(`base[-<profile>].env`, `redis[-<profile>]-<module>.env`, `upgrades/<m>.state`,
written by the provider). **No state file is ever sourced** — data is not code:
parse it (`cfg_kv_load` / `cfg_kv_get` / `cfg_kv_load_export` in the shared
library). `scripts/check-sources.sh` fails on a `. "…"` of a data file; spec
§State model.

### Content manifest (mandatory)

Any change under `tools/**` (hooks, `lib.sh`, `module.yaml`, compose files, `cli/**`,
`tools/_shared/**`) changes what hosts download — regenerate `modules.SHA256SUMS`
with `scripts/manifest.sh` in the same commit. CI runs `scripts/manifest.sh --check`
and fails on drift; `download_module` verifies every fetched file against it
(`AIBOX_VERIFY=0` is the explicit bypass).

### Pushing is not a formality (learned the hard way)

**Verify the remote before trusting a push.** A working copy was found with
`origin` re-pointed at `.` (the repository itself): `git push origin main` then reports
`Everything up-to-date`, exits 0, and pushes **nothing** — the commit and its release tag
never reach GitHub, and the release workflow never runs. Two commits were silently swallowed
that way.

- Before a release push: `git remote get-url origin` must print the GitHub URL (not `.`).
- Confirm what the remote ACTUALLY has: `git ls-remote origin refs/heads/main` (a local-looking
  answer — a bare SHA with no network — is a sign the "remote" is the working copy).
- When in doubt, push by URL: `git push https://github.com/lichengwu/aibox.git HEAD:main`
  (requires credentials; `gh auth setup-git` configures git for them).

### commit style

Conventional Commits: `fix:` / `feat:` / `docs:` / `style:` / `chore:`.

### Versioning & release flow

- Main CLI version: `AIBOX_VERSION` at the top of `bin/aibox`. **The main CLI and each module version independently** (a module's dispatched CLI carries its own `*_CLI_VERSION`).
- Module version: the `version:` field in `tools/<name>/module.yaml`.
- **A release = bump `AIBOX_VERSION` → push main.** GitHub Actions (`.github/workflows/release.yml`) reads `AIBOX_VERSION`; if the remote has no matching `v<version>` tag, it auto-creates the tag + release — the release notes carry the **curated `CHANGELOG.md` section for that version** (extracted by the workflow) plus the auto-generated commit list — and attaches a `SHA256SUMS` sidecar. Unchanged version → skip (idempotent, safe to re-push).
- **Changelog standard (every release, open-source bar — Keep a Changelog + semver):**
  1. Header: `## [x.y.z] — YYYY-MM-DD` (the `[x.y.z]` resolves via the compare-link definitions at the file bottom; add the new link when releasing).
  2. Body grouped by `### Added / Changed / Deprecated / Removed / Fixed / Security` — only the groups with content.
  3. Each entry: **what changed + why it matters to the user** (one or two sentences; link the commit/PR when the entry is a fix with a story, e.g. "live-caught on the deploy host: …"). No bare commit-subject dumps.
  4. A **semver rationale** line in the release commit message (why major/minor/patch).
  5. **Breaking changes / migration notes** get their own `### Changed` bullet starting with **BREAKING:** — plus the two-gate interaction model applies to any new destructive verb.
  6. Module version bumps ride in the release commit message (they are independent of `AIBOX_VERSION`).
  7. The `[Unreleased]` section accumulates during development; at release time it is renamed to the version header (date = release day) and a fresh empty `[Unreleased]` is NOT required (start it with the next change).
- Self-update still runs `curl install.sh | bash` (idempotent re-install), **independent of releases** — releases are publication/change records. Optional `AIBOX_SHA256`/`AIBOX_VERIFY=1` add checksum verification.

## Pitfall log

> The Chinese characters that appear in these examples are deliberate — they are the trigger for the bugs. Do not "fix" them by translating the example strings.

### #1 bash 3.2 + UTF-8 locale: full-width punctuation right after `$VAR` → unbound variable

**Symptom**: under `set -u`, a line like `log "当前 $name，继续"` reports `name<replacement-char>: unbound variable` (the variable name is followed by garbled bytes) and exits. Only triggers under a UTF-8 locale; the `C` locale doesn't trigger it, so it doesn't reproduce in some environments / CI.

**Root cause**: macOS's bash 3.2.57 has a multibyte variable-name parsing defect. When `$VAR` is immediately followed by a UTF-8 multibyte character (CJK text, full-width punctuation `，。、；：！？` etc.), bash erroneously absorbs its bytes into the variable name, producing a non-existent variable, which `set -u` flags as unbound. The `C` locale processes bytes one at a time; a leading byte like `0xEF` isn't a name character, so it terminates the variable name and no error occurs.

**Fix**: always delimit variable references with `${VAR}` — `当前 ${name}，继续` is fine.

**History**: `ad7cfed` swept the repo with perl and fixed 9 occurrences, but `876206d`'s new `cmd_self_update` missed `$before，`; fixed in 2026-09 by `e5311bf`.

**Detection**:

```bash
sed -n <line>p <file> | xxd   # see if full-width bytes like ef bc 8c follow the variable name
LC_ALL=zh_CN.UTF-8 bash <script>   # reproduce under a UTF-8 locale (the C locale won't reproduce it)
```

**One-liner self-check** (the regex requires a non-ASCII char immediately adjacent to the variable name to match):

```bash
grep -nE '\$[A-Za-z_][A-Za-z0-9_]*[，。、；：！？（）「」]' <file>
```

### #2 Don't use bash 4-only syntax in scripts dispatched to other machines

**Symptom**: a module's `svc.sh` only forwards to the dispatched script, but on macOS even `--version` fails with `syntax error near unexpected token '>'` — with no hint that it's about the local bash version.

**Root cause**: macOS ships bash 3.2, but the script used bash 4's `exec {fd}>file` (auto-allocated file descriptor). For THAT construct it was a **parse-time** error — the whole file can't execute, no branch is reachable.

**Correction (measured 2026-09, while building the coverage probe)**: `exec {fd}>file` is the EXCEPTION, not the rule — most bash-4 constructs (`declare -A`, `mapfile`, `${var,,}`) **parse fine under `bash -n` on 3.2** and only fail at RUNTIME. So:

- `bash -n` under 3.2 catches only parse-level breakage (a minority)
- **actually EXECUTING the code under 3.2 is the only complete detector** — CI's `macos-bash32` job runs the whole fast suite under /bin/bash 3.2 for exactly this reason (and ubuntu's `bash -n` accepts bash-4 syntax, so it proves nothing here)

**Fix**: use a fixed fd (`exec 9>file` / `flock -n 9`). If you genuinely need bash 4+, ensure the script is only parsed on the target machine — don't let it appear in a path that gets sourced/executed on macOS.

**Why "parseable" is worth the concession**: staying parseable lets you give a clear message on unsupported platforms ("this command requires a Linux deploy host") instead of throwing a syntax error. See `require_deploy_host` in `tools/openmaic/cli/openmaic`.

### #9 `exec` with a redirection makes it PERMANENT shell state — `exec 9>>f 2>/dev/null` eats stderr forever

**Symptom**: under the coverage probe, every `die`/`warn` message vanished — tests asserting stderr failed with EMPTY output, no error anywhere.

**Root cause**: `exec` without a command name applies its redirections to the SHELL itself, permanently. `exec 9>>$file 2>/dev/null` was meant to silence just the open-failure error — but fd 2 got pointed at /dev/null for the REST OF THE PROCESS. Every later `>&2` write (die/warn) went to the void, silently.

**Fix**: never hang a stray `2>/dev/null` onto an fd-opening `exec`. Guard the failure with `if exec 9>>"$file"; then ... fi` (a failing exec inside an `if` condition is safe under `set -e`) — one real error line to stderr on a bad path is the correct degradation, not silence.

**Detection**: any stderr assertion failing with empty output while the command exits nonzero; `grep -n 'exec [0-9]*>>.*2>'` over the codebase.

### #3 Proxy testing: status-code-only is always a false positive

**Symptom**: after adding proxy support, `proxy test` (now `proxy check <url>`) reports "proxy ok" (HTTP 200) — but with the port wrong or the proxy off, the test **still says 200**.

**Root cause**: the test was written as "reach GitHub via the proxy = proxy works". But on a directly-reachable network, that 200 came from the **direct connection** — the proxy may not have been used at all. The same machine, with a different network (Clash TUN on/off), flips the verdict without a line of test code changing.

**Fix**: judge by curl's `%{proxy_used}` (`1` = actually went through the proxy, `0` = direct), not the HTTP status:

```bash
curl -s -x "$url" -o /dev/null -w '%{http_code} %{proxy_used}' "$target"
```

And **also run a direct-connection control** (`curl --noproxy '*'`), telling the user whether the proxy is required on the current network. The criterion itself goes stale, so re-measure every time.

**Case-sensitivity** (measured, curl 8.7.1 / macOS):

| Variable | curl honors it? |
| --- | --- |
| `http_proxy` (lowercase) | yes |
| `HTTP_PROXY` (uppercase) | **ignored** |
| `https_proxy` / `HTTPS_PROXY` | both honored |

So export **both cases**, or one class of tools won't pick it up. Also: `file://` sources aren't affected by `http_proxy` (curl reads the file directly), so local/intranet sources need no special-case — but intranet HTTP sources get proxied out; the default `no_proxy` covers private ranges to handle that.

### #4 Env vars don't cross the "process boundary"; cross-machine/cross-time needs persistence to disk

**Symptom**: configured a proxy for `aibox`; `aibox install openmaic` used it; but later `openmaic upgrade` on the deploy host pulling source **can't connect again**.

**Root cause**: the proxy is injected via env vars, but `openmaic upgrade` is a process on **another machine, at another time** — by then `aibox` has exited and the env vars can't be inherited.

**Fix**: for "the module networks elsewhere on its own" scenarios, the module must **write the value into its own config file** (`tools/openmaic/lib.sh`'s `sync_proxy_to_conf` writes `/etc/openmaic/openmaic.conf`).

**Rule of thumb**: network requests run **in the hook right now** → env vars suffice; requests run by a **dispatched command elsewhere, later** → must persist to disk. Don't expect `export` to cross processes and hosts.

### #5 Three counter-intuitive curl semantics (all silently corrupt the verdict)

Three traps hit while writing `aibox proxy check`, all sharing "looks fine, conclusion is wrong":

1. **On connection failure `-w` emits nothing.** Pointed at a port nobody listens on, `curl -w '%{http_code}'` doesn't emit `000` — it prints nothing, so `code` becomes an empty string. Judging `code == 000` misses all failures. Must `|| out=""` as an empty fallback, then fill in `000` yourself.
2. **An explicitly `-x`'d proxy is still excluded by `no_proxy`.** If the check script doesn't clear `no_proxy` and the network happens to have an exclusion rule, "can this proxy reach X" **silently becomes "can a direct connection reach X"** — a false verdict with no error. Fix: `env -u no_proxy -u NO_PROXY curl -x ...`.
3. **`%{proxy_used}` can't be judged by "is it empty".** The fallback failure result has `used=0` (non-empty), so "all proxies dead" aggregates into "all traffic confirmed via proxy". Must strictly judge `= 1`.

**Lesson**: for proxy-connectivity checks, **"failed" and "not measured" must be distinguished**. When inferring state from empty/non-empty, first ask "what would this field be on the failure path".

### #6 Multibyte characters can't be sliced as substrings (bash 3.2 + C locale)

**Symptom**: spinner frames stored as `spin='⠋⠙⠹⠸'` with `"${spin:$i:1}"` to take one frame; works under a UTF-8 locale, garbled under a `C` locale.

**Root cause**: bash's substring expansion under a non-UTF-8 locale counts **bytes** not characters, slicing the 3-byte `⠋` into halves and emitting invalid UTF-8.

**Fix**: store **complete** frames in an array, take elements instead of slicing:

```bash
SPIN=( '⠋' '⠙' '⠹' )          # each item is a complete character; can't be sliced wrong
probe_render "${SPIN[$i]}"
```

Likewise, symbols like `✔ ✘ ⠋` must always be passed as **complete string literals** in code — don't concatenate or slice them.

### #7 Don't use `script` to test TTY interaction; use `expect`

`script -q /dev/null cmd` looks like it can feed piped input into a pty to test `read` prompts, but it's unreliable: stdin forwarding and pty line-buffering desync, so `read` doesn't get the line you expect (manifesting as "fed y but treated as N"), or the child gets killed early at EOF. macOS ships `expect` (`/usr/bin/expect`); use it directly:

```tcl
spawn bash bin/aibox proxy set http://127.0.0.1:1
expect -re {\[y/N\]} { send "y\r" }
```

Note that in Tcl regex the `[y/N]` brackets are a character class; write `\[y/N\]`.

**Verification**: pty output contains ANSI control sequences; use `cat -v` to see the real content; counting `^[[<n>A` (cursor-up) **and checking the number equals the block height** is the fastest way to catch a redraw-line-count bug.

### #8 bash 3.2: `local a="x" b="${a}/y"` same-line self-reference → unbound under `set -u`

When `local` declares multiple variables, bash 3.2 **expands all right-hand sides before any assignment** — the second variable's `${a}` expands while the first isn't assigned yet, and with `set -u` it blows up:

```bash
/bin/bash -c 'set -u; f(){ local a="/etc/w" b="${a}/windmill.conf"; echo "$b"; }; f'
# bash: a: unbound variable
```

bash 5 doesn't have this, so "ran fine locally on the dev box" scripts blow up on macOS bash 3.2.
(The windmill module's lib.sh actually hit this; `bash -n` can't catch it — pure runtime-expansion issue.)
**Fix**: split into two lines: `local a="..."` / `local b="${a}/..."`.
**Detection**: `grep -rnE 'local [a-z_]+="[^"]*"[ ]+[a-z_]+="\$\{[a-z_]+\}'`.

## Developing a new module (spec + tooling)

**Spec**: [`docs/module-spec.md`](docs/module-spec.md) §Onboarding a new module — the normative checklist (definition of done).

**Tooling** (repo-local, zero-dependency, same rules CI enforces):

```bash
scripts/new-module.sh <name> [--desc "..."] [--no-compose] [--out <dir>]   # scaffold a conformant skeleton (passes the validator out of the box)
scripts/validate-module.sh <name> | --all                                  # conformance gate: 0 ERRORs required (WARNs tolerated)
```

**Flow**: scaffold → fill `module.yaml` (ports/checks/deps/services/usage/includes) → implement the hooks → `scripts/validate-module.sh <name>` until PASS → `tests/docker/run.sh --fast-only` (never a local `bats` run — see §Test environment) → live smoke (install / start / status / logs / stop / uninstall on a docker host) → add the module row to README(.zh). **`tools/gitlab/` is the reference implementation onboarded with exactly this flow.**

**CLI conventions** (validator-enforced; the instruction-system contract, spec §CLI surface / §Exit codes): every manager verb answers `--help` (identical to `aibox help <verb>`); every service module declares `doctor` (the shared `module_doctor`: deps + docker daemon + `status_info` state + port listeners); dispatch-CLI modules alias `start`/`stop`/`restart` onto their CLI's own verbs; usage errors in hooks go through `usage_die` (exit `2`), never `die`; exit codes are stable for automation — `1` runtime, `2` usage/declined, `3` dependency missing, `4` precheck failed, `10` upgrade rolled back, `20` manual intervention (hooks add `30` not ready, `40` lock conflict, `50` cancelled).

**Help system**: every module carries a `usage:` map in module.yaml (one line per declared action, args hint first: `"<node-name> — switch the active node"`); `aibox <module> --help` renders the action table, `aibox <module> <action> --help` renders the single action — both local-first. The validator WARNs on actions without usage entries; the scaffolder emits a TODO skeleton. Unknown actions die with `run: aibox <module> --help` (never a hand-maintained action list).

**Iron rules** (validator + CI enforce; details in the spec):

1. `module.yaml` is the source of truth — the registry auto-discovers `tools/*/module.yaml` (no registry.sh to edit; local `file://` source needs zero global changes). Hyphenated names map to underscored variable keys internally (`pi-web` → `AIBOX_MODULE_pi_web_*`); hyphenated usage keys normalize the same way. `module.yaml` ships in the cache alongside the 5 hooks (standard-6 download set), so per-module help/metadata renders offline.
2. Hooks (`install/uninstall/update/svc.sh`): bash shebang, `set -euo pipefail`, idempotent; shared code lives in `lib.sh` (sourced library — no shebang, no strict-mode line) and infra helpers in the shared include (`tools/_shared/common.sh` — output helpers `log/warn/ok/info/die` + the docker.io pool; declared via `includes: [common]`, cached per-module as `_common.sh`; lib.sh resolves cache layout first, repo layout `../_shared/` for direct exec). NEVER copy pool/helper code into a new module — source the include (spec §Shared library includes). Module scripts are downloaded and cached to `~/.aibox/modules/<name>/`.
3. **Every module declares `checks:` (preflight contract)**: install/update is hard-gated by `preflight_module` — deps (strict), `checks.commands`, `checks.domains` (host-probed) — `checks.disk_gb` is INFORMATIONAL (a tight disk warns, it never blocks install/update) / `checks.docker_pull` (daemon-probed — the docker daemon's egress differs from the host's; never host-probe a daemon-consumed registry), with the `checks.docker_images` cache short-circuit, and `services:` readiness (recursive into base). Host network failures try the configured alternative routes (direct/clash/mirror/static proxy) and adopt a working one for the run. CI enforces the section's presence + field formats; `aibox check self` / `aibox check <module>` run it proactively; `--skip-checks` bypasses. Full spec + per-module matrix: `docs/module-spec.md` §Preflight checks.
4. **Install paths must be overridable**: start from `${AIBOX_BIN_DIR:-$HOME/.local/bin}` and expose a module-specific override (e.g. `OPENMAIC_BIN_DIR`) — deploy hosts often want `/usr/local/bin`.
5. **`svc.sh` is an "action entry point", NOT "must be a daemon"**: long-lived services (pi-web) use `start/stop/restart`; pure-CLI dispatch (openmaic) can pass actions straight through to the dispatched command. Service-type modules (`actions` contains `start`) MUST implement the full lifecycle `start/stop/restart/status/logs`.
6. **Platform differences: warn, don't hard-block**: install is usually cross-platform (just copying files); the real limit is reported by the script at execution time, which is less false-positive-prone than blocking at install.
7. The local machine is macOS (bash 3.2); hooks must be compatible. Scripts **dispatched to other platforms** must avoid bash 4 syntax — see pitfall #2. (The validator runs a bash-3.2 parse check when available.)
8. **Deploy-type modules: don't invent your own paths** (see [`docs/module-spec.md`](docs/module-spec.md) "Deploy directory & config path conventions"): deploy root is uniformly `$AIBOX_HOME/apps/<name>` (same expression on both platforms, no branch); config is `/etc/<name>/<name>.conf`. Two empirical hard constraints: Docker Desktop on macOS **doesn't share `/opt`** by default (putting things there makes compose relative mounts fail with `Mounts denied`); **a systemd system service has no `HOME`** (paths derived from `$HOME` resolve to empty in the service context; units must use explicit `Environment=`).
9. **Self-starting services use the platform-native init system**: long-lived services (pi-web, windmill's scheduled task, clash) auto-start/daemonize per-platform — don't mix:

    - **macOS → launchd**: user-level `~/Library/LaunchAgents/<label>.plist`, `launchctl bootstrap gui/$(id -u)`, no root needed. `KeepAlive` = auto-restart, `StartCalendarInterval` = scheduled.
    - **Linux → systemd**: user-level `~/.config/systemd/user/<name>.{service,timer}`, `systemctl --user` + `loginctl enable-linger` to survive logout, no root needed. `Restart=always` = auto-restart, `OnCalendar` = scheduled.
    - One service, one unit per platform, generated by a `case "$(uname -s)"` branch. System-level services (need root / start at boot) use `/etc/systemd/system` + `systemctl` (no `--user`) — see windmill's `cmd_systemd`/`cmd_launchd` and pi-web's `write_plist`/`write_service`.
10. **Docs are part of the contract**: `README.md` (commands / ports / env overrides / preflight — validator: ERROR if missing) and `docs/DEVELOPMENT.md` (upstream links, version-pin policy, design decisions, known quirks — validator: WARN if missing). **Never hardcode derived values** (profile-derived ports, container names, env-file paths, credentials): state the default-profile default if useful, label it as such, and point at `aibox status <module>` / `aibox <module> config list` for the authoritative values (spec §Doc hygiene; the validator WARNs when a profile-deriving module documents numeric ports with no status pointer).
11. **Consuming the shared `base`? Follow the dependency contract** (spec §Dependency contract): link through the shared helpers (`base_env_file`/`ensure_shared_base`/`ensure_shared_db`/`ensure_shared_redis_db`) — NEVER hardcode `base.env` (it points at the DEFAULT profile and silently attaches a named-profile module to the wrong instance), end `deploy_root()` with `$(profile_suffix)`, declare the resource you consume (`base:redis#<module>` — a bare `base:redis` allocates no logical DB), and keep the `base.env` contract ADDITIVE (bump `AIBOX_BASE_ENV_VERSION` on both sides for any rename/removal). The validator enforces all four.
12. **No hardcoded credentials** in compose files; shared-PG consumers read `${AIBOX_POSTGRES_*}` from the injected `base.env` (validator scans; base is the only exception by design).
13. **Residue is DECLARED by the module**: a `residue:` stanza in `module.yaml`
    (`paths`/`containers`/`volumes`/`networks`/`images`/`units`/`bin`/`npm`/`process`) — captured into
    `$AIBOX_HOME/residue.conf` at install time, so `aibox autoclean` can clean leftovers even
    after the module (or aibox itself) is uninstalled and offline (rescue: curl the
    single-file `bin/aibox` to /tmp and run `autoclean --apply`). A module whose residue is
    dynamic overrides `residue_paths()` in `lib.sh` (base does, for its per-profile env
    files). The manager carries NO per-module residue map — validator WARNs when a module
    declares neither; spec: `docs/module-spec.md` §Residue cleanup.

15. **One config, read at runtime**: credentials/endpoints live in the provider contract
    (`$AIBOX_HOME/base[-<profile>].env`) only. Never bake a literal secret, never default
    one silently (`${AIBOX_POSTGRES_PASSWORD:-aibox}` = ERROR), and derive URLs with
    `base_pg_url`/`base_redis_url` at call time. The manager exports the contract before
    dispatching every hook/CLI, so children inherit the CURRENT values. Spec §One config,
    read at runtime; validator S13k.

14. **Host ports MUST live in the aibox reserved band** (spec §Port allocation):
    **30000–32999** — existing services **30000–30999**, new services **31000–31999**,
    infrastructure **32000–32999**. Never ship a default on a
    privileged port (<1024), a common-service port (3000/5000/8080/8443/8888/9000/9090/
    7890/8000…), or inside Linux's ephemeral range (**32768–60999** — the kernel gives
    those to outbound connections first). Container-internal ports are exempt. A genuinely
    public port is declared explicitly with `:public` (e.g. `443/tcp:https:public`), and
    every host port a module publishes must appear in `module.yaml` `ports:`. Enforced by
    validator **S13j**; `port_policy_hint` explains a violation at runtime (doctor).

### #12 `while IFS= read -r a b c` reads the WHOLE line into the first variable

**Symptom**: a JSON emitter iterated `"name profile version"` lines and every module came out
as one giant name (`${AIBOX_MODULE_new_api base unknown_ports:-}: bad substitution`).

**Root cause**: `IFS=` (empty) disables word splitting for that `read` — a deliberate idiom
when you want the raw line (`while IFS= read -r line`), but a trap when you then expect the
extra variables to be filled: only the first one gets the entire line.

**Fix**: use `while read -r a b c` (default IFS) when you want fields, `while IFS= read -r line`
only for raw lines. Same family as the `${VAR}` brace rule: the idiom is right in one context
and silently wrong in the other.

**Detection**: a loop variable that "contains spaces you did not expect"; grep for
`while IFS= read -r [a-z]* [a-z]` (more than one variable after `read`).

### #10 macOS bash 3.2 (arm64-darwin26 build): a failing `[[ ]]` does NOT trigger `set -e` — bats mid-test assertions are silently swallowed locally

**Symptom**: tests "pass" on the dev Mac while the identical assertions fail on CI (ubuntu). Observed twice in one day: a stale header assertion ("Available modules", removed by an earlier TUI change) and a spacing regression — both green locally, red on CI.

**Root cause** (measured, minimal repro): on this machine's `/bin/bash 3.2.57 (arm64-apple-darwin26)`:

```bash
bash -e -c '[[ a == b ]]; echo SURVIVED'   # → SURVIVED, exit 0 (!!)
bash -e -c '[ a = b ];  echo SURVIVED'     # → exits 1 (correct)
bash -e -c '[[ a == b ]] || false; echo X' # → exits 1 (correct — the workaround)
```

`[[ ]]` is a keyword, and this build fails to raise errexit for it; `[ ]` (a builtin command) works. bats runs test bodies under `set -e` and treats the test's EXIT STATUS as the verdict — so a failing `[[ ]]` in the MIDDLE of a test is ignored locally; only the last command's status decides. **Every local "green" only validates the last line of each test.**

**Consequences**:

- CI (ubuntu bash 5, correct errexit) is the AUTHORITATIVE assertion gate — never trust local-only green for assertion changes.
- A test whose final line passes can hide any number of broken assertions above it.

**Mitigations**:

- For assertions that matter mid-test: `[[ ... ]] || false` (restores errexit semantics on the quirk build), or make the critical assertion the test's LAST line.
- Verified workaround list: `|| false`, `[ ]` form, or running the suite under a correct bash (CI / `docker run bash:5`).

**Detection**: a test that should obviously fail passes locally — re-run it on ubuntu (`docker run --rm -v "$PWD":/repo -w /repo ubuntu:24.04` + `apt-get install bats`) before trusting it.

### #13 bash TUIs: save/restore the EXACT terminal state, and prefer letter keys over Enter

**Symptom** (three separate ones, all hit while building `aibox dashboard`):
1. a frame's first column lost its first character — `▸●itlab` instead of `▸● gitlab`;
2. `a` and `d` opened the HELP overlay instead of about/detail;
3. a headless test (keys injected through `AIBOX_DASH_KEYS`) still went down the
   non-interactive path and printed one frame.

**Root causes**: (1) a `cut -c2-` used to strip an alignment space struck the padded
*name* column, not the prefix — column math must never post-process a rendered row;
(2) in a `case` pattern, `?` is a GLOB (any single char), so `\? | h)`'s unescaped
sibling swallowed every later one-character key — escape it (`\?`); (3) the degrade
rule keyed on `TERM=dumb`, which is exactly what CI containers report, so the
injected-key path must bypass terminal-capability rules entirely.

**Rules for any TUI here**: save `stty -g` and restore exactly (never `stty sane`);
enter the alternate screen (`tput smcup`) and leave it on every exit path through an
**idempotent** `trap … EXIT INT TERM HUP`; `read -n 1` returns "" BOTH on timeout and
when Enter is pressed (bash 3.2 has no `read -N`) — so bind actions to LETTER keys
(`d` for detail) and keep `AIBOX_DASH_*` seams injectable for tests; drive pty tests
with `expect` (pitfall #7), not `script`.

**Detection**: a rendered row whose first character is missing → grep the row
pipeline for `cut`; a single-character key that opens the wrong overlay → grep the
`case` arms for an unescaped `?`.

### #11 Test glob patterns: `*"X"` anchors at END-OF-STRING — a miss is pattern semantics, not a bash bug (a misdiagnosis post-mortem)

**Symptom**: an assertion like `[[ "$output" == *"pi-web"*"· ✓" ]]` fails while the output plainly contains `pi-web · ✓ running` — and it's tempting to blame the macOS bash 3.2 build.

**Root cause (verified — NOT a bash bug)**: `*"X"` requires X at the END of the
string (suffix anchor); `*"X"*` is containment. `· ✓` mid-string followed by
`running` correctly fails a suffix pattern. Measured on `/bin/bash 3.2.57
arm64-apple-darwin26` AND bash 5.3: `[[ "m 1 · ✓" == *"· ✓" ]]` matches —
multibyte tails are fine. (This entry replaces an earlier "measured bash
3.2 multibyte glob quirk" claim that was a misdiagnosis of exactly this
suffix-vs-containment mistake — caught by whole-branch review with a
cross-version repro.)

**Fix**: for containment use `*"X"*` (trailing `*`); for a real suffix test
keep `*"X"` and put X last in the string. Multibyte characters (✓ ⚠ ○ · — …)
are safe in either form.

**Detection**: when an assert contradicts visible output, test the pattern
against a string that ENDS with the literal before suspecting the shell:

```bash
[[ "x · ✓" == *"· ✓" ]]   # matches → the shell is fine; check your pattern
```

**Interaction with #10**: the suffix-pattern miss was invisible for months
because pitfall #10 swallows failing mid-test `[[ ]]` — the `|| false`
discipline is what surfaced it.

### #14 launchctl EIO-5 lies: "Bootstrap failed: 5: Input/output error" means "label already loaded" (or "no gui domain") — and a swallowed bootout is what sets it up

**Symptom**: `aibox pi-web restart` (any launchd module hook) prints
`Bootstrap failed: 5: Input/output error` + `Try re-running the command as root
for richer errors.` — while the service itself is (and stays) running.

**Root cause** (measured 2026-10-29, macOS 27.0.1, live-caught on the deploy
host): launchd internally rejects the bootstrap with errno 37 "Operation
already in progress" — the label is already bootstrapped in `gui/$UID` — but
launchctl's CLI prints it as "5: Input/output error" with a wrong "re-run as
root" hint. The same EIO-5 shows when the `gui/$UID` domain does not exist at
all (an SSH session while the user has no console login). The setup for the
error: the restart flow's `launchctl bootout … 2>/dev/null || true` — a failed
bootout (label stays loaded) was silent, so the bootstrap ran into the loaded
label. Bonus trap, measured with a self-bootout test job: launchd does NOT
refuse a bootout requested from the job's own descendant process — it executes
it and kills the requester, so a restart driven from inside the service's own
web terminal simply dies mid-command.

**Fix**: never swallow bootout stderr. Distinguish "not loaded" (message
"Boot-out failed: 3: No such process" / "Could not find service" — the MESSAGE,
not the errno, is the reliable signal) from real failure; warn on the latter;
translate EIO-5 into its two real meanings for the human. Implementation:
`tools/pi-web/svc.sh` (`_launchd_bootout` / `_launchd_hint`).

**Detection**: `launchctl print gui/$(id -u)/<label>` shows the live truth
(state/pid/runs); the unified log carries launchd's honest errno:
`log show --last 1h --predicate 'process == "launchctl" OR (process == "launchd" AND eventMessage CONTAINS "bootstrap")'`
— look for "failed (37: Operation already in progress)".

## License

MIT.