# aibox Module System Enhancement Spec > **⚠ SUPERSEDED — read [`docs/module-spec.md`](module-spec.md) instead.** That file is the > normative, current contract (hook contract, preflight, residue, usage:, includes:, exit codes, > interactive confirmation). This document was the ORIGINAL DESIGN DRAFT that drove the > module.yaml migration; it is kept as design history: it still describes registry.sh-era > mechanics in places and its §2.3 field reference has drifted from the implemented schema > (e.g. it does not know about `usage:` / `includes:` / the `upgrade:` stanza shape). > When this file and module-spec.md disagree, module-spec.md wins. > Status (historical): Design draft. This document collected enhancement requirements including port declaration, Dashboard, shared base components, module development guide, and the module.yaml declaration spec, as the basis for subsequent implementation. > > **Incremental revision (2026-09-16)**: module.yaml field de-duplication (implicit standard files set / common lifecycle contract for actions / slimmed-down dashboard) + shared-component dependency declaration and connection-info propagation (provider model + base.env + compose `--env-file` live injection, no generator). See [`docs/design/module-yaml-refactor-design.md`](design/module-yaml-refactor-design.md). §1 / §2.3 / §4 / §5 / Appendix B/C have been annotated accordingly. > > Positioning: aibox is a lightweight module manager for local development + single-machine deployment and ops (pure bash, zero runtime dependencies, compatible with macOS bash 3.2). The terminal-state design as modules grow from the current 4 to dozens. --- ## 1. Background and Goals Historically, aibox module declarations were centralized in `registry.sh` (shell-sourced variables), suitable for 4-5 modules; this has been migrated to a per-module `tools//module.yaml` (autonomous, discovered by scanning `tools/*/module.yaml`, registry.sh has been deleted, see §9). As modules grow to dozens, we still need: no port conflicts, visible module status, shared components to save resources, and a development guide to help AI upgrades. This spec defines: 1. **module.yaml spec** —— one YAML declaration file per module, structured and autonomous 2. **Port declaration and conflict detection** —— statically declared at design time, CI gates conflicts 3. **Dashboard TUI** —— module status + endpoint + credential visualization 4. **Shared base components** —— single PG/Redis instance with multiple databases, saving resources 5. **Module development guide** —— AI upgrades modules based on this 6. Dependency management, auto-start conventions (already implemented, documenting current state) --- ## 2. module.yaml Spec (Module Declaration) ### 2.1 Location and Discovery - **Location**: `tools//module.yaml` (same directory as the module code, autonomous) - **Discovery**: the main CLI scans `tools/*/module.yaml` to auto-discover modules (**no longer needs registry.sh to list module names**). Adding a module = create `tools//` + `module.yaml`, zero changes to global code. ### 2.2 YAML Subset (awk-parseable) To keep the main CLI zero-dependency (pure bash + awk), module.yaml is restricted to a subset that awk can reliably parse: - `key: value` (scalar, quotes optional) - `- item` under `key:` (list) - `subkey: value` under `key:` (shallow nesting, at most 2 levels) - `#` comments - **Forbidden**: anchors/aliases (`&`/`*`), multi-line strings (`|`/`>`), flow style (`{}`/`[]`), complex types This subset covers all module declaration fields. ### 2.3 Field Spec ```yaml # tools//module.yaml —— module declaration source of truth name: pi-web # Required. Module name (hyphenated) version: 1.0.0 # Required. Module version description: "..." # Required. One-line description platform: "" # Optional. Empty = cross-platform; darwin = macOS-specific dir: tools/pi-web # Required. Directory within the repo deps: # Optional. Runtime dependencies (command@platform:version) - "node:22" # node:22 = major version >=22 - npm # @linux = Linux-only check - "docker@linux" # empty = no dependencies (e.g. clash, mihomo download) ports: # Optional. Ports used (port/protocol:purpose) —— CI detects conflicts - 30141/tcp:http - 9090/tcp:api files: # Optional. List only "extra" files; the standard 5 files (lib.sh/install.sh/uninstall.sh/update.sh/svc.sh) are implicit by default (awk union, old yaml listing all still works) - lib.sh - install.sh - uninstall.sh - update.sh - svc.sh hooks: # Required. Hook file names install: install.sh uninstall: uninstall.sh update: update.sh svc: svc.sh actions: # Optional. Actions supported by svc (list all); the lifecycle subset of service-type modules (start/stop/restart/status/logs) must satisfy the "Common Action Contract", see §4 and design doc §4 - start - stop - restart - status - logs - diagnose upstream: # Optional. Development guide links (see §6) homepage: https://github.com/agegr/pi-web docs: https://github.com/agegr/pi-web#readme install: https://github.com/agegr/pi-web#installation test: https://github.com/agegr/pi-web#development dashboard: # Optional. Only a static fallback when not installed; only endpoints[] + hint subfields allowed; at runtime always goes through lib.sh's dashboard_info() (see §4) endpoints: - "http://127.0.0.1:${PORT}" hint: "Username pi / password: see aibox pi-web status" ``` ### 2.3.1 Incremental Fields (2026-09-16, see [design doc](design/module-yaml-refactor-design.md)) | Field | Required | Description | | --- | --- | --- | | `files` | No | List only **extra** files; the standard 5 files are implicit (awk union, old yaml listing all still works). CI verifies that listed items exist. | | `actions` | No | List all actions. **Containing `start` means service-type** and must satisfy the lifecycle contract (start/stop/restart/status/logs); without `start` means distribution-type, not enforced. The CLI lists actions as help when running `aibox ` (no action), and does not validate on every invocation. | | `services` | No | Shared-component dependencies, compact string `provider:component[#dbname]`. component is the **full name** (`postgres`/`redis`, `pg` forbidden). `postgres` should provide `#dbname`; `redis` is usually omitted. Declaration drives: CI validation, `install` auto-runs `base create postgres`, compose scheduling auto-injects `--env-file base.env`. Multiple DBs: `base:postgres#windmill_jobs`. | | `provides` | No | Only filled by provider modules (e.g. base). Declares the full names of components exposed to others. A consuming module's `services` reference must match a provider's `provides`. | | `dashboard` | No | Only `endpoints[]` + `hint`, as static fallback when not installed; at runtime goes through `dashboard_info()`. | **Full-name consistency**: component names always use the full name —— module.yaml `services`/`provides`, env variables (`AIBOX_POSTGRES_*`), compose service/container (`postgres:` / `aibox-base-postgres`), and port purpose labels (`35432/tcp:postgres`). ### 2.4 Parsing Approach (Approach A: built-in awk, zero dependency) The main CLI's `load_registry` is changed to scan `tools/*/module.yaml` + parse with awk: ```bash load_registry() { AIBOX_MODULES="" local f name for f in "$AIBOX_REPO_DIR"/tools/*/module.yaml; do [ -f "$f" ] || continue name="$(parse_yaml_field "$f" name)" [ -n "$name" ] || continue AIBOX_MODULES="${AIBOX_MODULES:+$AIBOX_MODULES }$name" parse_yaml_module "$f" "$name" # awk parse → eval-injected AIBOX_MODULE__* variables done } ``` `parse_yaml_module` (awk implementation, ~80 lines): - Iterates YAML lines, recognizing hierarchy by indentation + `-` - Scalar → `AIBOX_MODULE__="value"` - List → `AIBOX_MODULE__="item1 item2 ..."` (space-separated, compatible with module_field reads) - Shallow nesting (hooks/upstream/dashboard) → `AIBOX_MODULE___="value"` - Outputs `eval`-able variable assignments; the main CLI eval-injects them The awk parser only supports the §2.2 subset; CI guarantees YAML compliance (see §2.5). ### 2.5 CI Validation (module-lint) ```yaml module-lint: name: module.yaml lint runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: install yq run: sudo wget -qO/usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 && sudo chmod +x /usr/local/bin/yq - name: validate module.yaml run: | set -e fail=0 for f in tools/*/module.yaml; do [ -f "$f" ] || continue # 1. Required fields: name/version/dir/hooks.install non-empty (files is now optional, only lists extras) # 2. YAML subset compliance: no anchors (&/*), no multi-line (|/>), no flow ({}/[]) # 3. ports format: port/protocol:purpose # 4. files listed items must exist under tools// # 5. Modules containing start must contain the lifecycle set and svc.sh implements it (without start, not enforced) # 6. dashboard only allows endpoints + hint subfields # 7. services/provides component names are full names (whitelist postgres/redis/...) yq -e '.name' "$f" >/dev/null || { echo "::error file=$f::missing name"; fail=1; } ... done exit $fail ``` CI uses yq (authoritative parser) to validate; the main CLI uses awk (zero dependency) for runtime parsing. CI guarantees the YAML is within the awk subset. --- ## 3. Port Declaration and Conflict Detection ### 3.1 Goal Modules statically declare used ports at design time; aibox registers them so that multiple modules on a single machine do not conflict. **This is not a runtime error; CI gates submissions with tests.** ### 3.2 Declaration (module.yaml ports field) ```yaml ports: - 30141/tcp:http # port/protocol:purpose - 9090/tcp:api ``` Format: `port/protocol:purpose`. CI detects uniqueness by **port + protocol**. ### 3.3 Allocation Strategy: Static Declaration + First-Declared-First-Served (No Randomization) - **Static declaration**: ports are hard-coded in module.yaml (fixed at design time), stable and predictable - **First-declared-first-served**: whoever declares first wins; a new module declaring an already-occupied port → CI fail, and the developer changes the new module's port - **No randomization**: random ports suit multi-tenant/container scenarios, not local development (endpoints unstable, users can't remember them, dashboard must query dynamically) - **Port range**: aibox modules use 30000-49999 (avoiding common system ports <1024, 80, 443, 3000, 5432, 6379, 8080, 9000, etc.) ### 3.4 CI Conflict Detection (port-conflict job) ```yaml port-conflict: name: port conflict runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: install yq run: sudo wget -qO/usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 && sudo chmod +x /usr/local/bin/yq - name: detect cross-module port conflicts run: | set -e declare -A owner dup=0 for f in tools/*/module.yaml; do [ -f "$f" ] || continue m=$(yq '.name' "$f") for p in $(yq -o=ts '.ports[]' "$f" 2>/dev/null || true); do pp="${p%%/*}" if [ -n "${owner[$pp]:-}" ]; then echo "::error::port $pp conflict: ${owner[$pp]} and $m" dup=1 else owner[$pp]="$m"; fi done done [ $dup -eq 0 ] && echo "✅ no port conflicts" exit $dup ``` Conflict → `::error` + exit 1 → PR check fails, code cannot merge. ### 3.5 Port Table (inside `aibox dashboard`) ```bash $ aibox dashboard # the overview ends with the allocation table ... Port assignments: Module Port/Proto Usage Status base 35432/tcp postgres listening clash 7890/tcp mixed listening gitlab 8929/tcp http — openmaic 3000/tcp app — pi-web 30141/tcp http — windmill 80/tcp http listening ``` Developers run `aibox dashboard` before adding a module to see what's allocated — and which declared ports are actually listening (live `lsof`/`ss` probe) — then pick a free port. Collected live from module.yaml's ports field (dynamic, no need to maintain docs/ports.md). (History: this was the standalone `aibox ports` command; merged into the dashboard overview in CLI v2.1.) ### 3.6 Runtime Probing (at install, optional) `aibox install ` reads the module's ports and probes actual usage via `lsof -iTCP:port` → if occupied by a non-aibox service, it warns ("port X is occupied externally; you can override with PI_WEB_PORT"). This layer does not gate submissions; it only notifies. Users can override the actual port via environment variables (`PI_WEB_PORT`, etc.). ### 3.7 Shared-Component Ports The ports of shared PG/Redis (§5) are also declared in the base module's module.yaml + checked by CI. Recommended to use non-default ports (PG 35432, Redis 36379) to avoid collisions with system services. --- ## 4. Dashboard TUI ### 4.1 Goal Each module has a Dashboard showing: whether it is deployed, endpoint (browser URL/port), and initial password/key, for easy login and use. ### 4.2 Commands - `aibox dashboard` —— global overview table (MODULE / VERSION / ENDPOINT / CREDENTIALS) followed by the port-allocation table - `aibox dashboard --available` —— registry catalog (all modules, installed marks) - `aibox dashboard ` —— single-module detail (port/URL/credentials/log path/health) ### 4.3 TUI Implementation Pure bash (colors + tables, zero dependency). **Does not use dialog/whiptail** (not on macOS). Output like: ```text MODULE VERSION ENDPOINT CREDENTIALS ───────────────────────────────────────────────────────────────────────────── ✔ base 1.1.0 pg://127.0.0.1:35432 (use… PG user/password aibox/* (overr… ✔ clash 1.1.0 socks5://127.0.0.1:7890 API secret 420e47ce4c470caf19d80c… ✘ gitlab 1.0.0 — not installed ✔ windmill 1.2.0 http://127.0.0.1:80 CREDENTIALS.txt + .env (POSTGRES_… Port assignments: Module Port/Proto Usage Status base 35432/tcp postgres listening windmill 80/tcp http listening ✔ installed ✘ not installed — detail + health: aibox dashboard ; catalog: aibox dashboard --available ``` ### 4.4 Module Interface: `dashboard_info()` Each module's `lib.sh` implements `dashboard_info()`, outputting key=value (parsed by the aibox caller): ```bash # tools//lib.sh dashboard_info() { # output key=value, collected by aibox dashboard echo "endpoint=http://127.0.0.1:${PORT}" echo "credential=Username pi / password ${PASSWORD}" # password read from plist (resolve_password) echo "log=${LOG_DIR}/pi-web.log" echo "health=ok HTTP 200" # status string probed by the module (shown as-is) } ``` The convention is written into module-spec. **The module.yaml `dashboard` section is only a static fallback when not installed, and only the `endpoints[]` + `hint` subfields are allowed; once a module is installed, runtime info (endpoint/credentials/log/health) always follows `dashboard_info()` output, overriding the yaml.** ### 4.5 Credential Sources (per module) | Module | Credential source | | --- | --- | | base | `base[-profile].env` (PG user/password; injected into consumers via `--env-file`) | | pi-web | plist/unit `PI_WEB_PASSWORD` (read by resolve_password) | | clash | state `CLASH_SECRET` (API auth) | | windmill | `$WM_DIR/CREDENTIALS.txt` + `.env` (POSTGRES_PASSWORD, etc.) | | openmaic | `.env.local` (API Key, access password) | | gitlab | 24h `initial_root_password` in the container → `aibox gitlab credentials` (reset recipe after expiry) | ### 4.6 Health Check `aibox dashboard ` does a lightweight probe of the endpoint (curl health endpoint, short timeout) and shows the result. Depends on port declaration (§3) + the module's dashboard_info health. ### 4.7 Dependencies - Port declaration (§3) —— endpoint ports come from module.yaml ports - Module `dashboard_info()` interface --- ## 5. Shared Base Components (PG/Redis) ### 5.1 Goal When modules depend on the same component, start only one instance; each module uses an independent database (named `` or `_`) to save resources. ### 5.2 Current State - openmaic compose starts app + PG + render (its own PG) - windmill compose starts windmill + PG + caddy (its own PG) - Duplicate PG instances, wasting resources ### 5.3 Design **Add a `tools/base/` module**: manages shared PG 18 + Redis 7 (one docker compose). ```yaml # tools/base/module.yaml name: base provides: # declares components exposed to others (full names) - postgres - redis ports: - 35432/tcp:postgres # non-default port + full-name purpose label, avoids colliding with system PG - 36379/tcp:redis deps: - docker - docker-compose actions: - start - stop - restart - status - create ``` **Refactor each deployment-type module's compose**: - Remove its own PG/Redis service - Connect to the shared instance (join the `aibox-base` network + env points to shared PG via `${AIBOX_POSTGRES_*}`, values injected from `$AIBOX_HOME/base.env`) - At `init`, create its own DB in the shared PG: `` or `_` ### 5.4 DB Naming Convention - Single DB: `` (e.g. `windmill`, `openmaic`) - Multiple DBs: `_` (e.g. `moduleA_jobs`, `moduleA_cache`) - Prefix = module name, avoids cross-module collisions The convention is written into module-spec. ### 5.5 Implementation Notes (provider model + base.env injection, see [design doc](design/module-yaml-refactor-design.md) §5–§6) - `base` acts as a **provider**; the single source of truth = `tools/base/lib.sh` holds connection constants → `base start`/`restart` writes `$AIBOX_HOME/base.env` (`AIBOX_POSTGRES_*` / `AIBOX_REDIS_*`, full names). base.env is consistent with `base/docker-compose.yml` defaults, guarded by CI deps-lint. - Consuming modules declare dependencies in module.yaml `services: [base:postgres#]` (full names). - When aibox schedules compose, for modules declaring `services` it automatically appends `--env-file "$AIBOX_HOME/base.env"` (compose v2.24+ merges multiple `--env-file`s) → `${AIBOX_POSTGRES_*}` can be interpolated inside the module compose. - Modules **hand-write one stable** `docker-compose.shared.yml` (network join + `replicas:0` + `environment: DATABASE_URL=postgres://${AIBOX_POSTGRES_USER}:${AIBOX_POSTGRES_PASSWORD}@aibox-base-postgres:${AIBOX_POSTGRES_PORT}/${AIBOX_POSTGRES_DB}`), **zero hard-coded credentials**; the module's `.env` sets `AIBOX_POSTGRES_DB=` (the DB name belongs to the module). - Changing base credentials/ports → `base restart` rewrites base.env → each module `aibox restart` (compose up) interpolates the new values and reconnects automatically. **No generator, no extra files, no sync-deps command.** - `tools/base/lib.sh` still manages the shared compose lifecycle (start/stop/status) + DB creation tool (`base create postgres [purpose]`); `aibox install` sees `services` → ensures base is up + `base create postgres ` creates the DB. - Version compatibility: module.yaml can declare `deps` containing `postgres:16` (base manages versions, modules declare compatible versions; **full name**) - Network: the shared compose creates the docker network `aibox-base`; each module hand-writes an override to join ### 5.6 Phased Rollout (Avoid Big-Bang) 1. **Phase 1**: build the `tools/base/` shared PG/Redis module + DB creation tool 2. **Phase 2**: new modules use shared (compose connects to shared, no longer starts its own PG) 3. **Phase 3**: migrate existing modules (openmaic/windmill compose refactor + data migration `pg_dump` to shared PG) Migrating existing modules carries data risk; iterate separately. New modules use shared directly. ### 5.7 Fallback When a module needs an exclusive PG version (incompatible with shared 18), it can fall back to an independent instance (compose starts its own PG). module.yaml **does not** declare that `services` (aibox then does not add `--env-file`; the module's bundled compose starts a local instance); if you need to pin a version, use the existing `deps` mechanism to declare `postgres:14` (full name), without introducing `@standalone` syntax. ### 5.8 Dependencies - Port declaration (§3) —— shared PG/Redis ports registered + CI detection - module.yaml (§2) —— base module declaration --- ## 6. Module Development Guide ### 6.1 Goal Each module has a development guide; AI upgrades modules based on it (project homepage, docs, install/test manual, configuration status). ### 6.2 module.yaml upstream Field ```yaml upstream: homepage: https://github.com/agegr/pi-web docs: https://github.com/agegr/pi-web#readme install: https://github.com/agegr/pi-web#installation test: https://github.com/agegr/pi-web#development ``` aibox `aibox dev-guide` outputs these links (AI checks upstream docs first when upgrading). ### 6.3 `docs/DEVELOPMENT.md` (per module) `tools//docs/DEVELOPMENT.md`, containing: - **upstream**: homepage + official docs (same as module.yaml upstream, may elaborate) - **Install manual**: upstream native install + aibox module install (`aibox install `) - **Test manual**: how to run tests (upstream tests + aibox module tests) - **This module's configuration**: ports (§3), password/credentials (§4), file locations (deploy root/config), customization points (aibox's changes to upstream) - **Upgrade flow**: steps for AI to upgrade this module (check upstream new version → modify module.yaml version → run tests → `aibox update`) ### 6.4 AI Upgrade Flow ``` 1. Read module.yaml upstream.docs → check upstream new version + changes 2. Read docs/DEVELOPMENT.md → understand this module's customization points + test methods 3. Modify module.yaml version + sync code 4. Run tests (DEVELOPMENT.md test manual) 5. aibox update verify 6. commit (CI port/spec validation) ``` ### 6.5 Dependencies - module.yaml (§2) —— upstream field - Independent (pure docs) --- ## 7. Dependency Management (Already Implemented, Documenting Current State) ### 7.1 Current State - registry `deps` field → migrated to module.yaml `deps` (format unchanged: `command@platform:version`) - `bin/aibox`'s `check_deps` + `dep_satisfied` + `install_dep` are implemented - `cmd_install` calls `check_deps` (before the install hook) - Platform filtering: non-empty `platform` that doesn't match is skipped; `@platform` is skipped + notified - Auto-install: lightweight install (brew/apt/yum/dnf); sudo/GUI-requiring ones (Docker Desktop) prompt for manual install; node reuses nvm ### 7.2 Migration to module.yaml ```yaml deps: - "node:22" - npm ``` awk parses it as `AIBOX_MODULE__deps="node:22 npm"`; check_deps logic is unchanged. --- ## 8. Auto-Start Platform Conventions (Already Implemented, Documented) ### 8.1 Convention (AGENTS.md Item 10) Auto-start for persistent services follows the platform; **do not mix**: - **macOS → launchd**: user-level `~/Library/LaunchAgents/