# Incus Manager

Incus Manager builds immutable Ubuntu images and reconciles Incus instances from
strict TOML recipes. Packages, tools, users, files, source trees, services,
networking, persistence and checks are data. Repository-backed scripts remain an
explicit, audited escape hatch.

## Mental model

A recipe is a reusable blueprint; it is not deployed directly. A deployment
selects one recipe, gives it an instance name, and supplies machine-specific
overrides and secret references. Each deployment manages one Incus instance,
although that instance may run several services. A recipe may inherit from a
parent recipe by setting `recipe.base`; users extend the parent rather than
copying it.

Built-in recipes and their supporting files are shipped inside the installed
`incus-manager` package. A workspace may add recipes under
`<root>/recipes/<name>/recipe.toml`; user recipes are searched before built-ins,
so they can inherit packaged recipes without a source checkout. A user recipe
with the same name intentionally overrides the packaged recipe. Deployment TOML
files can live anywhere and refer to recipes by name.

Commands mirror that lifecycle: `recipe` → `image` → `deployment` → `instance`—
inspect the blueprint, build the artifact, reconcile desired state, operate the resource.

```text
optional parent recipe
    └── selected child recipe ── image build ──> immutable image
                 ├── deployment A + image ──> Incus instance A
                 └── deployment B + image ──> Incus instance B
```

This is not a Compose-style multi-instance file: run `deployment apply` for
each deployment separately. Multiple deployments may reuse the same recipe and
image.

## Install and inspect

Python 3.11 or newer and Incus are required. Distrobuilder is only required for
image builds.

```bash
uv tool install .
incus-manager doctor
incus-manager recipe list
incus-manager recipe schema > /tmp/incus-manager-recipe-schema.json
incus-manager recipe validate agent-sandbox
incus-manager image plan agent-sandbox
```

After updating an existing checkout, refresh the installed helper with
`uv tool install --force .`.

Membership in `incus-admin` is optional. When the current user cannot access
the local Incus socket, manager commands retry Incus through `sudo` and may ask
for the user's sudo password. `doctor` reports this as `sudo required` without
prompting.

For repository development, commands can instead be run as
`PYTHONPATH=src python -m incus_manager ...`.

For an installed tool with recipes in another workspace, pass that workspace as
the root or set `INCUS_MANAGER_ROOT`:

```bash
incus-manager recipe --root ~/incus-config list
incus-manager image --root ~/incus-config build my-recipe
INCUS_MANAGER_ROOT=~/incus-config incus-manager deployment apply ~/deployments/my-agent.toml
```

## Deploy

Copy `deployments/example.toml` to an ignored `*.local.toml` file and override
machine-specific values there. `pool = "auto"` and `network = "auto"` succeed
only when Incus reports exactly one usable candidate.

The resource namespaces operate at different layers:

| Command | Input | Purpose | When to use it |
| --- | --- | --- | --- |
| `image plan RECIPE` | Recipe name | Render the low-level distrobuilder definition | Inspect or debug an image build |
| `image build RECIPE` | Recipe name | Build and import an image as `incus-manager/RECIPE` | Initially and whenever the recipe's image content changes |
| `deployment plan DEPLOYMENT` | Deployment TOML | Preview instance configuration changes | Before applying a deployment |
| `deployment apply DEPLOYMENT` | Deployment TOML and imported image | Create or reconcile the actual Incus instance | Initially and after deployment configuration changes |
| `deployment verify DEPLOYMENT` | Deployment TOML | Run the declared health checks | After apply or update |

`image build` builds in a temporary workspace, imports the result into Incus,
and removes the intermediate files after a successful import. It does not create
a container. `deployment apply` automatically selects the recipe's default
`incus-manager/RECIPE` alias:

```bash
incus-manager image build agent-sandbox
incus-manager deployment plan deployments/my-agent.local.toml
incus-manager deployment apply deployments/my-agent.local.toml
incus-manager deployment verify deployments/my-agent.local.toml
incus-manager deployment status deployments/my-agent.local.toml
```

Use `image build RECIPE --alias ALIAS` to choose another image alias. Use
`--output DIR` when portable artifacts and the host build manifest must be
retained; the image is still imported. The advanced `--no-import` mode requires
`--output DIR`.

`distrobuilder` requires root privileges. `image build` tries it normally first
and retries through `sudo` when the tool reports that root is required.

`deployment plan` is read-only. `deployment apply` creates missing resources
and reconciles existing ones. It records separate image and instance hashes
under the ignored `.incus-manager/state/` directory. `deployment adopt` only
records the current deployment as managed; it changes no Incus resource.

Useful lifecycle commands include:

```text
image plan|build|list|inspect|prune
recipe list|show|validate|schema
deployment plan|apply|adopt|status|update|verify
instance list|status|start|stop|restart|clone|destroy
instance shell|root|exec|logs|push|pull
instance mount add|remove|list
instance gpu attach|detach|status
instance snapshot create|list|restore|schedule
doctor
```

`instance destroy` and `image prune` refuse destructive work without `--yes`.

## Daily operation

Commands under `instance` accept `project/container`, an unqualified container
with `--project`, or a deployment TOML. Managed instances record their primary
user and workspace during `deployment apply`, so direct commands retain
non-root defaults.
Unmanaged instances default to root unless `--user` is supplied. Shells,
commands, logs, and transfers start stopped instances automatically.
Commands under `deployment` require deployment TOML because they need recipe
context. `instance status` reports runtime state; `deployment status` also
calculates declarative drift.

```bash
incus-manager instance list                           # all projects
incus-manager instance list --project agents          # one project
incus-manager instance start agents/agent-example
incus-manager instance shell agents/agent-example
incus-manager instance root agent-example --project agents
incus-manager instance exec agents/agent-example -- git status
incus-manager instance clone agents/agent-example disposable-copy

# Defaults to the first user-owned managed directory (normally /workspace/).
incus-manager instance push agents/agent-example ./project
incus-manager instance pull agents/agent-example /workspace/project/output ./output

# Deployment TOML remains accepted when declarative context is useful.
incus-manager instance shell deployments/my-agent.local.toml
```

Permanent mounts belong in deployment TOML under `instance.devices`;
`deployment apply` then owns their configuration. Manual mounts are available
for unrestricted projects:

```bash
incus-manager instance mount add services/web host-data /srv/data /data --readonly --shift
incus-manager instance mount list services/web
incus-manager instance mount remove services/web host-data
```

The restricted agent recipes intentionally block additional disk devices, so
manual host mounts cannot weaken their filesystem isolation.

Snapshot schedules declared in TOML are applied automatically. They can also
be inspected or applied explicitly:

```bash
incus-manager instance snapshot schedule deployment.toml       # use declared defaults
incus-manager instance snapshot schedule agents/agent-example --show
incus-manager instance snapshot schedule agents/agent-example --schedule '@hourly' --expiry 2d
incus-manager instance snapshot schedule agents/agent-example --disable
```

## Host networking

Docker can set a forwarding policy that prevents Incus bridge traffic from
leaving the host. If that affects this machine, install the optional unit from
`host/systemd/incus-docker-forward.service` and adjust its bridge name if
necessary:

```bash
sudo cp host/systemd/incus-docker-forward.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now incus-docker-forward.service
```

The manager does not modify host firewall policy automatically.

## Recipes

- `ubuntu-service`: common Ubuntu/systemd image base.
- `agent-sandbox`: restricted unprivileged development environment. Its agent
  vendor remains a deployment choice; no vendor credential or installer is
  embedded.
- `craftax-cpu`: extends the sandbox with a CPU-only uv environment, upstream
  source and a local progress/checkpoint patch.
- `hermes-native`: native Hermes and Camofox users, services, proxy, persistent
  paths, health checks and a ten-minute idle check. The slow idle check only
  sleeps when `deployment verify --honor-idle` is used.
- `nes-rl`: GPU-enabled, ROM-free Stable-Retro and nes-py research environment
  with root-owned Python stacks and workspace-local game integrations.

Runtimes (`uv`, `bun`, `fnm`/node, `rust`) are not baked into images. Recipes
declare them as per-user `instance.toolchains`; `deployment apply` installs them
into the owning user's home, owned by that user, so the user can update both the
runtime and the packages installed through it without root. See
[`docs/recipe-schema.md`](docs/recipe-schema.md) for resources, merge behavior,
toolchains, shells, dotfiles, secrets, and the extension contract. A
machine-readable structural schema is available through
`incus-manager recipe schema`.

## Development

```bash
PYTHONPATH=src python -m unittest discover -s tests -v
python -m compileall -q src tests
```

The project is MIT licensed. Machine-local deployment TOML, secrets, plans,
exports, backups, build output and runtime state are ignored by Git.