# Operating the `arca` binary The `arca` binary is Automagik Arca's one program. It runs one node's one-shot host reads, mints scoped bearer tokens, runs the recording shim, and serves the Arca MCP. Every command that needs the node reads one JSON config file, given with `--config`. ## Status Automagik Arca runs your services on machines you own. Set up the way the quickstart shows, an assistant can only look and propose. You approve with your passkey. - Arca is one program. Arca is open source under the Apache-2.0 licence. - A fresh install only looks: it changes nothing until you tell it which parts are its own. - Arca is self-hosted: it runs on machines you own, and Arca itself needs no outside service to work. - Arca keeps an append-only journal of the changes it applies. Starts, stops and restarts are logged separately. **Status: alpha.** Arca is an alpha. Each machine runs its own Arca. Machines you pair form a group, and an assistant can call a paired machine with a token that machine made; Arca does not yet spread services across machines by itself. Today it needs a Linux machine (x86_64 or aarch64) with Incus, and a person comfortable with a terminal. The current version is on the [releases page](https://github.com/automagik-si/arca/releases). | Works today | Not yet | |---|---| | Ready to run today: Postgres, NATS, Omni, Automagik, a Cloudflare tunnel and a workspace virtual machine. | The installer leaves a node that only looks, and one step lets it change things: `arca init --enable-changes`, or `--enable-changes` on the install line. That step does not turn on virtual machines (the workspace and the GitHub and GitLab runner pools), tenants, or taking over a service that already runs on the machine: each is set in the config by hand ([Turning on changes](#turning-on-changes)). | | Arca can save snapshots of a service onto a disk you plug in. | Arca cannot yet restore a snapshot for you: restoring is done by hand with Incus. | | Arca speaks MCP, so an assistant can ask it questions and propose changes. | Arca cannot yet run an app you name as a service: a service comes from a ready template, or Arca takes over one that already runs on the machine. A small program in an image you name can run on a timer as a function, once you approve its card. | | Two machines can be paired with a one-time code. | The ChatGPT and Claude apps cannot connect to Arca in one click yet. | | Arca checks passkeys itself; no outside sign-in service is involved. | Arca does not repair a service by itself. Where the root part of [Host forwarding and the firewall](#host-forwarding-and-the-firewall) is installed, it re-applies the declared forwarding and firewall each time it starts (boot, Docker start or restart, reinstall). Drift after that raises an alert; an assistant can propose a restore card, and Arca itself puts nothing back without the owner's approval, though a local user can ask the root part to apply. | ## What it does today - **Reads.** Every project and instance is classed by owner and drift is reported (`watch_report`); host, pool, instance and per-card GPU readings (`node_metrics`); alerts with a store of their own and a Prometheus export (`alerts_list`, `GET /metrics`, `GET /alerts`). - **Writes.** A node whose config names kernel projects runs services from templates, Postgres databases, virtual machines on bridged and mixed nodes, tenants in projects of their own, GPU assignment, instance start, stop and restart, and snapshots onto a disk the owner plugs in. A `viewer` token only reads and proposes: a change an assistant proposes waits for the owner, who approves it with a passkey on the approval page that `arca serve` serves. A delete of a network, a project or a volume needs a signed owner approval. A node with no kernel project is watch-only and writes nothing. - **Functions.** A program in an OCI image by digest, run on a timer in a container of its own that is started for each run and stopped after it, with a time, a memory and a CPU limit, secrets given by name, and a network card that reaches only what the owner's card names ([Functions](#functions)). - **Passkeys.** Arca is its own passkey verifier (WebAuthn ES256 and RS256). The first passkey is enrolled with a one-time code shown on the node's own screen, and a trusted passkey adds or removes another. Until a passkey is trusted, every approval fails closed. - **Adoption and import.** `arca adopt` takes one listed container from the default project into a kernel project of its own and back, keeping its MAC and IPv4. `service_import` takes over, in place, a service another manager created. - **The link.** `arca link` pairs two nodes with a one-time code and carries MCP calls between them (`arca call --peer `). - **The canvas.** [`packages/arca-canvas-dsh`](../packages/arca-canvas-dsh) is a DSH plugin that draws the node's machines, projects and services, with a chat and in-canvas approvals. - **As a service.** `deploy/systemd/` holds two user units that run `arca serve` read-only, reaching Incus only through the GET-only shim, with restart and memory limits. `docs/OPERATING.md` is the reference for all of it, and [`docs/OWNERSHIP.md`](OWNERSHIP.md) states what the kernel may touch. ## Commands `--config` and `--token-file` have defaults, the files the installer writes: `~/.config/arca/node.json` and `~/.config/arca/assistant-1.secret`. The order is the flag, then the variable (`ARCA_CONFIG`, `ARCA_TOKEN_FILE`), then the default, and every command finds its config this way. `arca status` alone takes one step more: when no token file is named and `~/.config/arca/assistant-1.secret` is not there, it uses a viewer token of this node from that directory, the first `.secret` file there that the node's token directory holds as a viewer that may call `arca_status` (the `viewer.secret` of a node installed before that default existed). A token of a wider scope is never taken, a token file the flag or the variable names is read alone, and a refusal of the token it found names that file. A command that writes to a node or acts on it, and was not given `--config`, first says on stderr which config it uses and how it was found (`arca: using the config (the default)` or `(from ARCA_CONFIG)`). A command that removes or cuts something off (`token revoke`, `link remove`, `probe consumer remove`, `adopt apply`, `adopt release`, `init --watch-only`, `templates install --replace`) also names the node's state directory in that line. A command that only reads says nothing, and a config named with `--config` is not said back. One thing has no default: a call with `--peer` sends its token to another node, so its `--token-file` is always named. `token mint` and `token revoke` with `--hash-dir` alone read no config at all, not even the variable's. **The variable alone never picks the node for a command that hands out, removes or rewrites something.** `token mint`, `token revoke`, `token reload`, `call --peer`, `link pair`, `link join`, `link remove`, `probe consumer remove`, `adopt apply`, `adopt release` and both modes of `init` would succeed on whatever node `ARCA_CONFIG` was left set for, and the installer mints the same token id on every node. So with the variable set and no `--config`, they go on only when the variable names the same file as the default. When it names another file they refuse with exit 2, do nothing, and say the flag to run them with: `ARCA_CONFIG names , not the default config (). ... run the same command with: --config `. With `--config`, or with no variable set, nothing changes. `templates install --replace` overwrites templates that were edited, so it names the node's state directory in its first line as the removing commands do. A writing command also refuses a config it found by itself that belongs to another user, and says whose it is. This is root with a user's home directory kept (`sudo` without `-H`): the default path would lead to that user's node. Run the command as that user, or name the config with `--config`. `token mint` writes the hash and sends no signal: a node reads its tokens when it starts and on SIGHUP, so a running node takes a new token after `arca token reload` or a restart. | Command | Reads | Prints | |---------|-------|--------| | `arca status [--json \| --security] [--config ] [--token-file ]` | the running node, with the same call as `arca call arca_status` | the node in words: version, mode, uptime, whether Incus answers, the unix socket it answered on, the loopback door as the node itself reports it (open with its URL, or CLOSED with the reason), the other listeners the config file names (not checked); then approvals, from the node's own answer: whether changes are on, the page that approves and whether it is open, whether a passkey is trusted yet, and then two last lines: `Next:`, the ONE thing to do (a node that only looks names `arca init`; with no passkey, the page and its button at a terminal with a screen, and `arca setup-code` with the `ssh` line at one without), and `More:`, which names `arca status --security`. With `--json`, the `arca_status` result as it came. With `--security`, everything that is not the next step, from the config and the token directory alone, so the node need not be running: the cautions for each door this config opens, what a node that may change things may touch, which tokens do not wait for the page and what they may do, where the templates and the config `arca init` kept are, the way to the page from another computer, what a phone needs, and what a node set back to watch-only keeps | | `arca watch --once --json [--config ]` | every instance in every project, through Incus GET | the `watch_report` result | | `arca metrics --json [--sample-for ] [--config ]` | the host and the configured pools | the `node_metrics` result | | `arca store check [--config ]` | nothing from the node: it opens a scratch store in the state directory and removes it | the SQLite version and the pragmas the store reads back, as JSON; see "The store" | | `arca templates install [--config ] [--replace]` | nothing from Incus: it writes the shipped service templates, each with its entrypoint or first-boot script, into `/templates` | `{ "dir", "installed", "unchanged", "kept", "replaced" }` as JSON. A file that is an earlier release's own, byte for byte, is brought up to this release's and named under `replaced`; any other file that differs is the owner's edit and is kept unless `--replace` is given | | `arca update check [--config ]` | the releases and this systemd manager's units: nothing is changed and nothing is written | `{ "result", "reason", "available", "current", "detail": { "programs": [...] } }`: `available` when a program is behind the newest release; each program with its path, version, units and whether it is behind. See "Keeping Arca up to date: the kernel update" | | `arca update run [--config ]` | the same, and installs the release into every program behind it, restarts their units, and rolls back a unit that does not come back; what `serve` starts on its schedule | the same object, with the units moved under `detail.units`; exit 1 when the run `failed`, was `rolled-back` or `held-back` | | `arca probe consumer add --name --image [--config ]` / `remove --name [--config ]` | locks desired state; add reads the default profile root pool with GET, then proposes a consumer entry | entry id and sequence | | `arca serve [--config ]` | runs until SIGTERM or SIGINT; SIGHUP reloads the token store only | nothing | | `arca init --enable-changes [--config ] [--yes] [--service-cidr ]` | the config, the machine's routes and addresses, and Incus through GET | the plan and two questions; it writes the config's missing keys after a yes, and ends in five lines: what changed, who does not wait for the page, the one next step, where a database comes from, and one pointer to the rest. See "Turning on changes" | | `arca init --watch-only [--config ]` | the config | it empties `kernel.projects` and nothing else, and says in a few lines whether the running node took that up; until it has, the first line says the node still makes changes | | `arca setup-code [--config ]` | asks the running node, on its unix socket, for the setup code of the first passkey; for a machine with no screen | `{ "code", "expiresInSeconds", "page" }` as JSON on stdout, and one sentence on stderr; see "The page on this machine alone" | | `arca host-network status [--live] [--config ]` | the root part's status file, or with `--live` its socket | the host as declared, or what drifted, with the status's age; see "Host forwarding and the firewall" | | `arca host-network install [--takeover ] [--bridges ] [--replace] [--dry-run]` | the host's forwarding and firewall now (root only, `--dry-run` too) | the steps and the declared state it wrote, and the root part's first reading | | `arca host-network run` | nothing: it is the root part the system unit runs (root only) | nothing | | `arca link serve [--config ]` | runs the node's link until SIGTERM or SIGINT | nothing on a clean run | | `arca link pair [--config ] [--minutes ]` | the running link's status | `{ "link", "expires" }` as JSON on stdout; a terminal QR on stderr | | `arca link join [--] [--config ]` | pairs with the endpoint in the link | `{ "id", "name" }` as JSON | | `arca link peers [--config ]` | the paired peers file and, while serving, the path each is reached on | the peers array, as JSON | | `arca link remove [--] [--config ]` | removes every peer with that name through the running link, or from disk when stopped | `{ "removed": }` as JSON | | `arca call [--config ] [--token-file ]`, with `--args ` or `--args-file `; `arca call` alone lists the verbs | one tool call against the local node | the verb's result, as JSON | | `arca call --peer [--config ] --token-file `, with `--args ` or `--args-file ` | one tool call through a paired peer. The token is sent to the peer, so it is one the peer minted and `--token-file` is always named: this node's own token is never sent by default | the verb's result, as JSON | | `arca token mint --scope --id --out [--tools ] [--tenant ] [--services ] [--expires ]` with `--config `, `--hash-dir ` or neither (the config every command finds) | nothing | the id and scope, and the tenant, services and expiry when given | | `arca token revoke ` with `--config `, `--hash-dir ` or neither (the config every command finds) | the token's hash file | the id and `revoked: true` | | `arca token reload [--config ]` | the pid file and the reload record in the state directory, and `/proc` | one line: `reloaded: the node on this config (pid ) read its tokens again and holds token(s)`. It exits non-zero, with the reason on stderr, when no node runs on the config, when the node could not read its token directory, or when it cannot tell | | `arca shim --upstream --listen --log [--log-max-bytes ]` | forwards GET, refuses the rest | the shim summary, on SIGTERM; the summary so far on stderr, on SIGUSR1 | | `arca adopt plan\|apply\|release\|verify\|status --instance ... [--config ]` | one listed container, and its adoption record | the plan, the steps run, or the check made; see "Adopting a container" | | `arca --version`, `arca version` or `arca -V` | nothing | one line: `Automagik Arca (, )`; the commit reads `unknown` when the binary was built with no git checkout | | `arca service import plan --name [--config ]` | one instance `kernel.importServices` lists, and the instance listing, through Incus GET | the import's dry run as JSON: every check, the one write and the pre-image digest; see "Importing a service another manager created" | `watch` and `metrics` print the same JSON as the MCP verb of the same name, so a command's output can be compared with the server's answer. Exit codes: `0` success; `1` a runtime failure (a socket, a read or a write failed); `2` a usage or config failure (a bad command line, a refused config, or a token directory that is unsafe or missing); `3` a tool error from `call`, whose JSON-RPC error object is printed on stdout; `4` a token the node refuses, HTTP 401 or 403. Other messages go to stderr. A message never carries a token, a secret or a key. `metrics` samples once at start and then every 5 seconds for `--sample-for` seconds (default 5, from 0 to 60). Its `mean` covers every sample of the run, and the `node_metrics` window it prints is one minute. ## The config file One JSON document with camelCase keys. Every unknown key, at any level, is refused. A value that is not the right type is refused without the value being echoed back. ```json { "stateDir": "/var/lib/arca", "incus": { "socket": "/var/lib/incus/unix.socket", "pools": ["default"] }, "identityPhones": ["phone1"], "listen": { "unix": "/run/arca/arca.sock", "tailnet": null, "proxy": null }, "tokens": { "hashDir": "/etc/arca/tokens" }, "kernel": { "projects": [], "scratch": false, "droiddProjects": ["app1"], "reservedCidrs": [] } } ``` | Key | Required | Meaning | |-----|----------|---------| | `stateDir` | yes | An existing directory, mode 0700. The node key and the desired-state files live in it. See the state section below. | | `incus.socket` | yes | The Incus unix socket. Reads only. | | `incus.pools` | no | An allow-filter on the storage pools Incus lists. Absent, or `[]`, means every pool Incus lists. A non-empty list samples and alerts only the listed pools that Incus also lists. A listed name that Incus does not list has no row, so its firing alert resolves with `subject-gone`. A pool entry with no name makes the pool reading unknown for that tick. See "Alerts". | | `incus.local` | no | Whether `incus.socket` reaches the local Incus daemon. When absent, the node asks Incus `GET /1.0` for `environment.server_name` and compares it with the local host name: a match is local, a mismatch is remote. The socket path rule (the standard local daemon sockets) is the fallback only when `/1.0` or the host name cannot be read. An explicit value wins. See "Host metrics". | | `identityPhones` | no | Instance names the watch flags with `identity: true`. | | `listen.unix` | yes | The MCP socket. Created with mode 0600. | | `listen.loopback` | no | One loopback address and port, such as `"127.0.0.1:7471"`, or `null`: the HTTP MCP door for a client on the same machine. It serves the MCP routes alone, never the approval page, to loopback peers that name it by its own address or as `localhost`. `null` is the owner's word for no door: `scripts/install.sh` writes it for `--no-door` and never opens a door in a config that says it. See below. | | `listen.tailnet` | no | One tailnet address, or `null`. See below. | | `listen.proxy` | no | `{ "addr": ":", "peer": "" }`, or `null`: one address for a reverse proxy on the same host, served to the single peer address `peer`. See below. | | `listen.services` | no | The path of a second unix socket, or `null`: the services socket. It serves the MCP routes alone, never the approval page, to the server's own user and to root. A service with kernel access connects to it. See below, and "A service that may talk to the kernel". | | `tokens.hashDir` | yes | The directory of hash files. See below. | | `kernel.projects` | no | The project family the kernel may write in. With projects named, the writer serves only those projects. With none, the kernel is watch-only: the reconciler's provider reads only, service writes are refused, no GPU device writer exists, and a removal cannot reach Incus. | | `kernel.scratch` | no | When `true`, every kernel project must start with `arcak-scratch-`. | | `kernel.droiddProjects` | yes | droidd's projects. Required, and it must not be empty while `kernel.projects` is not. | | `kernel.serviceCidr` | no | Service bridge `/24`, outside every `kernel.reservedCidrs` range. Required for Postgres creation and probe consumer add; no default. | | `kernel.postgresImage` | no | The old way to name a database's image; leave it unset on a new node. A digest-pinned reference (`name@sha256:<64 hex>`) to an image Incus already holds under that fingerprint: Arca does not fetch it. With it, `postgres_create` makes a database of the old kind from exactly this image, proposed or not. Without it, `postgres_create` makes a service from the node's `postgres-autopg` template, whose image Incus fetches by digest. See "A database". | | `kernel.reservedCidrs` | no | Subnets reserved for droidd. Read by later steps only. | | `kernel.adopt` | no | `[{ "instance": "", "to": "arcak-adopt-" }]`: the containers `arca adopt` may take from the default project, each with its adoption project. A config that lists one is for `arca adopt` alone, and `serve` refuses it. | | `kernel.attachNetworks` | no | The bridges in the default project an adoption project's grant may name, though they carry no kernel mark. Nothing writes them. An adoption whose `eth0` is on a bridge this list leaves out plans that bridge's grant as an allowance. | | `kernel.bridged` | no | Bridged mode: `{ "bridges": ["", ...], "pool": "", "limits": { "virtualMachines": n, "cpu": n, "memory": "", "disk": "" } }`, every key required. The host bridges, which already exist, that a virtual machine's NIC may join. Without `kernel.serviceCidr` the node is bridged: it runs virtual machines only and Arca creates no Incus network on it. With `kernel.serviceCidr` the node is mixed. See "Bridged nodes and virtual machines" and "Mixed nodes". | | `kernel.tenants` | no | `{ "capacity", "maxTenants", "defaultCaps", "createRate", "front"? }`: the node may hold tenants. It needs `kernel.serviceCidr` and is refused beside `kernel.adopt`. See "Tenants". | | `kernel.vmLimits` | no | `{ "virtualMachines": n, "cpu": n, "memory": "", "disk": "" }`, every key required: the virtual-machine caps of a node that has `kernel.serviceCidr` and no `kernel.bridged`. Without it such a node creates no virtual machine. It is refused beside `kernel.bridged`, which states the same caps in its `limits`. See "Mixed nodes". | | `kernel.importServices` | no | `[{ "name": "", "project": "", "cidr": "/24" }]`: the services another manager created that `service_import` may take over where they stand. Each `project` must be listed in `kernel.droiddProjects`, and `cidr` is that project's service subnet. Empty by default: nothing is imported and nothing is written in another manager's project. See "Importing a service another manager created". | | `kernel.serviceDestinations` | no | `[{ "service": "", "address": "", "port": n }]`: the private addresses named services of the node's own project may reach through the node, each on one tcp port. It needs `kernel.serviceCidr`. Empty by default: a service reaches no private address. An entry alone opens nothing. See "A service that may reach one private address". | | `kernel.serviceListenAddresses` | no | `["", ...]`, at most 8: the host addresses a service's private port may listen on, each an address of an interface of this host, such as the address of a private link to a second node. Empty by default: no service has a private port. An entry alone opens nothing. See "A service port on one host address". | | `link` | no | The node link: `name`, `forward`, `pairMinutes`, `localOnly`, `port` and `peers`. See "The link". | | `follow.intervalSeconds` | no | Seconds between two looks of the Automagik follower at Automagik's published record, 60 to 86400. 600 when absent. | | `update.automatic`, `update.channel`, `update.checkEverySeconds`, `update.window`, `update.source`, `update.healthSeconds` | no | The kernel update: whether the node looks for a newer Arca release and installs it on its own (`true`), which releases (`stable`; `prerelease` takes pre-releases too), how often it looks (600 to 604800 seconds, 21600 when absent), when an install may start (`HH:MM-HH:MM` in UTC; absent, at any hour), where the releases are listed (`https://api.github.com/repos/automagik-si/arca`), and how long a restarted unit has to come back and then stay up (5 to 600 seconds, 30). A node installs only a signed release, and no signing key exists yet. See "Keeping Arca up to date: the kernel update". | | `hostNetwork.socket`, `hostNetwork.status` | no | Absolute paths of the root part's socket and status file; `/run/arca-host-network/host-network.sock` and `/run/arca-host-network/status.json` when absent. See "Host forwarding and the firewall". | | `console.tokenFile` | no | Absolute path of the file `serve` writes the machine's own console's token to: one viewer token limited to `node_metrics`, `arca_status`, `alerts_list` and `service_list`, made at each start and held in memory only. Absent, no such token is made. See "The console's token". | | `approvalHost` | no | The host name every owner approval statement names. When it is not set, the system's host name is used. | | `approvalOrigin` | no | The page origin the owner approval page is served at: `https://` plus a lowercase host name and at most a non-443 port, with no path (for example `https://arca.example.com`); or `http://localhost:` with the port of `listen.loopback`, the page of a node on its own machine with no domain and no certificate. Its host is the WebAuthn rp id; it is not the host an approval statement names. | | `canvasOrigin` | no | The loopback DSH origin the remote canvas proxies: an `http://` literal loopback IP and port, with no path. Default `http://127.0.0.1:3081`; any other origin refuses the config before `serve` listens. | | `approval.trustOwnerKeysFrom` | no | Absolute paths of other kernels' owner passkeys files on this machine, which this kernel also trusts, read-only, for action approvals only (never to change its own passkeys). See "Approving on another kernel of this machine". | | `approval.origins` | with `trustOwnerKeysFrom` | The page origins (`approvalOrigin` form) a passkey from those files may answer from. A line for any other origin is not trusted. | | `approval.linkedHosts` | no | `[{ "name": "", "socket": "", "tokenFile": "", "incusSocket": "" }]`: the kernels on this machine whose proposals this kernel's approval page lists and relays approvals to. The optional `incusSocket` is that host's Incus socket on this machine (local, or forwarded to a path here); `instance_move_preflight` reads the host through it, with GETs only, and an approved `instance_move` writes through it (the copy it receives, its start, and its removal on a rollback). The optional `ceph` (`{ "cluster": "", "client": "", "readClient": "" }`) names that host's Ceph cluster as this machine reaches it; a stopped virtual machine on Ceph then moves there by `rbd` from this machine (see [Stopped virtual machines](#moving-containers-and-stopped-virtual-machines-to-a-linked-host-instance_move)). | The config is refused, with exit 2, when: - a key is unknown, a required key is missing, or a value has the wrong form; - `follow.intervalSeconds` is not a whole number from 60 to 86400; - `hostNetwork.socket` or `hostNetwork.status` is not an absolute path; - `console.tokenFile` is not an absolute path that names a file (not `/`, not ending in `/` or `..`); - `approvalHost` is not a host name: one letter or digit, then up to 252 letters, digits, dots or hyphens; - `approvalOrigin` is not `https://`, a lowercase host name, and at most a non-443 port, with no path, nor `http://localhost:`; or it is `http://localhost:` and `listen.loopback` is not set to `127.0.0.1` or `[::1]` on that port; - `canvasOrigin` is not `http://`, a literal loopback IP, and a nonzero port, with no path; - `listen.loopback` is not a loopback address (`127.0.0.0/8` or `::1`; a wildcard, a tailnet address, any other address of the machine and an IPv4-mapped address are refused), or its port is 0, 80, 443, 7443 or 9443; - `listen.tailnet` is not a tailnet address: it must be a fixed port, not a wildcard, inside `100.64.0.0/10` or `fd7a:115c:a1e0::/48`, and not on port 80, 443, 7443 or 9443; - `listen.proxy.addr` is a wildcard, a tailnet address (IPv4-mapped included), port 0, or port 80, 443, 7443 or 9443; `listen.proxy.peer` is missing or a wildcard; or `listen.proxy` carries any other key; - `listen.services` is not an absolute file path of at most 100 bytes made of letters, digits, `/`, `.`, `_` and `-` with no `..` part, or it is `listen.unix`'s path; - a kernel project is `bench`, `default`, or a name in `kernel.droiddProjects`; - `kernel.projects` is not empty and `kernel.droiddProjects` is empty; - `kernel.scratch` is `true` and a kernel project does not start with `arcak-scratch-`. - `approval.trustOwnerKeysFrom` names a relative path, or names a file while `approval.origins` is empty; an `approval.origins` entry is not an `approvalOrigin`; or a linked host's name is not a host name or is listed twice, or its `socket` or `tokenFile` is a relative path. - an adopted instance is not a plain name (`^[a-z][a-z0-9-]{0,56}$`), its `to` does not start with `arcak-adopt-` or is not in `kernel.projects`, an instance or a `to` is listed twice, an attach network is not a plain name, or `kernel.scratch` is `true` while `kernel.adopt` is set. - a `kernel.importServices` entry carries a key other than `name`, `project` and `cidr`, its name is not a service name, its project is not listed in `kernel.droiddProjects`, its cidr is not an IPv4 /24, or a name is listed twice; or the list is not empty on a node with no `kernel.serviceCidr` (a bridged node among them) or with `kernel.scratch`. A mixed node, which sets `kernel.bridged` beside `kernel.serviceCidr`, may list imports. - a `kernel.serviceDestinations` entry carries a key other than `service`, `address` and `port`, its service is not a service name, its address is not one IPv4 address (a range and a name are refused), the address is outside the private ranges of RFC 1918 and `100.64.0.0/10` or inside `kernel.serviceCidr`, it is `100.100.100.100` or `100.100.100.200`, its port is 0, an entry is listed twice, one service has more than 4 entries or the list more than 16; or the list is not empty on a node with no `kernel.serviceCidr`. - a `kernel.serviceListenAddresses` entry is not one IPv4 address written as four plain decimal numbers (a range, a name and an address with a leading zero are refused), is not a unicast address (`0.0.0.0`, a loopback, a multicast address and `255.255.255.255` are refused), is listed twice, or the list has more than 8 entries. Whether an address is on an interface of the host is not asked here, since a config is read on machines that are not the node: `serve` asks at start (see "A service port on one host address"). No command reads Incus before the config passes these checks. ### Loopback listener `listen.loopback` is the HTTP MCP door for a client on the same machine: an assistant that takes an MCP server URL and cannot open a unix socket. `"listen": { "unix": "...", "loopback": "127.0.0.1:7471" }` gives the URL `http://127.0.0.1:7471/mcp`. `scripts/install.sh` writes this key, on port 7471 or the next free port up to 7480. The key takes a loopback address and nothing else. No value of it reaches another machine: for that there is `listen.tailnet` or `listen.proxy`, each under its own rule. What the listener serves and refuses: - **The MCP routes, and the page of this machine alone.** `POST /mcp` and the read-only GET routes, each behind a bearer token: 401 without one. An approval page with an `https://` origin is never on this listener: `/approve` and everything under it is 404. The one page it carries is the page whose origin is `http://localhost:` with this listener's port; see "The page on this machine alone". - **Loopback peers only.** A peer that is not a loopback address is closed at accept with no bytes sent. - **Only a request that names it.** The `Host` must be the listener's own address and port (`127.0.0.1:7471`) or `localhost` and that port (`localhost:7471`), compared without regard to case. Every `Host` header a request carries must be one of the two, and so must the authority of an absolute request target. Any other name, the same name with another port or with none, and a request with no `Host` are 403 on every route, before the token is looked at and before the body is read. This is what stops DNS rebinding: a web page loaded from a foreign name that is then pointed at `127.0.0.1` sends that foreign name as its `Host`. - **No `Origin`.** As on every listener, a request that carries an `Origin` is 403, so a web page cannot call the node from a browser. - **A bounded wait for each request.** A peer has 30 seconds to send a request's head, the first of a connection and each next one on a connection that is kept open. A peer that sends nothing in that time is closed, so idle connections cannot hold the listener's 64 slots. Use the address, `http://127.0.0.1:7471/mcp`, in a client, not the name `localhost`. The name can also mean `::1`, where this listener does not listen and another program of the machine could. What it does not do. The connection is plain HTTP, which on loopback never leaves the machine. Every user and every process of the machine can connect to the port; without a token each gets 401, and the token's file is readable by the node's user alone. A client that is given the token keeps it in its own config. **The port can be taken by someone else.** A TCP port belongs to whoever binds it first. On a machine shared with other accounts, another account can take the port while the node is not running (before it starts, after a crash, during an upgrade) and would then receive what a client sends there, the bearer token included. The unix socket and `arca call` are not exposed to this: the socket lives in a directory only the node's user can open. On such a machine, leave the door off (`"loopback": null`, or `sh install.sh --no-door`) or accept that the token a client holds is a `viewer` token another account could come to read. A port that is in use when `serve` starts does not stop the node. The unix socket and every other listener serve as usual, `serve` says once on stderr `arca: the loopback door is closed: address in use: another program holds this port. The node runs without it and tries the address again`, and it tries the bind again after 1 second, then after twice as long each time, up to 30 seconds, for as long as it runs. When a try succeeds it says `arca: the loopback door is open`. `arca_status` reports the door of a node whose config names one, as `"loopback": { "addr": "", "open": true | false, "reason": "" | null }`, with a `"page"` object beside them when the node's approval page is on this listener (see "The page on this machine alone"). A node that serves an approval page also reports `"approvalPage": { "url", "open", "reason", "passkeys" }`: the page's address, whether it is open (for the page of this machine alone, whether the node holds both loopback addresses), the reason while it is not, and how many owner passkeys the node trusts (`null` while the kernel is busy with a change). `arca status` says these in words. Any token may call `arca_status`, a `viewer` token among them, so an assistant given one sees the page's address and the count. `arca status` also reads the node's token directory, to say beside "Changes: on" whether any `doctor`, `operator` or `admin` token changes things without the page. While it reads `"open": false`, whatever answers on that port is not this node: do not send the token there. ### Tailnet listener Serving on a tailnet address is a separate step. The owner approves it before `serve` is started with a tailnet address, and until then the node runs on its unix socket alone. A peer outside the tailnet ranges is closed with no bytes sent. ### Proxy listener `listen.proxy` is for a reverse proxy on the same host, such as Caddy in a container on the host's bridge, which connects from its bridge address rather than from a tailnet address. The listener binds `addr` and serves exactly one peer address, `peer`; an IPv4-mapped peer is compared as its IPv4 address. Every other peer is closed at accept with no bytes sent. The bearer and `Origin` rules are the same as on the unix socket, so the proxy's peer gets the same 401, 403 and 200 answers. The bridge may not be up yet when the server starts at boot. While `addr` is not assigned to the host, the bind is tried again for up to 120 seconds, after a pause of 1 second that doubles each time up to 16 seconds, and each try is logged on stderr (`arca: the proxy address cannot be bound yet ...`, then `arca: the proxy address is bound`). The unix socket and the tailnet listener serve meanwhile. An address still not assigned after 120 seconds stops the server with exit 1, and any other bind error, such as a port in use, stops it at once. The connection is plain HTTP on the bridge, and a host that can take the peer's address can reach the listener; the bearer token is still required. Serving on a proxy address is a separate step, approved by the owner like the tailnet listener. ### Services socket `listen.services` is the path of a second unix socket, for example `/run/arca/services.sock` beside `listen.unix`. It is bound as the first one is (mode 0600, renamed into place) and serves the server's own user and root, and nobody else. It serves the MCP routes and nothing else: `POST /mcp` and the read-only GET routes, each behind a bearer token. The approval page is never on it, on any node and whatever the config becomes: a request for `/approve` or anything under it is 404, and a request that carries an `Origin` is 403. A node may serve its approval page and have this socket too; the two sockets are separate, and one path cannot be both. Incus connects to this socket for a service with kernel access (see "A service that may talk to the kernel"), and Incus's proxy helper runs as uid 0: that is why uid 0 is served. Serving uid 0 means every process of the host that runs as uid 0, and Incus's proxy helper acting for any instance whose device names this path. Arca gives such a device to a service of the node's own project alone, a tenant's project blocks proxy devices, and whoever can add one by hand already administers Incus. None of them gets anything from the socket without a token. No other local user reaches the socket, and it has no port: nothing on the host can take a port in its place while the server is down. The directory that holds the socket must belong to the server's user or to root and be writable by its owner alone, and so must every directory above it, except that a directory above it may be sticky instead, as `/tmp` is. `serve` refuses to bind the socket anywhere else, naming the directory: a user who could write there, or rename a directory on the way, could put a socket of their own at the path and be sent a service's token. The socket has its own limit of 64 connections, apart from every other listener's. A clean stop removes it. ## Turning on changes A node the installer leaves only looks. One step lets it change things, and one step takes it back. Neither needs a second machine, a domain or a certificate. ```sh arca init --enable-changes --config ~/.config/arca/node.json ``` or, at install time, `sh install.sh --enable-changes`, which installs and then runs the same step with `--yes`. **What waits for the owner, and what does not.** What an assistant proposes waits: nothing of it runs until the owner approves it with a passkey on the approval page, and until the first passkey is enrolled nothing can be approved at all. The installer's token is a `viewer` token, which can look and propose and nothing more, so on a node as the installer leaves it every change waits for the passkey. A `doctor`, `operator` or `admin` token does not wait. On a node that may change things, whoever holds one may, with no card and no passkey, create a service from a template (the shipped ones among them), connect and disconnect services, set a service's secret, unpublish one, start an instance, link and unlink a database, create a database from an image the token names, remove a database's record, password and links, and use the Omni verbs; on a node that holds tenants, also create and remove a tenant. Such a token gains that reach the moment the node may change things, so the step reads the token directory first and **names every `doctor`, `operator` and `admin` token before it asks**. With none it says so, and then "every change waits for your passkey" is true as written. `arca token revoke --config ` takes one away. `--yes` does not answer for such a token: where one exists, or the token directory cannot be read, the step under `--yes` stops, names them and changes nothing, and so does `sh install.sh --enable-changes`. Run it without `--yes` to answer yourself. **What it asks**, each with a default: 1. `Let Arca make changes on this machine, each only after you approve it? [y/N]`, or, where a `doctor`, `operator` or `admin` token exists, `... what is proposed after your approval, and directly for the tokens named above?`. The default is no. `--yes` answers it on the command line. 2. `Service network is free. Use it? [Y/n]`. The default is yes. `--service-cidr ` names another, which is checked the same way. **What it writes.** Only keys that are missing; a key that is set is never changed. | Key | Written when | Value | |-----|--------------|-------| | `kernel.projects` | it is empty | `["arcak-svc"]`, the one Incus project Arca's services live in | | `kernel.droiddProjects` | it is empty | `["default"]`. Arca never writes in `default`; the entry only names it as not Arca's, which the config requires of a node that names a project of its own | | `kernel.serviceCidr` | it is absent | the /24 for the bridge `arcak-svcbr`, which Arca makes for its services | | `approvalOrigin` | no approval page is named | `http://localhost:`, on the loopback door the config already opens | An approval page the config already names, `https://` or `http://localhost`, is kept as it is, and the step says which page approves. The config as it was is kept beside it as `.before-init`, and both files are the user's alone. The new config is checked as `serve` checks it before it is written, so the step never leaves a config the node refuses. Then it installs the shipped templates into the state directory (`arca templates install`), and restarts the node only when the node is the installer's own service for this config (`arca.service`, written by `scripts/install.sh`). A node started any other way is the owner's to restart, and the step says so. **What it says when it is done** is five lines, so the one thing to do is not lost among the rest: 1. what changed, and whether the running node took it up. "Arca may now make changes" is said only when the restarted node answered so itself; 2. who waits for the page: either that no token of this node may do more than look and propose, or every `doctor`, `operator` and `admin` token by name, with the line that revokes one. This line never leaves the report; 3. `Next:`, the ONE thing to do. A node that was not restarted: restart it, with the command. A node that was restarted while init had no token to ask it with: ask it, with `arca status`, whether its page is open, before a setup code is typed or anything is approved there. A page the node does not hold: type no code there. Otherwise the passkey: at a terminal with a screen (`DISPLAY` or `WAYLAND_DISPLAY` is set), the page and its "Get a setup code" button; at one without, `arca setup-code`, the page to type the code on, and for the page of this machine alone the `ssh` line that brings the page to your own computer and the sentence that a phone cannot open it. With a passkey already trusted: ask your assistant for something; 4. `A database: ask your assistant for the postgres-autopg service.`; 5. `More:`, one pointer: `arca status` for what is open and what comes next, `arca status --security` for everything else, and `arca init --watch-only` to go back. Nothing the longer report used to say is gone. What Arca may touch, that it never writes in `default`, what a `doctor` or `operator` token may do, the cautions of the page of one machine, the `ssh` line, the phone, the templates' directory and where the config as it was is kept are all said by `arca status --security --config `, from the config, with the node running or not. **How the service network is chosen.** It is read from the machine, never guessed. The step takes the first of `10.47.0.0/24` to `10.47.255.0/24` that meets none of these, and names each source in what it prints: - a route in the kernel's routing table (`/proc/net/route`); - an address of the machine itself (`/proc/net/fib_trie`); - a network Incus manages, and an address on any interface Incus lists; - a range `kernel.reservedCidrs` names; - the tailnet range `100.64.0.0/10`, which is never used whatever the tables say. When one of the sources cannot be read, the step asks whether to go on with what it could read (default no), and under `--yes` it stops. A network named with `--service-cidr` must be an IPv4 /24 and pass the same checks. **What it refuses, changing nothing:** - a config with no approval page and no loopback door for one. The step opens no door: a missing or `null` `listen.loopback` may be the owner's no. Open it (`sh install.sh --door`) or set an `https://` `approvalOrigin` first; - a loopback door that is not on `127.0.0.1` or `[::1]`, which the name `localhost` does not reach; - a machine where an instance carries another manager's marks. It names what it saw; list that manager's projects in `kernel.droiddProjects` by hand, and the step fills that key only on a machine that has none; - a config set up by hand for something else: `kernel.scratch`, `kernel.bridged`, `kernel.adopt`, or `kernel.projects` that names projects without `arcak-svc`; - a service network that is not free, or no free one among its own; and a network named with `--service-cidr` that is not inside the private ranges of RFC 1918, since a public, multicast or link-local one would pass every other check and then hide those addresses from the machine; - a config that is a link (name the file it points to), and a `.before-init` that is a link or a directory. The copy and the new config are written as new files of the user's own, no link is followed, and the directory is synced after the rename; - an Incus that does not answer on `incus.socket`; - root. It runs as the node's user and never runs `sudo`. The one program it starts is `systemctl --user`, to restart the installer's unit. It finds its config like every command that writes: the flag, then `ARCA_CONFIG` (which alone never picks the node for `init`), then the default file. **Going back.** ```sh arca init --watch-only --config ~/.config/arca/node.json ``` empties `kernel.projects` and changes nothing else, then restarts the installer's service as above. Its report is three or four lines. It says "Arca now only looks" only when the restarted node answered so itself; otherwise its first line says that the node that is running still makes changes until it is restarted, and its `Next:` line is that restart. Then that nothing was removed, and one pointer: `arca status`, `arca status --security` for what was kept, and `arca init --enable-changes` to turn changes on again. Once restarted, the node only looks: it starts, stops, creates and removes nothing, and a proposal that waits can no longer run. Nothing is removed: the Incus project `arcak-svc`, its bridge, the services in it and the owner's passkeys stay as they are. Running `--enable-changes` again adds the project back and keeps the network and the page. **A node that only looks still shows what is there.** After `--watch-only` the reading tools answer as they did: `service_list`, `service_status`, `service_logs`, `postgres_list`, `tenant_list`, `tenant_status` and `tenant_usage` read the node's own records and Incus, exactly as on a node that makes changes. What differs: - Every answer says what the node is. Its text, in a block after the answer, and its structured content under `watchOnly`, carry `this node only looks: nothing here can be changed until the node is running with changes turned on (arca init --enable-changes, then restart the node if init did not)`. The sentence is about the node that is running: one whose config already turns changes on, but that was not restarted, still only looks, and the restart is the step that is left. - Nothing is adopted. An instance in `arcak-svc` that the node holds no record of is shown as it is, with the state `unknown` and the note `not recorded by this node`; a node that makes changes would have written a record for a marked one. - `service_logs` reads the instance's console log, and its `note` says so: the journal is read by running a command inside the instance, and a node that only looks runs none. A container's console log holds only what its init system wrote to the console, which is often much less than its journal, and may be nearly empty. - `omni_status` is refused, with that reason: it answers by running a command inside the service. - Every tool that would change an instance, a service, a database, a tenant, storage or a GPU assignment is refused with that same sentence, before an approval is spent and before anything is saved: a tenant's own calls, the `deploy_*` tools, `tenant_create` and `tenant_remove` included. A refusal of an instance start or stop, of a storage change or of a GPU assignment keeps its own code (`InstanceLifecycleDisabled`, `StorageDisabled`, `GpuDisabled`) and carries the same sentence. The tools that only keep or settle a note for the owner (`action_propose`, `owner_approval_request`, `owner_approval_status`, `approval_submit`, `enroll_confirm`, `plan_decline`, `peer_keys_sync`) still record it: what they record cannot change the node until changes are turned on. A node that only looks and holds records of services it ran, but whose config lost `kernel.serviceCidr` too, shows them the same way: the records are enough to tell which instances are its services. A node that never had a service config and holds no record answers an empty list. **What this step does not do:** - It does not make every shipped template start from the page alone. A template that makes every secret it needs starts once its card is approved: `postgres-autopg`, `nats`, `omni-api` and `automagik-server`. A template that needs a secret of the owner's is created stopped until it is set: `cloudflare-tunnel` (`TUNNEL_TOKEN`), the runner pools and `automagik-workspace` (their tokens and keys). Omni starts, and its Telegram and WhatsApp tools need `telegram_bot_token` and `whatsapp_phone_number`. `automagik` needs an image built from a checkout before it can be created, and its `tailscale_authkey` before it can be published. `service_secret_set` cannot be proposed, so it needs a `doctor` token, which changes things with no card and no passkey: mint one for that step and revoke it after, or leave those templates until setting a secret can go through the page. - It does not enrol a passkey. The owner does, on the page, with a setup code from the node's own screen or from `arca setup-code`; see "The page on this machine alone". - The page it names is the page of one machine, for a machine where every account is trusted. Approving from a phone or from another machine needs an `https://` origin, set by hand. - It sets no `kernel.postgresImage`, and none is needed. **A database after this step is the shipped `postgres-autopg` service:** ask the assistant for it, or propose `postgres_create` with a name, or `service_create` with `"template": "postgres-autopg"`. Both make the same service and wait for the owner's card like any change (see "A database"). The template fetches its own pinned image and makes the database's password itself at create (32 letters and digits, stored as the service's `POSTGRES_PASSWORD` secret and never shown), so the approved create starts by itself, and a proposed `service_connect` gives the linked service its `DATABASE_URL`. No `service_secret_set` is needed, which matters because that verb cannot be proposed and needs a `doctor` token. See "Templates an earlier release installed" below for an older node. No call chooses the image: a `doctor` or `operator` token's own `postgres_create` runs without a card, as its `service_create` of that template does. It sets no `kernel.vmLimits`, so no virtual machine is created. Tenants, adoption and import are not turned on. A node that may change things also runs its snapshot writer (snapshot pools and the project `arcak-snapshots`) and its GPU device writer: each acts only from a card the owner approves. - It does not check that Incus can fetch a template's image; a create that cannot answers in Incus's own words. - It grants no access to Incus. The node's user must already be able to open the Incus socket, which is the machine's administrator's to grant, as is keeping the service running without a login session. ## The link `arca link serve --config ` runs one iroh endpoint with a persistent key. The optional `link` config section has camelCase keys and refuses unknown keys: - `name`: the node's name, from 1 to 60 characters, without U+0000..U+001F control characters. When absent, the name is the first 10 characters of its endpoint id. Incoming device names have those controls stripped, are capped at 60 characters, and fall back to `device` when empty. - `forward`: an ALPN-to-unix-socket-path map. Each admitted stream is forwarded unchanged in both directions to its socket. `arca link serve` adds `arca/mcp/1` pointing to `listen.unix` unless the map already has an explicit entry for that ALPN. - `pairMinutes`: the default code lifetime in minutes, overridden by `link pair --minutes`. The default is 10, and the selected value is clamped to 1 through 60. - `localOnly`: when true, binds only loopback and disables discovery and relays. The default is false (N0 discovery and public relays). It also disables mesh reachability: the node neither reaches peers on other hosts for membership or forwarding, nor is reached by them. This also permits testing `link serve` without external network access. - `port`: a fixed UDP port, 1 to 65535, that the endpoint binds on every address (IPv4, and IPv6 where the host has it), so this node's address is stable and a firewall can open it. When absent, the port is random. With `localOnly` it applies to the loopback address. A port that is already in use stops `link serve` at start. - `peers`: a map from a peer's name (as `arca link peers` shows it) to 1 to 8 `ip:port` addresses it is also reached on, such as `{ "gpu-node": ["10.0.0.2:7000"] }` for the address a private fiber link gives that node (IPv6 is written `[2001:db8::2]:7000`). An address needs a port and must be a unicast address; a bad one stops the config at load. See "Preferring a private network". ### Preferring a private network Without `peers`, iroh chooses between the relay, a VPN address and a LAN address by itself. With it, the named addresses are used in two ways, and both only add: nothing is excluded, so a peer stays reachable when the named link is down. - Every connection this node opens to that peer (membership sync, `call --peer`) also tries the named addresses, next to the address recorded when the two nodes paired. - Among the paths open on a connection this node opens, a path on a named address carries the traffic while it works, ahead of any other direct address and the relay. Inside that group the lowest round trip wins, and a switch needs a gain of 5 ms. When the named path closes, the next best path takes over. Only the side that opens a connection chooses its paths, so each node's `peers` governs the connections that node opens. A connection the other node opens is steered by the other node's settings: without `peers` there, it may still use the VPN. For the private link to carry both directions, set `port` on both nodes and name each other in `peers`: on each node, `:`. Each node must accept UDP on its port on the private interface. iroh 1.4.1 has no per-peer setting for this; the rule is a `PathSelector`, an unstable iroh interface behind its `unstable-custom-transports` feature, which is why iroh is pinned to that exact version. `crates/arca-link/src/path.rs` lists what to re-check when iroh is updated. To see which path a peer is reached on, run `arca link peers` while `link serve` runs: each peer then has a `path`: `{ "via": "direct" | "relay" | "idle", "addr": "", "named": true | false }`. It comes from the connections open at that moment, each one's selected path, so `"via": "direct"` with `"named": true` and the private address confirms the link is in use right now. `idle` means no connection to that peer is open, which is the usual state between two syncs (every 30 seconds; a sync lasts moments): run a call to the peer, such as `arca call --peer `, and look again while it runs. A peer has no `path` key when `link serve` is not running or its record is older than 10 seconds. An older arca refuses a config with `peers` or `port` (the section refuses unknown keys): take the two keys out before going back to an older release. The endpoint accepts `arca/pair/1` for pairing, `arca/peers/1` for membership, `arca/mcp/1` for peer calls, plus the ALPNs in `forward`. Both membership and forwarded connections require a live entry in `peers.json`; a forwarded connection checks that entry before its local socket is opened. `arca/pair/1` and `arca/peers/1` cannot be forward entries. Each forwarded connection runs at most 16 streams concurrently; further streams wait until a running stream finishes. Handshakes have a 10-second deadline, with at most 16 pending and 4 pending per address. The 4-per-address cap applies only to pending handshakes; open forwardings are capped only in total, at 64 open forwarded connections. `link remove` uses the running server's control socket, closes that peer's live connections, and triggers mesh sync immediately. Without a running server it writes the removal to disk; the next start propagates it. External disk removals are detected within 2 seconds while serving. `arca/peers/1` exchanges one full JSON snapshot on one bidirectional stream: `{"peers":[...],"removed":[{"id","at"}]}`. The receiver merges the request and replies with its merged snapshot, so both ends learn membership changes. Each JSON line is bounded at 64 KiB including its newline; pairing retains its 4 KiB bound. Sync runs after successful pairing, after removal, at startup, and every 30 seconds. Offline members do not make these operations fail; they catch up on a later exchange. Any admitted member can introduce or remove peers received from another member; membership changes rely on that mesh trust. A removal tombstone wins when its `at` timestamp is at least the peer's `paired` timestamp. A newer pairing survives and clears the tombstone; older snapshots cannot resurrect a removed device while its tombstone is retained. Existing peer records change only when an incoming `paired` timestamp is newer. Tombstones are retained for offline members for up to 30 days, keeping at most the 64 newest (ties broken by endpoint id). Their timestamps are capped at 5 minutes ahead of the local clock. An older snapshot can reintroduce a removed device whose tombstone has expired or been displaced by the cap. All link state lives directly in the top-level `stateDir`: | File | Contents | |------|----------| | `secret.key` | The endpoint's 32 raw secret bytes, generated once. | | `peers.json` | An array of `{ "id", "name", "paired", "seen", "ticket"? }`; timestamps are epoch milliseconds. The optional endpoint ticket supplies transport addresses; older records without one use N0 lookup. Admission reads this file on every connection. | | `removed.json` | A tombstone map `{ "": }`, propagated in membership snapshots. | | `pairing.json` | The pending single-use code, expiry, node name and wrong-attempt counters. Deleted after successful pairing. | | `status.json` | The running server's `{ "pid", "id", "ticket", "name" }`. Removed on clean stop; stale or absent status prevents `pair`. | | `paths.json` | The running server's `{ "": { "via", "addr", "named", "at" } }`, rewritten every 2 seconds and read by `link peers`. Removed on clean stop. | | `.link-control.sock` | The mode 0600 local socket for joining and dialing through the running endpoint. Removed on clean stop. | The link creates a missing `stateDir` with mode 0700 and refuses an existing directory with any group or other permission bit, reporting `link state directory must have mode 0700` as a config failure (exit 2). For an older state directory, run `chmod 0700 ` before starting; the link does not change existing directory permissions automatically because the top-level directory may be shared with other services. These files are written atomically with mode 0600. `pair` requires a running `link serve`. It creates a code and a link of the form `arca://pair?v=1&n=&c=&t=`. A code works once, until its expiry. A peer is refused after three wrong tries, and 1,000 wrong tries overall exhaust the code. `join` consumes that link and stores the host peer locally; when a local server is running, it reuses that server's endpoint. `pair`, `join`, `peers` and `remove` print JSON on stdout. `pair` draws its terminal QR on stderr, so redirecting stdout captures only JSON. The link and QR contain the pairing code; keep them private until the code is used or expires. ## Tokens A token is a 256-bit secret of the form `arca_` followed by 43 base64url characters. The server never stores the secret. It reads one hash file per token from `tokens.hashDir`: ``` scope=viewer sha256=<64 lowercase hex digits> tools=service_list,service_logs,journal_list ``` The file name is `.hash`, and the id is 1 to 64 lowercase letters, digits or hyphens. The scopes are `viewer`, `doctor`, `operator` and `admin`, in that order. `admin` admits every verb `operator` does and one thing more: approving a waiting plan from the terminal, described under "Approving a plan from the terminal". **A node holds one admin token at a time.** `arca token mint --scope admin --id ` with another admin token on the node is refused with exit 2, names the token that exists and writes nothing. `--replace` mints the new one, then removes the old one's hash file and signals the running node to read its tokens (as `token revoke` does), so the old token stops at once and the node never has two nor none by a failed mint; the answer says `retired` and `reloadSignalled`. `--replace` is for the admin scope only. `token_list` shows an admin token as `scope: operator` with `admin: true`. The `tools=` line is optional; without it the token reaches every verb its scope admits, and with it the token reaches only those named tools. A tool above the scope is refused at mint, so the list only narrows. One id is no token's: `owner-approval`, the name an action runs under once the owner approved it. `arca token mint` refuses it. A release before this one did not, so a file `owner-approval.hash` may exist: the node starts and reads every other token, leaves that file out, and writes one line each time it reads its tokens ("token file owner-approval.hash is ignored: ..."). Its secret is refused as any unknown token is. Remove the file with `arca token revoke owner-approval`. An action approved in the terminal runs as `terminal-approval:`, which no token can be, since an id holds no colon. ### Tokens limited to services and to a time A token may also be limited to named services, to a time, or both. Three more lines, each optional, each at most once, may follow the digest in any order, beside `tools=`: ``` scope=doctor sha256=<64 lowercase hex digits> services=dev,gtm,abcdefgh/web expires=2026-12-01T00:00:00Z by= ``` - `services=` is a comma list of 1 to 64 services, none twice: `` is a service of the node's own service project, and `/` one inside that tenant. - `expires=` is a moment in UTC to the second, exactly in that shape. From that moment on the token authenticates nobody: a call is `401`, as for an unknown token. Nothing is removed; revoke the file as any other. - `by=` names the token that handed this one down. It loads only while that token loads, and only when it is no wider: no higher scope, and no tool, service or later expiry its parent lacks. Revoke the parent and the whole chain stops at the next reload. Nothing mints with `by=` yet; a later release adds the verb that does. `arca token mint --services ` writes `services=`, and `--expires ` writes `expires=`: a span from now (`30m`, `12h`, `7d`, or `s`), or a moment such as `2026-12-01T00:00:00Z`, after now and within ten years. An admin token is never limited to services: the mint refuses it, and a hand-made one is left out at load. **What a limited token reaches.** Every verb says what it touches. A call outside the list is refused before anything runs, with the refusal `OutsideTokenServices`, whose words never name a service: - a verb on one service (`service_status`, `service_logs`, `service_update`, `service_secret_set` and the other `service_*` verbs, the `omni_*` verbs) needs that service, with the call's `tenant` when it names one, on the list; - `service_connect` and `service_disconnect` need both services on the list; - `instance_start`, `instance_stop` and `instance_restart` need the instance of a listed service: `project` `arcak-svc` and the service's name, or inside a tenant its `service` (or its instance `-`); - `action_propose` is checked when it is made, against the proposed action's own arguments: once the owner approves it, it runs as the owner's action, outside any token's limit. A proposal of an action no service names is refused; - `service_list`, `snapshot_list` and `watch_report` run, and the door cuts their answer to the list: rows of other services go, and so does every link to one; a snapshot of a volume or a folder, and a project with no listed instance, go too; - every other verb reaches the whole node (`arca_status`, `journal_list`, `plan_list`, the tenant, database, function, route, storage and GitHub verbs, and the rest) and is refused. `tools/list` does not offer them to a limited token; - a call with `node` is refused: the limit is this node's, and a linked node does not hold it; - every GET route (`/metrics`, `/alerts`) is refused with `403`: its answer has no cut. **Older releases and new lines.** From this release on, a line `=` whose key the release does not know leaves only that token out: the node loads every other token, writes one line on stderr each time it reads its tokens ("token file .hash is left out, so its token is refused: ..."), and refuses that token's secret. The key may be a limit a later release added, so the token is never taken without it. Any other line that does not parse still refuses the whole directory. **A release before this one refuses the whole token directory when it meets `services=`, `expires=` or `by=`**: on such a downgrade the node starts with no token at all. Before downgrading past this release, revoke every token minted with `--services` or `--expires` (`grep -l -E '^(services|expires|by)=' /*.hash` lists them). ### Tenant-bound tokens `arca token mint --tenant ` adds `tenant=` to the hash file. The id is eight lowercase base32 characters (`a-z`, `2-7`). An admin token cannot be bound to a tenant. Minting needs no connection to Incus; the tenant must exist when a call uses it. For this token, omitting a verb's `tenant` selects the bound tenant. Stating the same tenant is accepted; another is refused with `OutsideTokenTenant` before a handler, relay or approval runs. A token without `tenant=` keeps its current default: omitted `tenant` means the node's own project. With `--tenant`, bare names in `--services` are tenant-local; `/` is also accepted for that same tenant. Names from another tenant and duplicate names after resolution are refused. The file and mint answer spell these service limits as `/`. The token reaches the tenant-aware service, instance and snapshot verbs, plus `deploy_confirm`, `deploy_rollback` and `deploy_remove`, within its scope, tool list and service list. Of the verbs without `tenant`, only `arca_status`, `tenant_list`, `tenant_status`, `tenant_usage` and `action_propose` are available. The tenant list contains only its own row; status and usage default an omitted `id` to its own and refuse another. A proposal resolves its inner arguments and checks the action's tenant, scope, tools and services before saving it. Every other node verb, including `node_metrics` and `watch_report`, is refused. Every linked-node argument and every GET and POST route outside `/mcp` is refused too. The tool list applies these same limits. No binding gives an approval or a standing grant. A token handed down by a bound parent must bind to the same tenant and remain within the parent's other limits. An unbound parent may hand down a bound child when that child's resolved services are within its own list. A release that skips unknown token lines leaves a token with an unknown `tenant=` line out. A release from before unknown-line skipping refuses the whole token directory on that line. Tenant-bound hash files may use up to 8 KiB; unbound files keep the 4 KiB limit. The mint checks the exact serialized size before writing either file and refuses combinations of tools and services that exceed the file's budget. An older reader with a 4 KiB limit refuses the whole directory for a larger file even if it skips unknown lines. Revoke bound tokens before downgrading to either reader. ### Modes and the umask **The token directory must be mode 0700 or 0500.** The server refuses any directory with a group or other write bit, and it refuses a directory it does not own. A host umask of 002 makes a new directory 0775, which the server refuses, so set the mode explicitly after creating it: ```sh mkdir /etc/arca/tokens chmod 0700 /etc/arca/tokens ``` Each hash file must be a regular file, not a symbolic link, mode exactly 0400, and owned by the server's user. A file of any other mode, owner or kind makes the whole load fail, and the message names the file, never its content. `arca token mint --tools ` writes the optional allowlist; an unknown name or one above the scope is refused with exit 2 before anything is written. `arca token mint` writes both files at mode 0400, refuses to overwrite an existing id or output file, and removes its hash file if it cannot write the secret. It needs the token directory to exist and to have no group or other write bit. Create the directory and mint into it before the mode is tightened to 0500, because a 0500 directory cannot take new files. ```sh arca token mint --scope viewer --id viewer-1 --hash-dir /etc/arca/tokens --out "$HOME/.config/arca/viewer-1.secret" ``` `--config ` may name the token directory instead, as `tokens.hashDir` of the node config, and then `--hash-dir` is not needed. The config is loaded and checked first. When both are given, `--hash-dir` must name that same directory; a different one is refused with exit 2 and nothing is written. Giving neither is refused with exit 2. **Run `arca token mint` as the user that runs `serve`.** A mint run as another user writes hash files owned by that user, and the server refuses every file it does not own, so the load fails until those files are minted again as the server's user. `arca token reload --config ` makes the node that runs on that config read its tokens again: the step after a `token mint` on a running node, which sends no signal itself. It finds the node as `token revoke` does, and signals only a process it has checked to be the node: one that belongs to this user and whose command line is `arca serve` on that very config, with `serve` as the command itself. The node's own link (`arca link serve` on the same config) is never taken for it, and neither is a pid file left by a node that crashed, whose pid is gone or is another program's. It then waits for the node's own record of the reload, `/serve.reload`, and answers in words: `reloaded`, with the pid and the count of tokens; `no node runs on this config`; `not reloaded`, when the node could not read its token directory and keeps the tokens it had; or `could not tell`, when the signal was sent and the node did not say so within 30 seconds (a node older than this command reloads without saying so). Only `reloaded` exits 0. A node that is still starting is left alone: `serve` takes `SIGHUP` as the first thing it does, and until it has, the signal would end the process. `token reload` then answers `the node on this config (pid ) is still starting; nothing was signalled. Run this again in a moment`, and `token revoke` says the same on stderr and reports `reloadSignalled: false`. Run both as the node's own user: a node that another user runs is never signalled, and the answer says so (`no node of yours runs on this config`). The node is found by the config path its `serve` was started with, so a config file that was moved or renamed after the node started is not found, and the answer is that no node runs on it; give the path the node was started with, or restart the node on the new path. `arca token revoke ` removes the token's hash file. With `--config`, it sends `SIGHUP` to a live `arca serve` for that config: it checks `/serve.pid` only when its pid is a live `arca serve`, and otherwise finds the same process in `/proc`. Once its unix socket is this process's own, the server writes and syncs the console token file (when configured), then publishes the pid, before it serves a request. A token write failure stops the start without publishing a pid. It removes the pid file only when it still holds its own pid; a pid file write that fails is said once on stderr and does not stop the node. A second start that fails to bind leaves the running server's pid file unchanged. The answer carries `reloadSignalled`; `token_list` drops the token and a call with it is 401 without a restart. With `--hash-dir` alone no state directory or serve is known, so no signal is sent (`reloadSignalled: false`) and a running server keeps the token until it is restarted or signalled. The hash files stay mode 0400. The signal goes through a pid descriptor (Linux 5.3 or later), never the host's `kill` command: the process is held, checked again and signalled through the descriptor, so no other process can receive it. The secret file holds the secret and one newline. A client reads that file, strips the newline, and sends the secret as `Authorization: Bearer `. Keep the secret out of command lines, logs and shell history. ### The console's token The machine's own console (on Arca OS, the screen a root program draws) reads from the node with a token like any other client. Nobody is at that machine to mint one, so the node makes it, when the config names a file for it: ```json "console": { "tokenFile": "/run/arca/console.token" } ``` - **What it may do.** It is a `viewer` token limited to four reads, fixed in the code and not in the config: `node_metrics`, `arca_status`, `alerts_list` and `service_list`. Any other verb is refused `403`, as for any token with a tool list, and the reply's one-line body names the verb (`forbidden: journal_list is not on this token's list of tools`). Its id is `node:console`; no hash file can carry that id, since an id there holds no colon. - **When it is made.** At each start, `serve` checks the file's place before anything listens, holds the new token in memory, and writes and syncs it to the file once its unix socket is this process's own, before publishing its pid or serving there: created new, mode 0400, never through a link, the token and one newline. It is never in `tokens.hashDir`. - **The start is refused, with exit 2 and words that name `console.tokenFile`,** when the file's directory is missing, is a link, is not owned by the user that runs `serve`, or is open to its group or others (it must be mode 0700); when a link is at the file's path; and when something other than this node's earlier file is there. The node leaves such a link or file alone. A file a node left when it was killed (this user's regular 0400 file, one token and a newline) is replaced. - **How long it lives.** A token reload (`SIGHUP`, `arca token reload`) keeps it and does not rewrite the file. A restart replaces it: the earlier value is `401` from then on, and the file holds the new one. A stop removes the file, when it is still the one this process wrote. - **Where the owner sees it.** `token_list` lists it after the stored tokens as `{ "id": "node:console", "scope": "viewer", "tools": [...], "stored": false, "console": true }`. `arca status --security` counts it among the tokens that only look and names its file and its four reads. `arca token revoke node:console` is refused with exit 2: there is no hash file to remove. **To turn it off, remove `console.tokenFile` from the config and restart the node.** Root on that machine can read the file, and root there can already do everything, so the file gives nobody new power; the bearer rule stays one rule on every route and every listener. A client that is not an MCP client sends two requests, each a `POST /mcp` over the node's unix socket (`listen.unix`) on its own connection, with the token read from the file at each call and the newline stripped. The socket answers only the user that runs the node. First `initialize`: ``` POST /mcp HTTP/1.1 Host: localhost Authorization: Bearer Content-Type: application/json Accept: application/json, text/event-stream {"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"console","version":"1"}}} ``` Its answer is `200` with `result.protocolVersion` `2025-11-25`. Then the call, here `node_metrics`: ``` POST /mcp HTTP/1.1 Host: localhost Authorization: Bearer Content-Type: application/json Accept: application/json, text/event-stream {"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"node_metrics","arguments":{}}} ``` Its answer is `200` with the reading in `result.structuredContent`, the same object `arca call node_metrics` prints, and a summary in `result.content[0].text`. A tool-level failure is `result.isError: true`. `401` means the token is not the node's current one (the node restarted, or the file is from an earlier start): read the file again and retry once; a missing file or socket means the node is not running. `403` means the verb is not one of the four. With curl, as the node's user, the header goes in on curl's standard input (`-H @-`), so the token is never on a command line, where other users of the machine could read it, and never in a second file: ```sh { printf 'Authorization: Bearer '; cat /run/arca/console.token; } | curl -s --unix-socket /run/arca/arca.sock -H @- \ -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"node_metrics","arguments":{}}}' \ http://localhost/mcp ``` A service is one template-derived container in the service project. Its template lives in `/templates/.json`; `service_create` allocates `.10`-`.239` on the service bridge, creates the service profile and NIC ACL, records the desired entry, and starts the container when every environment secret it needs is available; with `start: false` it never starts it (see "Creating a service stopped" below). `service_list` and `service_status` derive state from Incus and report instances that Arca did not create as `unknown` without touching them. A row's `ports` are the container's own ports; `lanPort` is the host proxy port when the template sets `lan`, or null. `lanPort` is a host listener on every interface; a row of a service with a private port also carries `privatePort`, its one host address and port (see "A service port on one host address"). A row carries `kernelAccess` only when the service's template declares it (see "A service that may talk to the kernel"). `service_logs` returns console lines as untrusted text with stored secrets redacted and a capped, truncated result. The thirteen shipped templates live in `templates/`, with their pinned images and vendored npm lockfile. Five of them, `workspace-vm`, `github-runner-pool`, `gitlab-runner-pool`, `gitlab-runner-pool-large` and `automagik-workspace`, are virtual machines (see "Bridged nodes and virtual machines", "Mixed nodes", "A pool of GitHub Actions runners", "A pool of GitLab runners" and "An Automagik workspace in its own virtual machine"). `service_list` and `service_status` rows include a stable `id`, the journal's instance `uuid`, RFC 3339 UTC `createdAt` and `startedAt` timestamps, recorded resource `limits`, and the service's own `data.volume`. The `id` survives rebuilds and kernel restarts; older records derive it on reads and save it on their next normal write. `startedAt` is null when it is unknown or the service never started. Do not downgrade below this release once a service was created or written: older releases cannot read a record that carries an `id`. `service_labels_set {tenant?, name, labels}` lets a doctor merge plain caller data onto a service; labels are never secrets or policy, and do not change boot behavior. A string sets a value (including an empty string), and null removes a key. Bounds apply after merging: 128 bytes per key, 4096 bytes per value, 128 keys and 65536 bytes of compact JSON. The full map is returned and appears as `labels` in `service_status` and `service_list`, empty when none is stored. Label text is caller data, so both reads are marked as carrying untrusted text. Label writes check ownership and If-Match, read the instance back, and are not journalled. Result `code: unconfirmed` means the PATCH answer was uncertain: the returned map is what was read, and the write may still land later; read status again before deciding what to do. If read-back is unavailable, `labels_unconfirmed` reports that uncertainty. Adopted and imported services show existing labels but take no label write. Snapshot rollback keeps labels best effort; a label read or repair failure is logged without values and never prevents an otherwise successful rollback. Rollback takes no node write gate: a label write that lands while the restore itself is running can be lost. Put-back only replaces a stored value still exactly equal to the snapshot's; a different value is left alone, including after an If-Match conflict is re-read. **Templates an earlier release installed.** A release may change a shipped template: up to 0.12.0, for one, the shipped `postgres-autopg` template declared `POSTGRES_PASSWORD` and did not make it, so a create came up stopped until a `doctor` token set the secret. A node keeps no such template by accident. `arca templates install`, which `arca init --enable-changes` also runs, compares each installed file with the files earlier releases shipped (the program holds their SHA-256 hashes): - a file that is an earlier release's own, byte for byte, is replaced with this release's and named under `replaced`; `arca init` names those in one line before its report. Nobody edited it, so nothing is lost; - a file that matches no release is the owner's edit. It is kept, under `kept`, as before, and only `--replace` overwrites it (that flag names the node first); - a template and the script it reads move together: where one of the two is edited, the other is kept as it is too, though it is an earlier release's, so a new template is never paired with a script it was not shipped with. The kernel update runs this step after every update it installs, for every node that has templates (see "Keeping Arca up to date: the kernel update"). The installer runs it on an upgrade (`sh install.sh` on a machine that has Arca), for the node it runs as a service and only where templates are already installed: a node that only looks has none and gets none. Its report names what it brought up to this release, and what it kept because it was edited, with the one command that replaces those (`arca templates install --replace --config `). Where the run puts an older program over a newer one (`--version`), the files it keeps may be the newer release's own: the report says so and gives no command to replace them. The step writes every file before it puts any in place; if it stops between two files, its words name what it had already put in place, and running it again finishes. For a node started any other way, run `arca templates install --config ` once after the upgrade, or `arca init --enable-changes` again; neither needs `--replace` for a template nobody edited. What a replaced template changes is the next create. A create that was proposed before the replacement and still waits is not made from the new file: its card bound the content of the template it showed, so approving it is refused and nothing is created; propose it again (see "What the owner saw is what is created"). `arca init` and the installer say so beside the names they replaced. Nothing changes for a service that already exists: a `postgres-autopg` service keeps the password that is stored, and none is made over it. One that was created stopped with no password gains none by itself either: set it with `service_secret_set`, or remove the service and create it again from the new template. A runner pool reads its script when its virtual machine is created, so a pool that runs keeps the script it has. `arca templates install --config ` installs the shipped templates in one step: each template's JSON and the script its entrypoint or its first boot reads (`postgres-autopg.sh` for `postgres-autopg`, `nats.sh` for both `nats` and `nats-primary`, `nats-replica.sh` for `nats-replica`, `github-runner-pool.sh` for `github-runner-pool`, `gitlab-runner-pool.sh` for both `gitlab-runner-pool` and `gitlab-runner-pool-large`, `automagik-workspace.sh` for `automagik-workspace`), from the build itself, so no checkout is needed. An entrypoint's `{ "file": "" }` names a plain file in the templates directory: letters, digits, dots, dashes and underscores, not starting with a dot, so no path. A template whose script is missing is refused by `service_create` with the file it looked for (`templates/`). A template whose `image` is `local:` (`automagik`; `automagik-server` is an OCI image) needs that image in the service project's image store first: `service_create` refuses with the alias when the project's image listing answers and no image there carries it, before anything is created. A listing that cannot be read decides nothing, and the create goes on. #### Service disk and network limits A template may add these optional caps beside its existing CPU and memory limits: ```json "limits": { "cpu": "2", "memory": "2GiB", "diskRead": "100MB", "diskWrite": "1000iops", "ingress": "200Mbit", "egress": "100Mbit" } ``` `diskRead` and `diskWrite` become Incus `limits.read` and `limits.write` on the **root disk**. Disk throughput is bytes per second: `B`, `kB`, `MB`, `GB`, `TB`, `PB`, `EB`, or the binary units `KiB` through `EiB`. An integer without a suffix is bytes/s; `iops` caps operations per second. Incus also accepts one of each, such as `"10MB,1000iops"`. A separate attached data volume is outside the root disk's cap. `ingress` and `egress` become `limits.ingress` and `limits.egress` on the service NIC, for traffic into and out of the instance respectively. Rates are bits per second: `bit`, `kbit`, `Mbit`, `Gbit`, `Tbit`, `Pbit`, `Ebit`, or binary `Kibit` through `Eibit`; an integer without a suffix is bits/s. Units are case-sensitive: `MB` is a disk rate and `Mbit` is a network rate. These follow [Incus's unit parser](https://github.com/lxc/incus/blob/v7.5.0/shared/units/units.go). Values must be positive whole numbers whose converted rate fits a signed 64-bit integer. Empty values, zero, negative values, fractions, wrong units and overflow fail `service_create` before any Incus write. Omit a field for no cap. A container may specify just I/O limits; virtual machines still need both CPU and memory limits and a root disk size. The caps travel in the instance create for containers and virtual machines, on either a managed service network or a VM's host bridge. Incus applies them at **start**. If a create requests a start and that start fails, Arca rolls back the fresh capped instance, its service record, ACL, new data volume and newly generated secrets, and returns the error. Files supplied before the create are kept. Arca never retries without the cap. Resolve cleanup or retention of objects left by an earlier attempt before a fresh capped create; rollback cannot claim that earlier data. If cleanup itself fails, the error says so. An explicitly stopped create applies its caps at its later `instance_start`, so creating it stopped does not prove the host can enforce them. Container disk throttling requires Incus's I/O cgroup controller and a backing block device the storage driver can identify. VM disk throttling uses QEMU. Network caps require the bridged NICs Arca creates. Validate the caps on the storage driver and network in use before relying on them for isolation; support on one driver proves nothing about another. Caps are fixed at create. Editing a template changes future services. `service_status` of a service created with an I/O cap includes its recorded limits under `settings.limits`, alongside any VM guest settings. This reports the create's requested configuration, not a live bandwidth measurement. Services created without I/O caps keep their existing status shape. Adopted services and functions are not covered: adoption does not apply these template fields, and `function_create` accepts only CPU and memory limits. In a capped service's status, `settings.limits` reports the configured caps; unset CPU and memory fields are omitted from this view. #### A script entrypoint and the Incus version Incus 7.6 refuses a line break anywhere in `oci.entrypoint`. From Incus 7.3 on, Incus splits the value as a shell would, quotes again each word that holds whitespace or starts with a quote, and hands the line to liblxc, which splits it once more: liblxc 6.0 and later honour a quote at the start of a word, while liblxc 5.0 and older (an Incus 7.6 build may link liblxc 4.0) split on spaces alone and keep the quotes inside the words. So the script of a shell, `["/bin/sh", "-ec",