# Changelog What changed in each release, in the terms an operator upgrading would care about. The published GitHub Release for a version carries the checksums, the SBOM and the verification steps; this file carries what the version *does*. Versions are `X.Y.Z` and are set by the release workflow, so the top entry names the version being prepared rather than the version in `Cargo.toml`. ## 0.4.62 ### Fixed - A `container:` job on a Linux boot service whose steps run as the image's own user (`runuser -u ci`, `USER` in the image) could not run a toolchain it installed with `setup-node`, `setup-go` or `setup-java`: `env: 'node': Permission denied`. Since 0.4.57 the default ACL on shared cache directories gave every account outside the `rm-runner` group nothing, so a folder created under `/__t` was `2770`, and the image's user is outside the group. Shared caches now let any account read and run what is in them, as a umask of `022` did before 0.4.57; only the pool's group may write. A cache shared by 0.4.57 to 0.4.61 is opened up at the next launch that uses it. ## 0.4.61 ### Added - Remote management through the coordinator (protocol version 5). A person signed in to the coordinator's dashboard can read a host's repositories, profiles and limits, and change them on a host whose operator allowed it there with `runner-manager coordinator remote-changes allow`. Reads are always answered; changes are off until allowed. A host runs only repository and capacity commands, never one that signs in, installs, updates or removes the service. Every command is recorded on both sides: in the coordinator's database (`GET /api/audit`) and in the host's `state/remote-commands.jsonl`. - `runner-manager coordinator login [--url URL]` signs the CLI in to a coordinator with GitHub: it prints a link and a code, the person approves the code in a browser signed in to the dashboard (the same `COORDINATOR_DASHBOARD_USERS` list), and the session (90 days) is kept in that account's own secret store, never the service's. `runner-manager coordinator logout` ends it on both sides, and `runner-manager coordinator hosts [--json]` lists the hosts with whether each is online, takes commands, and allows changes. - `--host remote:NAME` runs a command on the host the coordinator knows as NAME, as in `runner-manager --host remote:ivan-ms-a2 repo set-capacity octo/app --max-capacity 4`, with that host's own output and exit code. - `runner-manager repo list --json`: one versioned document of the host's capacity and every repository policy with its profile, labels, limit and workspace. - The coordinator serves `/auth/cli/*` for the CLI sign-in and accepts a CLI session as `Authorization: Bearer` on `/api/*`. Its database moves to schema 8 (schema 7 adds the command record; 8 marks each session as a browser's or the CLI's). ## 0.4.60 ### Changed - `runner-manager wsl install` moves a distribution first set up before 0.4.57 from `/root` to `/var/lib/runner-manager`, the tree every root service uses, through the same `service install --start-at boot` move: the old tree is kept, and its jobs then run under their own `rm-runner-N` accounts. - That move, from `wsl install` or `sudo runner-manager service install --start-at boot`, now waits for an idle host: with an attempt still active in the old tree's database, or a process still running from its runtime, the service is started again unchanged and the move happens on a later run. An operator's drop-in that names the old tree stops it too, with that file named, since the moved service could not see the directory it points to. ### Fixed - `deploy/coordinator/install-updater.sh` hands `.git` back to the account the update timer runs as when a command once run with sudo left part of it to root, which made every update fail at `git fetch`. `update.sh` says so when a fetch fails. - Linux and macOS hosts report their CPU load as the share of CPU time spent busy since the last heartbeat (`/proc/stat`, `HOST_CPU_LOAD_INFO`), 0 to 100, as Windows hosts already did. They reported the one-minute load average over the CPU count, a queue length that runs far past 100 on a host waiting on its disks, so the coordinator dashboard showed CPU above 100 % and placement could treat such a host as busier than it was. The dashboard caps an older host's figure at 100. ## 0.4.59 ### Fixed - A WSL host first set up before 0.4.57 starts again after `runner-manager wsl install` rewrites its systemd unit. Its directories are under `/root`, which the unit's `ProtectHome=tmpfs` hid, and systemd failed the unit before it started (`226/NAMESPACE`, "Failed to set up mount namespacing"). A boot unit now binds directories under `/root`, `/home` or `/run/user` back with `BindPaths=`, and the WSL update creates any that are missing before it replaces the unit. ## 0.4.58 ### Added - Coordinator web dashboard at the coordinator's own address: machines with CPU, memory, disk and runner slots, the queued, running and recently finished self-hosted jobs, pool queues and leases, and a week of history as charts. It works on phones and desktops, signs in with GitHub, and lets in only the logins in `COORDINATOR_DASHBOARD_USERS`. Off until `COORDINATOR_DASHBOARD_CLIENT_ID` and its secret are set. The coordinator's database moves to schema 6. - Each release publishes the coordinator images to GHCR (`ghcr.io/ivanmurzak/runner-manager-coordinator` and `-backup`), with build provenance attestations. - `deploy/coordinator/update.sh` and a systemd timer keep a coordinator on the newest release: every 15 minutes it backs up the database, moves to the new release, checks its health, and rolls back to the previous version and database if the new one does not come up. The dashboard shows its status. ## 0.4.57 ### Changed - Linux boot services run each concurrent runner as an account of its own, `rm-runner-1` to `rm-runner-32` in the group `rm-runner`, instead of every runner as `rm-runner`. A job can no longer read another job's workspace or its runner's credentials, read its processes' environment, or signal them. Dependency caches stay shared: each cache directory belongs to the group, is set-group-ID and carries a default ACL, so every runner can use and update it. Re-register with `sudo runner-manager service install --start-at boot` to create the accounts; until then runners keep sharing `rm-runner`. - Handing an attempt to its account and sharing caches with the pool walk the tree through directory descriptors and never follow a link at any depth, so a job cannot redirect those `root` changes by swapping a directory for a link mid-walk. ### Added - A Docker policy for native Linux jobs, checked by the per-attempt Docker proxy: host bind mounts outside the job's own attempt and caches, privileged containers and execs, added capabilities and devices, weakened security profiles, host namespaces, LAN networks, containers the job did not create, host-wide prunes and the swarm, plugin and secret APIs. It starts in `audit`, recording each violation without refusing it; `host docker-policy report` summarises them and `host docker-policy set-mode --mode enforce` refuses them. `host docker-policy allow-path` admits a host directory every job may mount. Podman's native API is reported as `unchecked_api`. - The Docker socket the GitHub runner mounts into `container:` jobs and service containers is now the job's proxy socket instead of the engine's own, so Docker inside job containers is accounted and checked like any other step. - `host protect-path add|remove|list`: host directories no runner or job may reach on Linux. The systemd unit lists them in `InaccessiblePaths=`, applied at the next `service install --start-at boot`, and the Docker proxy refuses a container mount of one in every policy mode, `audit` and `off` included. - Isolated execution works under a Linux boot service: the daemon reaches the system's rootful Podman service through its socket, and every isolated job runs in a container with its own user namespace (`--userns=auto`) and its own network namespace, so two jobs can listen on the same port without colliding or reaching each other. `host isolation prepare` sets up the subordinate id ranges and `podman.socket`; `host isolation set-disk-quota --required false` accepts storage that cannot enforce a quota. `host isolation status` now probes the OCI provider instead of always reporting it not installed. ## 0.4.56 ### Changed - Linux boot services run every job as a new system account, `rm-runner`, instead of as `root`. `service install --start-at boot` creates it, and the daemon keeps only the capabilities it needs to start runners as that account and to hand them their directories: `CAP_SETUID`, `CAP_SETGID`, `CAP_KILL`, `CAP_CHOWN`, `CAP_FOWNER` and `CAP_DAC_OVERRIDE`. A job runs with no capabilities at all. npm, `node-gyp` and `tar`, which failed as a capability-less `root` with `EPERM` changing the owner of extracted files, now work. Re-register existing services with `sudo runner-manager service install --start-at boot`. - The Git checkout cache is not offered to `rm-runner` jobs: preparing it ran `git` as the daemon over a checkout the job wrote. ## 0.4.55 ### Fixed - Linux services no longer set `RestrictSUIDSGID=yes`. Its seccomp filter answers `openat2` with `ENOSYS`, and GNU tar 1.35 extracts through `openat2` with no fallback, so every job that unpacked an archive, including the runner downloading an action, failed with `Cannot open: Function not implemented`. `NoNewPrivileges=yes` and the read-only filesystem outside the service's own `0700` directories still cover what it guarded against. `service status` reports existing units as outdated; re-register them with `sudo runner-manager service install --start-at boot`. - A Linux runner whose service cannot write its own `HOME` (a boot service's `/root`, which the unit makes read-only and hides) gets its own `HOME` inside its attempt directory, as a Windows runner already does. Actions that install into `~`, such as `oven-sh/setup-bun` (`mkdir '/root/.bun/bin'`), no longer fail. A login service keeps the operator's `HOME`. ## 0.4.54 ### Fixed - GitHub sign-in checks credential-store write permissions before requesting a device code. Boot-mode permission errors suggest the absolute native binary with `sudo`, so npm installations managed by nvm do not require Node on root's PATH. `auth logout` gives the same `sudo` command when it cannot remove a boot-mode credential. Every other remedy that says to run `sudo runner-manager …` (re-registering a drifted service, signing in again, granting workspace access, updating) also names the native binary whenever it is not on `sudo`'s `secure_path`. - On Linux, `root` keeps its data in `/var/lib/runner-manager` and `/var/log/runner-manager` instead of whichever home directory `sudo` left in `$HOME`, so a boot service no longer fails with `Permission denied` on an operator's private directories. Boot units keep no capabilities and hide every home directory with `ProtectHome=tmpfs`, so workflow jobs cannot read other accounts' files. `service install --start-at boot` refuses directories a boot service could not use, and re-running it moves an existing boot service's configuration and state into the system tree. The least-privilege review compares the capability bounding set as a set. - Service status includes systemd drop-ins and reports the account the service manager says the service runs as, instead of the account saved at installation time. Every manager now reports it in the same spelling, and a placeholder such as a raw SID falls back to the saved account. `ExecStart` prefixes such as `-` and `@` and `RestartSec` values with units such as `1min` are read correctly. - Applying the data directories no longer re-runs `chmod` on directories that already have mode `0700`, and a directory another account owns, such as one reached with `--data-dir` under `sudo`, is used as it is rather than failing the command. ## 0.4.53 ### Added - **Memory admission before every runner start.** The coordinator filtered hosts by memory only for the pool jobs it placed; a host's own demand (a repository, organization or selector label) was bounded only by `host_capacity` and `max_capacity`, so MacM6 (24 GiB) could start an 11.8 GiB Unreal build beside four 5.9 GiB App builds. Now the daemon starts a runner, whoever asked for it, only when free memory less what its running attempts may still grow into (each holds its estimate less what it uses now, until it ends) holds the runner's estimate plus a margin. The estimate is per profile, for the heaviest job its labels can take: the coordinator's figure for the target (from this host's own runs once it has any), else the p90 of the last 20 job peaks this host measured for the profile (or a heavier one of the newest three), else 1 GiB. A refused start is deferred, not failed: the job stays queued, the next pass asks again, `status` shows `waiting for memory: … needs X GiB (… estimate + … margin), Y GiB free less Z GiB reserved for running jobs` (and `memory` in `status --json`), and the log says `memory_deferred` with `free_bytes`, `reserved_bytes`, `estimate_bytes`, `estimate_source` and `margin_bytes`. With nothing running a runner always starts, however large (`memory_admitted_alone`). In a simulation of MacM6 with that queue the busiest moment went from 32.0 GiB to 17.4 GiB, the queue taking 16 ticks instead of 10. See [Keep heavy jobs from crowding one machine](README.md#keep-heavy-jobs-from-crowding-one-machine). - **`runner-manager host set-memory-margin SIZE|off|on`**: the free memory kept beyond the estimate (default `1GiB`, in `config/memory.toml`; a size needs a unit), or admission off. On by default; `host show` prints the setting and the free memory now. A running service applies a change to its next start. - **Coordinator: `memory_estimates` for each host's targets** (additive, version-4 links): after each `hello` and whenever they change, the heaviest job's expected peak per target, from that host's own runs once it has any (another OS's build measured elsewhere does not count). A host older than 0.4.53 skips the message; with an older coordinator the host estimates from its own measurements. - A heavy profile waiting for memory is served before lighter profiles that began waiting after it, so steady light work cannot keep it waiting for ever. - **Launch refusal `memory`.** A coordinator launch the host's memory cannot hold (counting the launches it accepted whose runners are not up yet) is refused with reason `memory`, and the coordinator places the job elsewhere; it keeps the host a candidate as for a capacity refusal. An older coordinator reads the reason as `other`. ### Changed - The host samples its runners' process trees every 2 seconds while memory admission is on, not only while joined to a coordinator, and so also concludes an exited runner's attempt within seconds without a coordinator. Turning admission off without a coordinator stops the sampling. ## 0.4.52 ### Added - **`runs-on: rm-pool-any`: a shared pool across operating systems.** A host of any OS may opt a profile in with `repo profile set-pool OWNER/REPO [--profile NAME] --pool on --any` (or `org set-pool ORG --pool on --any`; `--pool off --any` removes it), beside its own OS pool label on the same profile or on another. The coordinator's candidates for such a job are every connected host whose profile carries the label, whatever its OS, and its usual scores and statistics choose: in the simulation a fast Linux host and a slow Mac with more memory, mean run 466 s without statistics to 197 s once learned. Extra labels (`[rm-pool-any, heavy]`) narrow it as they do an OS pool. Only one profile of a target on a host may carry it. A host directs an `rm-pool-any` profile only when its coordinator says it places that pool; with an older coordinator the profile counts its own demand, so upgrade the coordinator first. See [Shared pools](docs/coordinator.md#shared-pools). - **A host's CPU load counts in placement.** Idle cores (cores less the last heartbeat's CPU load) add up to `cpu` points (weight 1, `COORDINATOR_PLACEMENT_WEIGHTS` `cpu=`), and a host at or above `COORDINATOR_CPU_BUSY_PERCENT` (90 %) is a last resort: tried after every other host that can take the job, never excluded. Simulated against a host 95 % busy with work of its own: round-robin started 30 jobs there over 3 seeds, placement none; mean run 348 s to 238 s. - **Free disk in every heartbeat, and a low-disk guard.** A host now measures free space on its runner root's volume at every heartbeat, not only at connect. A host under `COORDINATOR_MIN_DISK_FREE` (10 GiB) is a last resort, after a busy one. Simulated with a 4 GiB host: round-robin started 30 jobs there, placement none. - `stats` shows the **connected hosts** with their cores, CPU load, free memory, free disk and slots, and each placement score its `cpu` points, CPU load, free disk and last-resort reason; `--json` adds `connected` and those score fields. `runner-manager-coordinator hosts list` shows a connected host's CPU load and free disk, and its `--json` adds `connected`. `a pool queue was weighed` adds the chosen host's cores, CPU load, free disk and last-resort reason. All protocol changes are additive: an older host or coordinator ignores them and works as before, with one exception: a profile carrying both its OS pool label and `rm-pool-any` under a coordinator older than 0.4.52 serves its OS pool jobs from its own polling, up to 5 minutes late; keep the labels on separate profiles until the coordinator is upgraded. `stats --json` is schema version 4. `hosts list` says how old its connected-host figures are. ## 0.4.51 ### Fixed Placement between several hosts of one pool had only ever run with one host per pool. A deterministic simulation of the real placement code over several hosts (`crates/coordinator/tests/balance_sim.rs`, numbers are its own) found the following, each now fixed and asserted there. - **A heavy job no longer starts on a host too small for it because the big host's slots are all taken.** A job no host could hold is run anywhere, and only hosts with a free slot were asked. It now waits for the big host. Jobs already measured landing on a host too small, with heavy jobs in a queue of their own: 6 in 9 rounds to none. - **A long build's memory stays counted on its host until it ends.** The reservation lapsed 2 minutes after a job started, and a build whose peak comes later left its host reading free, so more builds landed there: up to 18 GiB past a 64 GiB host. It now counts against the host's total memory, through a broken link too, until the attempt's report, a heartbeat saying the host runs fewer jobs than are held for it, or 6 hours. - **A launch reserves the job its runner took**, once a webhook names it, rather than the queue's next heaviest when it was placed. A job taken by a runner whose launch was already gone now counts too. - **A job that turned heavy is seen at its next run**, not after four. What it is expected to need is its p90, or more if one of its 3 newest samples needed more. A job heavy from then on: on a host too small for its first four runs to its first only. - **Placement now finds out which host is faster.** A host that never ran a job other hosts have run scores as fastest for it until its first run finishes. The host with the most memory otherwise kept every job for good: a job twice as fast on a smaller host ran on the bigger one for ever. Mean run 595 s to 238 s. - **A warm workspace no longer beats a measured result.** A host measured over at least 3 runs as more than 10% slower than usual at a job loses its `cache` points for it, since its measured times already hold what the checkout saves. With a warm workspace on a slow host, mean run 178 s to 155 s by the sixth round. - **A failed job no longer reads as a fast one.** How long a job that did not succeed took is not kept as its duration: one failing at once made its host look the fastest there is. Its memory still counts. - **The biggest host idle and still too short for a job's estimate no longer holds that job for ever.** It runs on that host. With its estimate following its newest samples, one spike could otherwise stall it for good. ### Changed - **Heavy jobs wait for the memory they need.** A third 28 GiB job on a 64 GiB host now waits about 15 minutes for the first to finish rather than take the host 18 GiB past its memory, so such a burst takes longer to finish: about 1,800 s, from 1,400 to 1,950 s with memory overcommitted. Light jobs in the same queue wait with a heavy one, since a runner started for them could take it instead. Give heavy jobs a label of their own (`runs-on: [rm-pool-linux, heavy]`) to spare them: see [Weighted placement](docs/coordinator.md#weighted-placement). - `a pool queue was weighed` and `a pool queue is not placed` gain `committed_bytes`. ## 0.4.50 ### Fixed - **A stalled package download no longer holds a host's launches for good.** A download had no time limit, and every copy of the package waits for it. It now fails after 30 seconds without a connection or 60 seconds without a byte, however long a download that keeps sending takes. Its partial file is removed, the launches queued behind it get its failure and retry on their own back-off. - **A native runner's slot on Linux frees within about 10 seconds of its process exiting, not at the next scheduled poll (about 60 seconds).** The 2-second check read a runner that had exited as still running, because an exited process stays in the process table until the daemon collects it, and only a pass did that. Launches were refused with `max_capacity` for that long (live: 35 refusals in 90 seconds). A pass also no longer waits for a coordinator launch's bring-up. The host log says how long each took: `attempt_slot_released`, `exit_to_report_ms`. - **A coordinator launch starts at once, whatever its target is doing.** One accepted while its target's pass polled GitHub or brought up a batch of runners waited for that. Each target now serves its launches beside its passes, under the same host lock, journal and ceilings; a pass does not supervise a runner still coming up. - **Launches now reserve strictly oldest first across a host's targets and profiles**, in the order the host accepted them, rather than a younger one waiting for at most 2 seconds. Only the reservation is ordered, a lock hold and a journal write, so one slow package holds nothing up. An older launch that cannot reserve at all is passed over after 2 seconds without progress, once, and the log says so (`older_launch_passed_over`). `attempt_launch_slow` gains `order_wait_ms`. - **Jobs of the same name in two workflows no longer share statistics, when the workflow is known.** With the GitHub App, a job's key gains its run's workflow file, from the sweep or one request per run. Existing samples keep working under the job alone, with no migration, and hosts of any version are unaffected. See [Job statistics](docs/coordinator.md#job-statistics). ### Security - **An edit a job makes to the shared runner package no longer reaches later jobs' runners.** 0.4.49 shares a prepared package between attempts, and on Windows did not look at it again. Each launch's copy is now read in full against what the package's digest-verified archive holds, outside the host lock. One that differs is removed and not used, and a shared package that differs too is prepared again (`package_copy_differs`, `package_verify_failed`). Windows now downloads the package at the first launch after a daemon starts, as Linux and macOS already did. Jobs on one account, and above all a boot-mode host whose jobs run as root or LocalSystem, are still not isolated: see [what this does and does not guarantee](docs/coordinator.md#shared-runner-packages). ### Added - **The coordinator's log says why a host does not fit a job.** `a pool queue was weighed` and `no_memory_fit` carry `free_ram_bytes`, `reserved_bytes`, `ram_margin_bytes`, `required_bytes` and `total_ram_bytes` for the candidate nearest to fitting. The fit rule itself is unchanged. - **A daily backup of the coordinator's database.** The compose file gains a `backup` service keeping the last 7 consistent copies; `docker compose up -d --build` starts it. To keep them on another disk, and to restore, see [Back up and restore](docs/coordinator.md#back-up-and-restore). ## 0.4.49 ### Fixed - **Runners of one host no longer start one at a time.** The host-wide allocation lock was held for a runner's whole start, package copy included, so one slow package held every other target back (live: 143 s, with 40 other launches deferred). The lock now covers only the reservation, which is journalled: every ceiling holds as before, and a daemon that dies before the runner is up leaves an attempt that startup recovery concludes. Runners then prepare at the same time, a quarter of the host's cores at once and at least two, and register in parallel. On a WSL guest the launch fence is shared by the launches in progress: a Windows drain request stops new ones and waits for those. - **A host serving two targets no longer downloads the runner package at every switch between them.** Each target kept its own record of the package tree it trusted. One target's reinstall replaced the tree the other had recorded, which then reinstalled in turn. The record is now shared. The first launch after a daemon starts still downloads the package once, by design. - **A launch being worked on is no longer dropped.** The 45-second limit dropped launches whose runner was waiting for a turn. It now counts from a start that failed for a real reason. A launch whose runner is coming up keeps its lease past its lifetime, for up to 15 minutes, because the heartbeat that lists it now renews the lease at the coordinator. So does one waiting for its target's loop to finish bringing up other runners of its profile. A launch whose runner is up and took no job expires as before, and one whose runner failed coming up is retried within the 45 seconds. Upgrade the coordinator first: an older one does not renew a lease, so a slow runner's job can be placed a second time, leaving a surplus runner. - **Launches start oldest first across the targets of a host.** A younger launch lets an older one reserve first, for at most 2 seconds, and runners take their turns to prepare in the order they were reserved. ### Added - `attempt_launch_slow` and its siblings in the host log say where a start went: `turn_wait_ms`, `package_wait_ms`, `package_install_ms`, `package_verify_ms`, `package_copy_ms`, `package_downloaded` and `prepare_ms`, beside the existing fields. ## 0.4.48 ### Fixed - **A job's statistics now apply to its next run, whatever the run is called.** A job's key included GitHub's `workflow_name`, which is the run's title, so a workflow with a `run-name` got a new key at every dispatch. Its samples were never found, and almost every queue was estimated at its repository's p90 (104 of 106 placements in a live test). The key is now `owner/repo/job`, and existing samples are rekeyed on upgrade, as a best guess where a run title or job name holds a `/`. Jobs with the same name in two workflows of one repository now share their statistics. - **A job that ended stops reserving memory.** A lease's estimate stayed reserved on its host for 2 minutes after its runner took a job, even if the job had already finished, so a run of short jobs could keep a host "full" for the next ones. It is released as soon as the job ends. - **A launch that cannot start no longer holds its job for 2 minutes.** A host counts its other repositories' open launches against its slots, and refuses a launch it has no slot for. It drops one it can no longer start, or that has no runner after 45 s for any reason, and tells the coordinator at once, which places the job again. Its heartbeat's free slots leave out those its waiting launches have spoken for. Two launches can no longer share one runner. - **A launch outlives a runner that took nothing.** When the idle runner serving a launch exits without a job, the host starts another or hands it to another waiting runner, and the lease stays open. - **The lease list in a heartbeat is current.** In 0.4.47 a target with nothing running could list a launch that had already ended until its next poll. - **A queue no longer reserves its heaviest job for every runner.** Each lease reserves the queue's next heaviest job, so one heavy job among light ones counts once. A host must still hold the heaviest job, since a runner can take any of them. - **A pool job is weighed by its own memory use.** A job only the coordinator's sweep had seen counted as its repository's p90, so a light job could be estimated at a heavy neighbour's memory and passed over by a host with room for it. The sweep now names the jobs it lists, as a webhook does. - **A repository just added is covered within seconds, not 10 minutes.** The coordinator reads its GitHub App's repositories again (at most every 30 s) when a host advertises one they do not include, or a webhook names one. Until now such a repository stayed in `fallback`: its pool jobs were only woken, and waited for the host's next poll. - **`repo add` no longer refuses a repository the coordinator will poll for you.** The REST budget (also in `host show`, `host set-poll-interval` and the TUI) counts a coordinated target at the coordinated interval instead of 60 s. A new repository (not a new organization) on a host whose link is up and whose coordinator runs a sweep is counted the same way, with a printed note. If coordination is lost, a set that no longer fits stretches its charged polls to the interval `repo add` would allow (64 s for 11 repositories) and keeps starting runners. `host set-poll-interval` now also refuses a shorter `--coordinated` that would go over the budget. ### Changed - **Only a job's own samples can keep it off a host.** A job never measured used to be held back by its repository's p90, so one heavy job could block every new job of that repository. Now such a job is only ranked by that figure, and reserves at most `COORDINATOR_DEFAULT_JOB_RAM`. It may land on a host too small for what it turns out to need, once: from its first sample it is filtered by its own figure. ### Added - The coordinator's `a pool queue was weighed` log line says how many of the queue's jobs are named, measured and unnamed. ### Upgrade notes - The coordinator's database moves to schema 5: sample keys lose the run's title. To roll back, stop the service and run `sqlite3 coordinator.db 'PRAGMA user_version = 4'`; 0.4.47 then starts new keys beside these. - The launch fixes are on the host, so hosts need 0.4.48 too. An older host keeps working: only the coordinator's fixes apply to it. ## 0.4.47 ### Fixed - **Coordinator statistics show a job's real run time.** A job's duration in `coordinator stats` is now GitHub's own `started_at` to `completed_at`, so a 14-second job reads 14 s instead of the 60-80 s its runner lived (start-up, waiting for the job and cleanup included). Only when GitHub's `completed` webhook never arrives (within 2-3 minutes) does a sample fall back to the host's measured time. Peak memory and CPU are measured by the host as before. Samples recorded by 0.4.45-0.4.46 keep their old durations until newer runs replace them (the last 30 per job). - **A freed slot is used at once.** When a runner ends, the host tells the coordinator its free slots right away instead of at its next heartbeat (up to 15 s later), and the coordinator places a waiting pool job on it immediately. A joined host also notices within about 2 seconds that a runner's process has exited and frees its slot then, instead of at the target's next poll (up to 60 s). Cleanup of the runner's files never held the slot; a container, VM or WSL runner still keeps it until its environment is proven gone, as before, and its host reports the free slot as soon as that cleanup is done. - **Hosts reconnect within 30 seconds after a coordinator restart.** The link's reconnect wait now stops at 30 s, random spread included (it could reach 90 s, which added up to minutes), and starts over after a connection that stayed up for over a minute. A refused key or version still waits up to 5 minutes. - **A pool job no longer waits minutes behind a launch that is not coming.** A lease (a launch the coordinator sent to a host) used to count for 5 minutes even after its host had dropped it, for example after the link reconnected, or after another runner of that host took the job. The coordinator now closes it, and places the job again at once, when the host connects again, when the host's heartbeat no longer lists it, or when another runner of the same host takes a job of that queue. A lease lasts 2 minutes at most by default, set with `COORDINATOR_LEASE_TTL` (`--lease-ttl`, at least 30 s). - **Adding or removing a repository on a host no longer interrupts the others.** `repo add`, `repo remove` or enabling or disabling a profile used to make the daemon finish every running job, stop every target and drop its coordinator link and accepted launches before reloading. Now only the changed repository's loop starts or winds down, within 30 seconds. The link, its launches and every other repository keep running. - **Accepted launches start together, at once.** Each launch a host accepted used to wait for its own pass, 5 seconds after the previous one (6 launches took 25 s more to start). Now every launch open at that moment is started in one go, without polling GitHub. The runners are still created one after another under the host's allocation lock, so every capacity ceiling holds. - A test of the runner process fixture could read its readiness file before it was written and fail at random. ### Added - **Queue wait in `coordinator stats`.** How long jobs waited from being queued to starting on a runner (GitHub's `created_at` to `started_at`), as p50 and p90 over all samples and per job: the figure that shows whether jobs wait for a free host. - **Why a pool job waits.** `coordinator stats` lists the pool queues with fewer runners on their way than jobs queued, the oldest job's wait, and the reason: no host serves it, unfit or draining, no free slot, not enough memory, refused by a host (with the host's reason), or no answer. It also lists the open leases per host with their age. The coordinator logs the same reason, at most once a minute per queue. ### Upgrade notes - `coordinator stats --json` (on a host and on the coordinator) is schema version 3: `p50_duration_secs` is the job's run time, and `p50_queue_wait_secs` and `p90_queue_wait_secs` are new, overall and per job, as are `pending`, `leases` and `live_at`. - The coordinator's database moves to schema 4 (one new column, one new table). To roll back to 0.4.46, stop the service and run `sqlite3 coordinator.db 'PRAGMA user_version = 2'`. - The coordinator protocol stays at version 4 and every change is additive: 0.4.47 hosts and coordinators work with 0.4.45-0.4.46 ones. An older host still gets prompt placement once its next heartbeat shows a free slot; an older coordinator ignores the extra heartbeat and its lease list. An older host's leases are still closed when it reconnects or another of its runners takes the job, and otherwise expire as before. ## 0.4.46 ### Fixed - 0.4.45 reached the GitHub Release and the install scripts, but not the `runner-manager` crate on crates.io, npm or Homebrew; 0.4.46 publishes the same coordinator features to every channel. Nothing else changed: the coordinator protocol and every command behave exactly as in 0.4.45. - Internal: the coordinator wire protocol now lives in the already-published `runner-manager-domain` crate, and a test fails any pull request whose crates.io publish list misses a crate `runner-manager` depends on. ## 0.4.45 ### Added - **An optional coordinator.** `runner-manager-coordinator` is a small self-hosted server (container image, Caddy example and `docs/coordinator.md` included) that hosts can join so they poll as soon as a job is queued instead of at their next interval. Each host still decides under its own limits and GitHub credential. Without a coordinator, or while it is unreachable, a host behaves exactly as in 0.4.44. Protocol and database notes: `docs/coordinator.md`. - `runner-manager coordinator join URL --code CODE`, `coordinator status [--json]` and `coordinator leave`. A host authenticates with an Ed25519 key created at `join` and stored beside the GitHub credential, enrolling with a one-time code; `hosts revoke` on the server closes its live connection within a second. - **Webhook wake.** The coordinator receives GitHub's `workflow_job` webhooks (HMAC-verified) and wakes the hosts that serve the job's repository and labels. A woken target is polled at most once every 5 seconds and never inside a rate-limit back-off. - **Reconciliation sweep.** Given a personal GitHub App (`COORDINATOR_GITHUB_APP_ID`, `COORDINATOR_GITHUB_APP_PRIVATE_KEY_FILE`), the coordinator reads the queues of the repositories its hosts serve every 2 minutes with conditional requests, and wakes hosts for jobs whose webhook was lost. Without the App, every target stays in `fallback`. - **Poll modes per target:** `standalone` (no coordinator), `coordinated` (link up and the target covered by the sweep: polled every 300 s while none of this host's runners runs for it, `host set-poll-interval --coordinated`, at least 60 s) and `fallback` (link down for 30 s or target not covered: the normal intervals, at once). `status`, `host doctor` (`github.polling`, `coordinator.link`), `coordinator status` and the TUI show the mode and the link. - **Shared pools.** `repo profile set-pool OWNER/REPO [--profile NAME] --pool on|off` and `org set-pool ORG --pool on|off` give a profile this host's pool label (`rm-pool-windows`, `rm-pool-linux` or `rm-pool-macos`). For a job with `runs-on: rm-pool-…`, the coordinator sends exactly one host a `launch`; the host still applies every local ceiling and may refuse, and the coordinator then tries the next one. In `fallback` and `standalone` a pool profile serves its own demand as before. A second pool profile on one target, another OS's pool label or an unknown `rm-pool-` label is refused; `profile list` and `profile show` print `pool=`. - **Job weight statistics and weighted placement.** A joined host samples each native runner's process tree (peak memory, CPU, duration) and reports it; the coordinator keeps the last 30 samples per `repository/workflow/job` for up to 90 days and places pool jobs by memory fit, warm workspace, headroom and each host's past speed (`COORDINATOR_PLACEMENT_WEIGHTS`). With no statistics the order is warm workspace, then free memory, then fewest busy slots. `runner-manager-coordinator stats` and `runner-manager coordinator stats` show the heaviest jobs and the last placement decisions. - Heartbeats report free memory on macOS and CPU load on Windows. ## 0.4.44 ### Added - Daily release checks in the TUI and background daemon, coordinated across processes and persisted across restarts. Automatic updates default to enabled for both new and existing installations. Host settings and `host set-auto-update true|false` control installation; checks continue when disabled. - Automatic updates reuse the installation channel and checksum validation. Background services drain active jobs before restarting, and the TUI restores its terminal and relaunches with the original arguments. Windows, macOS, Linux and WSL share the scheduler. Update results are saved in `state/update-status.txt`. ## 0.4.43 ### Fixes - Changes to `host set-capacity` take effect without restarting the service, including on WSL hosts. All projects share the current host ceiling, which is checked again under the allocation lock before each new runner attempt. Lowering the limit preserves existing attempts and waits for free capacity. - Missing host records and unreadable host settings block new runners and report their own diagnostic reasons. An unreadable local journal no longer makes a busy host switch to the faster idle polling interval (#113). ## 0.4.42 ### Added - Native repository runners can reuse an independent local Git object cache, including across ephemeral attempts and profiles with different workspace directories. Enable it in Host/Repository Settings or with `repo cache set-tool OWNER/REPO git --state on`. It is off by default and supports standard GitHub HTTPS checkout; the workflow still fetches its exact ref/commit. Existing dependency-cache roots, limits and pruning apply. - Native Linux/WSL runners account for temporary Docker resources through a per-attempt local proxy. The default `report` mode does not delete resources; `host docker-cleanup set-mode --mode apply` enables cleanup after an attempt is cleaned. Named volumes, build caches, images and unrelated resources are preserved; no global prune is used. ### Fixes - Completed-attempt summaries expire after 30 days and retain at most 1,000 files. Rotating service logs retain at most 30 files per stem. Dependency caches and persistent workspaces are excluded from diagnostics cleanup. - Owned Windows Hyper-V container cleanup also removes anonymous volumes. - Unix file locks release immediately when their owning guard is dropped, even while a subprocess temporarily holds a duplicate descriptor. - Git cache publication keeps concurrent shallow histories, excludes stale refs and credentials from new workspaces, and stages checkout copies outside retained workspaces for crash cleanup. Oversized merged shallow metadata is refused before replacing the last readable copy. - Docker accounting journals enforce their reader's size limit before atomic replacement and bound reads even if a file grows during inspection. ## 0.4.41 ### Fixes - Autoscaling starts additional runners for queued jobs even when other runners are busy. Busy runners still consume host and policy capacity, but only unassigned runners that are preparing, starting, or idle cover queued demand. This prevents the last few jobs of a parallel workflow from staying queued while capacity is available (#108). - `update --force` recognizes the application's `X.Y.Z+git.` source-build identity when upgrading an installed service to a newer published release. Version comparisons still prevent downgrades and same-version replacements with different bytes; published release versions remain strictly `X.Y.Z`. ## 0.4.40 ### Fixes - `runner-manager update` on a Windows copy installed with npm no longer says npm is not on the PATH when it is. Rust's process API looks only for `npm.exe`, and npm ships as `npm.cmd`, so `update` now searches the PATH with each `PATHEXT` extension the way `cmd.exe` does and runs the `npm.cmd` it finds. Unlike `cmd.exe` it skips the current directory and relative PATH entries, and tries only `.exe`, `.com`, `.cmd` and `.bat`. `host doctor`'s tool checks use the same search. The standard library escapes the arguments for `cmd.exe`, so no command line is assembled by hand. - `update` on an npm copy passes `--prefix` with the prefix that copy lives in, so it updates the copy you ran and not the global prefix of the account running it. Seen on a machine with two accounts, each with its own copy under `%APPDATA%\npm`, where the machine PATH lists the first account's copy first. Afterwards `update` runs the updated binary and says which version it now reports. A copy that is not a global install (one in a project's `node_modules`, or a dependency of another package) is refused with the npm command to run, instead of having a second copy installed somewhere else. - When the newest release is already installed but the service still runs an older build (for example after `npm install --global` was run by hand), `update` no longer says "Nothing to do". It says the service still runs the older build and starts the same job-safe swap that follows an update: the service finishes every running job, then restarts on the new build. It does this by giving the file the service watches a new modification time; the file's contents are not changed. If the service is not running, `update` says so and names the command that starts it, after which it swaps by itself. When the service was installed from another copy, `update` names `runner-manager update --force`, which hands this copy over without interrupting a job. After an update the same case used to name `service install`, which refuses while the service runs. A registration from before the service ran its own copy is still told to re-register with `service install`. `--check` reports what would happen and changes nothing. - The advice for a service that has stopped making progress, in `status`, `service status` and `status --json`, named `runner-manager service stop` and `service start`, which do not exist. It now names the service manager's own restart command for the service's start mode: `Restart-Service` or `schtasks` on Windows, `launchctl kickstart` on macOS, and `systemctl restart` on Linux. The Windows install script's advice for a binary it cannot replace no longer names `service stop` either. ## 0.4.39 ### Fixes - Native runners start with `POWERSHELL_TELEMETRY_OPTOUT=1` on every platform, so a job's PowerShell step shells on Windows no longer hang at startup while another job runs. The cause is a telemetry lock in PowerShell 7.6.6 and earlier, which the first `pwsh` of each job takes in its fresh profile and keeps until it exits, while the first `pwsh` of every other job waits for it with no timeout. Seen as several `pwsh` step shells holding their runners for two hours with no output and almost no CPU. The opt-out makes PowerShell skip that code. macOS and Linux get it too, as a precaution. - A non-empty value the service's own environment already has for `POWERSHELL_TELEMETRY_OPTOUT` is left as it is, and `runner.env` overrides it like any other default: `runner-manager host env set POWERSHELL_TELEMETRY_OPTOUT=0` turns PowerShell telemetry back on. `host env show` lists it. - The 0.4.31 note that .NET known-folder APIs still report the service account's profile was not true of local application data, which follows `USERPROFILE`; the README and `host env show` now say so. ## 0.4.38 ### Features - A repository with nothing happening is polled for queued jobs every 10 seconds instead of every 60, so a job that arrives on an idle host gets its runner up to 50 seconds sooner. The polls are conditional requests: the service keeps each listing's `ETag`, sends it back with `If-None-Match`, and GitHub answers `304 Not Modified` while nothing changed. GitHub does not count a `304` against the hourly rate limit, so idle polling is free against it. A repository with a runner up or runs in progress, or a job this host matched but has not started, is still polled every 60 seconds. - `runner-manager host set-poll-interval --idle 10s --active 60s` sets both intervals. The idle interval is at least 5 seconds and the active one at least 30 (the active one is the existing refresh interval). A running service reads them again every 15 seconds and each target adopts them at its next poll, without a restart. An organization target too large to poll that often stretches its own idle interval to stay under 300 requests a minute. - A repository covered by two targets on one host, such as an organization and a repository inside it, is read once for both within the idle interval, and every target shares one set of `ETag`s. - The hourly rate limit is your GitHub user's, shared by every host signed in as you. Every response carries what is left of it, and the service now slows down on it: under a fifth left, it polls at twice the active interval, under a twentieth at four times, never past the hour's reset. A secondary rate limit (`429`, or a `403` saying so) now stops every poll on the host, not only the one that met it, until the `retry-after` GitHub sent, or a minute without one, doubling each time it comes back after a wait has ended. Answers to one limit that arrive together count once, and the loops wake a few seconds apart rather than all at once. - `host show` and `status` print what polling actually cost, measured by the service: full (`200`) and `304` responses per hour and the share that were `304`, the busiest minute against GitHub's 900-a-minute secondary limit, what is left of the hourly limit, and whether polling is throttled and why. The old projection is still printed, as the cost if no poll were answered `304`. `status --json` carries the measurement under `polling`. A new `host doctor` check, `github.polling`, reports the same and never blocks runners. ### Fixes - A second account on a machine whose service another account installed is told so, and is no longer told to break it. A Windows host's boot service had been installed by one administrator, and another account that had once run a login-mode install opened `runner-manager tui`: RUNNER READINESS: BLOCKED, a "STALE" service binary, a start mode "mismatch", a secret store "MISMATCH" and a login-only warning, with `service uninstall`, `service install` and `auth login` as the fixes. All of it compared that account's abandoned 0.4.30 record against the other installation's registration. Followed from that account, those fixes would have stopped, replaced or broken a service that was running jobs. The service manager's registration names the data directories the service runs against, and when they are not this account's, `service status`, `status`, `host doctor` and the TUI now report the service as the other installation's: its state, its account, the data root it was installed from and whose that is. Where this account can read that installation's records, the report shows that installation's own problems, each with the account to fix it from, and none of the owner's remedies, since those act on the directories of whoever runs them. Where it cannot, it says that, and to look from an elevated terminal or as that account, instead of guessing. No reinstall or sign-in is suggested for it. The TUI shows UNKNOWN rather than BLOCKED when it cannot read the service, and notes that its counts are this account's own configuration, which the service does not run. `status` never runs the other installation's binary, so its `service_binary_version` is `null`. `status --json` carries the other installation's data root as `product.service_installed_from`, and the `host doctor --json` document carries it as `service_installed_from`. `host doctor` reads that installation's own records for its service checks. The same holds on macOS and Linux for a root-owned boot service seen by a user with a leftover login record. A registration made before the directory options existed names no directories, so it is still judged as this account's own. - `service install`, `service uninstall`, the TUI's start-mode switch, and the service start and stop used by updates refuse another installation's registration. They name the account it has to be changed from, where they used to replace, remove or restart it with this account's directories. - `runner-manager host forget-local` moves an account's unused configuration (its old install record, database, host row and policies) into a `forgotten-