# Changelog All notable changes to jobd. Format roughly follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] ## [0.5.57] — 2026-11-09 ### Fixed - **Queue explanations now match dispatch on the GPU opt-out.** v0.5.56 kept VRAM requests and `any-gpu` pins off a `gpu: false` worker at dispatch. The views that explain a queue still counted that worker as eligible, because `eligible_workers` and `submit_preflight` never saw the job's VRAM. So a VRAM job pinned to an opted-out host was reported as "will queue behind job N" and then as "the only eligible worker is busy", although it could never run there. One shared predicate now drives dispatch, `eligible_workers`, `selectors_only_match` and `submit_preflight`. Submit says plainly that every matching worker has GPU work switched off, and `vram_gb` is now a required argument, so no caller can skip the check. ## [0.5.56] — 2026-11-08 ### Fixed - **A worker set to `gpu: false` no longer gets GPU work through a side door.** The `--gpu` selector already respected the opt-out, but two other routes ignored it. A `--vram-required N` job without `--gpu` only compared free VRAM, which made the opted-out host the best fit when it had the most free VRAM. A job with `--host any-gpu` matched the alias, which NVML presence sets regardless of the opt-out. The matcher now reads any VRAM request (explicit, or a `cuda-Ngb` tag) and any `any-gpu` pin as a GPU request (`explain_skip` reason `no_gpu`), as `adopt` already did for `vram_gb > 0`. - **`update-worker.sh` no longer strands a staged upgrade.** When a job was claimed between the install and the restart, the new version stayed in the venv while the old process kept running, and every later tick read "installed == broker" as "already on" and never restarted (gt76 sat on 0.5.54 under a 0.5.55 broker). "Done" now means the worker *process* reports the broker's version in its heartbeat; a staged version is restarted onto at the next idle tick, and the post-restart check reads that heartbeat instead of the venv. The idle gate still counts job rows, never the `/workers` gauge. ## [0.5.55] — 2026-11-08 ### Fixed - **A stale report from an earlier run of the same job on the same host is refused.** The broker fenced worker reports (`/log`, `/started`, `/complete`, `/refuse-admission`) by hostname only, so when a job was requeued and the same host claimed it again (a frozen worker that thawed), the first run's `/complete` was accepted into the re-run and recorded its outcome. Every claim now bumps a `dispatch_seq` (returned in the job), the worker sends it back as `X-Jobd-Dispatch`, and a mismatch gets 409. A resurrected orphan keeps its seq. A worker that sends no header (older than this release) gets the hostname check alone, as before. - **A job that closes its stdout can still be cancelled and still hits `max_wall_s`.** The worker stopped polling for signals and running the wall-clock watchdog at stdout EOF, so a workload that redirects its own output (`exec >log 2>&1`, a daemon that closes its fds) could not be cancelled and had no wall limit while it kept running. Polling now stops when the process exits. After EOF the idle and first-output watchdogs stand down, since a closed stdout can never produce output. ## [0.5.54] — 2026-11-08 ### Fixed - **A `depends_on_any_exit` child no longer starts while its parent may still be running.** A parent orphaned as `worker_died`/`worker_restarted` is terminal, but its worker may only have been frozen, and the next heartbeat resurrects it to RUNNING; the any-exit child had already dispatched on the orphan and ran beside it. Such an orphan now releases any-exit children only once it settles: its worker heartbeats without reporting the job, or `JOBD_ORPHAN_SETTLE_S` (default 1800) passes. Default-policy children are unchanged (cascade-cancelled, restored on resurrect). - **`job submit --count N --wait` waits on every member.** It exited on the first member's verdict (the shared wait helper called `sys.exit`), so members 1..N-1 were never waited on and `job submit --count 3 --wait ... && next` ran `next` with a later member failed or still running. It now waits on all of them and exits 1 if any did not complete, as `job status A` does. - **A job killed by a signal exits 128+N from `job wait` / `submit --wait` / `logs -f` / `status`.** The worker records a signal death as a negative exit code (-15 for SIGTERM), which the CLI passed to `sys.exit` and the shell read as 241. It is now the shell's convention, 143. ## [0.5.53] — 2026-11-07 ### Fixed - **A cancel on a job orphaned as `worker_died`/`worker_restarted` is no longer dropped.** The cancel route treated it as terminal: 200, no signal. If the frozen worker thawed, the heartbeat resurrected the job with nothing to poll and it ran to completion. The route now stamps `signal=cancel` on such a resurrectable orphan (one write, `_arm_cancel_signal`), so a resurrect hands the worker the cancel and the job ends `cancelled`. MCP `jobd_cancel` reports `signal_sent="cancel"` for it. - **`job wait`, `job submit --wait`, `job logs -f` and `job status` exit nonzero for every terminal state but `completed`.** They used to exit `exit_code or 0`, and several terminal writers leave `exit_code` NULL (sweeper `orphaned`, `scheduling_timeout`, a queued cancel, `cwd_unreachable`, a drain-refused `preempted`), so `job submit --wait ... && next` ran `next` after a job that never succeeded. One mapping, `exit_for(state, exit_code)`, now serves every call site: `completed` is 0, a real nonzero `exit_code` passes through, anything else is 1 (including `cancelled`, `preempted`, and an adopted job's `orphaned`). The wait also reads its verdict from the SSE `terminal` event only (a log line printing `{"state": ...}` no longer ends it), and a stream that ends with no terminal event exits 1. ## [0.5.52] — 2026-10-19 ### Security - **pyjwt 2.13.0 -> 2.15.1 in `uv.lock`** (10 advisories fixed in 2.14.0); this reaches the container image, not the PyPI dependency range (#160). ### Changed - **README: "Does it help?"** A short section gives the agent benchmark's result: accuracy is not separated (both tool setups at 41 of 41), and the server's median answer is about twice as fast. The benchmark README is shorter and now starts with the answer, includes an Invalid column, and corrects the q26 sizes to what Claude received (all 194 failures, 41,808 characters). The perf ratchet's cross-version note moved to `benchmarks/perf/ratchet.py`. - **Agent benchmark re-run (2026-10-18).** A harder question set (38 questions from the documented use cases, `build_hard.py`) was pre-registered and run against jobd-mcp at an unreleased commit and the `job` CLI: a tie on accuracy (38 vs 37 questions right in all runs, p = 1), with the server cheaper and faster on every question (median cost ratio 0.20, time ratio 0.16). `run.py` gains `--server-spec`, `--exclude-newer`, a per-run TMPDIR, a usage-limit gate and a re-run-once rule for API failures; `analyze.py` gains the pre-registered win rule; `diagnose.py` breaks down where each arm's tokens go. - **README is less than half as long.** "Does it help?" now shows the benchmark chart, re-rendered narrower (8.5 in, one-line subtitle) so its text is legible at README width; data and counts are unchanged. Job arrays, `--explain`, `job adopt`, the full `job submit` synopsis and the pueue/task-spooler command map moved to `docs/cli.md`; concurrency (multislotting) to `docs/concurrency.md`; the architecture diagram's mermaid source to `docs/ARCHITECTURE.md`; the self-update network note to the runbook. The README's retention section, which said job logs are kept forever, is replaced by a pointer to the runbook (logs are pruned after 60 days by default). ### Added - **Benchmarks: an agent A/B and a work-count perf ratchet.** `benchmarks/agent_qa/` asks Claude 41 read-only questions about a live broker (exit codes, hosts, run times, a project's failures on a day, log lines, routing) with no tools, with the `job` CLI, and with jobd-mcp. The key is drawn from the broker's HTTP API without importing `jobd`. Every setup reaches the broker through a loopback proxy that refuses anything but reads and dry-run submits, and the CLI setup's `job` refuses mutating subcommands. `benchmarks/perf/` counts the work per broker endpoint, scheduling pass and MCP tool on a fixed in-process fleet: jobd function calls, matcher calls, SQL statements and broker HTTP requests. `tests/test_perf_ratchet.py` fails on any difference from `baseline.json`, in either direction. The counts are identical on Python 3.11 to 3.14. Controls: `analyze.py` scores two no-model baselines (a constant guess, 4 of 41; the best constant fit to the key, 10 of 41) through the same scorer, applies one validity rule to every setup, scans every transcript for leaks of the key or the benchmark's files, and gives bootstrap intervals and paired tests. `run.py` refuses to start unless the key's own answers score right and near misses score wrong, and unless the broker and both pinned packages are the versions the key assumes. `ratchet.py --update` refuses a changed workload output without `--allow-witness-change`, as it refuses a rise without `--allow-increase`. - **`jobd_list` finished_at window.** `finished_after` / `finished_before` (ISO-8601, or relative `2h`/`3d`/`1w`) restrict the list to jobs that finished in a window; `counts` then cover the window exactly, every terminal state included. The filter runs in jobd-mcp over the broker's existing pagination, so no broker upgrade is needed. Rows now carry `finished_at` and `duration_s`. A bad timestamp is reported as `invalid_arguments`. The server's instructions now say the client lists each tool under its prefix (`mcp__jobd__jobd_list`). ## [0.5.51] — 2026-09-26 ### Changed - **A `/mnt/c/` working directory now needs a pin to one host, whichever host it is.** The broker used to accept such a job only when pinned to one of three hard-coded host names, two of which no longer matched any worker, and it refused every other WSL host even though each has its own `/mnt/c`. The rule is now: `any`, or a pin that matches more than one known worker (a pool alias such as `any-gpu`), is refused with a 400; a pin naming a single host is accepted. A pin that matches no known worker is left to the routing checks. ## [0.5.50] — 2026-09-26 ### Fixed - **README claims now match the code.** The default bind is `127.0.0.1` and the runtime control is the tailnet source-IP check (the `JOBD_HOST` rule is a CI lint on the Docker deployment, not enforced by the broker); the optional self-update scripts do reach PyPI, the GitHub API and GHCR; preemption gives a grace window and a `JOBD_CHECKPOINT_DIR`, and resuming means resubmitting; VRAM routing tracks GPU index 0 only, and the multislot `free_vram` formula and the `# CONCURRENT_OK` opt-out are spelled out; `job submit --stdin` and MCP `jobd_submit` run their command through `bash -c`; the CI-tested marks say CI has no GPU runner and skips systemd-scope tests; `JOBD_CONFIG_DIR` defaults to the Docker path and `config/` is not in the wheel. The comparison table separates Modal (cloud-only) from SkyPilot/dstack and no longer claims Ray cannot run arbitrary commands. - **`server.json` declares `JOBD_API_TOKEN`** as an optional secret input for the MCP server. - **Example hooks:** `jobd-block-gpu.sh` suggests the real `--vram-required` flag (it named a nonexistent `--vram-gb`); `jobd-nudge.sh` drops its site-specific wrapper/host rules for two opt-in variables, `JOBD_NUDGE_WRAPPERS` and `JOBD_NUDGE_SSH_HOST_PAT`. ### Removed - **`docs/plans/` is gone.** The six internal design, plan and deploy notes are removed from the tree (they remain in git history); the SIGTERM-drain phases the code comments refer to are now described in `docs/runbook.md`, "Drain / restart a worker". ## [0.5.49] — 2026-09-24 ### Fixed - **A GPU with a resident non-jobd model is no longer closed to all GPU work (#144).** The matcher subtracted foreign-process VRAM from `free_vram_gb`, which is NVML's device-wide free and already excludes it, so the same bytes were counted twice: a card with an ollama model holding 22.9 of 32 GB refused a 4 GB job with 8.5 GB genuinely free. Placement, `explain_skip` and the submit-time contention warning now use free VRAM minus the 1 GB safety margin, matching the worker's own admission gate. A card a foreign process has filled still drops out (its free VRAM is below the 2 GB implicit floor), and `unregistered_vram_gb` stays in the worker ad and the warning text for diagnosis. ## [0.5.48] — 2026-09-23 ### Fixed - **`jobd_status` and `GET /jobs/{id}` report a pending cancel/preempt signal.** `JobInfo` omitted the `signal` column, and jobd-mcp filled the gap with `signal: null` for every job, so a job about to be SIGTERMed read as having nothing pending. `JobInfo.signal` now carries the broker's value (`"cancel"`, `"preempt"` or `null`), and jobd-mcp passes it through instead of inventing it. ## [0.5.47] — 2026-09-23 ### Fixed - **jobd-mcp validates every tool call against its advertised schema.** mcp 2.x does not validate tool arguments, so the schemas were documentation only: top-level `depends_on`/`env`/`priority`/`max_wall_s`/`preemptible` on `jobd_submit` were silently dropped while the submit reported success, typo'd keys inside `extra` vanished, a bare string for `needs`/`state` was iterated into characters, and an unknown `jobd_list` state returned an empty page. Every schema is now closed (`additionalProperties: false`), `extra`'s keys and types are derived from the broker's `JobSubmit`/`JobRequires` models, and a violation returns `invalid_arguments` with a hint that names the right nesting level (e.g. "`depends_on` belongs under `extra`"). - **`jobd_submit gpu:false` no longer makes a job unroutable on an all-GPU fleet.** It was forwarded as `requires.gpu=false` ("must NOT be a GPU worker"). `gpu` is now a pin flag like the CLI's `--gpu`: `true` requires a GPU worker, `false`/omitted is no preference. - **`jobd_submit wait=true` returns a superset of the async reply** — `warning` and `project_label` are kept, and `log_size_bytes`/`log_returned_bytes`/`log_truncated` say whether the 8 KiB `log_tail` is the whole log. - **Every httpx transport failure is a transport error.** `JobdClient` caught a hand-picked tuple of httpx exceptions; `RemoteProtocolError`, `WriteTimeout` and `PoolTimeout` escaped raw and surfaced from the MCP as a bare "Internal server error" logged as a success. It now catches `httpx.TransportError`, with a new `protocol` failure kind. - **jobd-mcp classifies failures by origin, not by builtin exception type.** Unknown tools raise their own exception (a `ValueError` catch had labelled bad values and non-JSON broker replies `unknown_tool`); anything unexpected is `internal_error` with the exception type, a traceback on stderr, and an `error_kind` in the call log. - **`jobd_cancel` reports `signal_sent` from the broker's cancel reply** instead of a status read taken before the call, so a job that finished in between is no longer reported as signalled. - **`jobd_worker_delete` URL-encodes the host** — `broker-host#x` was sent as `/workers/broker-host` and deleted worker `broker-host`. - **404/409 hints name the tool's resource** — a worker-delete 404 no longer tells the agent to look for a job id with `jobd_list`. - **`GET /jobs?state_filter=` rejects unknown state names with 422** instead of returning an empty page to every client. ### Added - **Cancel reasons are recorded.** `POST /jobs/{id}/cancel` accepts an optional `{"reason": ...}` body and stores it on the `job_cancelled` event; `job cancel --reason` and `jobd_cancel reason` send it. The MCP tool and the client had accepted a reason the broker silently discarded. ## [0.5.46] — 2026-09-19 ### Fixed - **A starting worker killed the running jobs of every other jobd worker under the same user.** The startup sweep exists to kill workloads a crashed predecessor left behind, but a scope was named `jobd-.scope` and the sweep took every such unit in the session as its own. Starting a second worker for a different broker (an isolated test broker, the `JOBD_LIVE` suite) SIGKILLed the production worker's jobs, and two brokers issuing the same job id collided on one unit name. Scopes are now `jobd--.scope`, where `` is derived from the broker URL, and the sweep is limited to its own ``. Scopes it leaves alone are named in a warning. One consequence on upgrade: a scope left behind by a worker older than this release has no namespace and is no longer swept; the warning names it and gives the command to stop it. A normal restart drains first, so this only matters after a crash. `systemctl --user status jobd-.scope` becomes `jobd--.scope`; the worker logs the unit name when it starts a job. ### Added - **`job adopt --pid N`: register a process you already started as a job.** A GPU process launched outside the broker (`nohup`, tmux, a notebook) was invisible to it except as anonymous `unregistered_vram_gb`. Run `job adopt --pid N --project P [--gpu | --vram-gb N]` on the host where it lives and that host's worker watches the PID — launching nothing — so the process holds a slot, can be waited on, depended on (`--depends-on-any-exit`) and cancelled, and its VRAM counts as the job's rather than as foreign load. The process is identified by PID **and** `/proc` start time, so a recycled PID is refused (`adopt_pid_mismatch`); a process owned by another user is refused too (`adopt_uid_mismatch`), because the worker could not signal it. **The exit code of a non-child is not observable**, so an adopted job never ends `completed`: when the process exits it ends `orphaned` with `termination_reason=adopted_exit_unobserved`. No output is captured. Linux only. New `POST /adopt`, `job_adopted` event, and two nullable `jobs` columns (`adopt_pid`, `adopt_start_ticks`) added by the usual in-place migration; workers advertise support with a `jobd-adopt` tag and the broker refuses adoption onto one that does not. Design and limits: `docs/adoption.md`. ## [0.5.45] — 2026-09-18 ### Added - **`job submit --retries N [--retry-delay S]`: opt-in retry for a job that exits non-zero.** The default is still 0 — a failed job stays failed, and the broker never re-runs a job nobody asked it to. With `--retries`, only a workload that exited non-zero *on its own* is put back in the queue; a wall-clock or idle timeout, a preempt, a cancel (including one still pending when the failure is reported) and launcher/exec faults are never retried. Dependents keep waiting while the parent retries and cascade only when an attempt ends it. Each retry emits `job_retry_scheduled`; `job status` shows `retries = used/allowed`. Same fields on the API and MCP surfaces (`max_retries`, `retry_delay_s`). Adds four nullable-safe columns to `jobs`; the additive migration runs at broker start. ## [0.5.44] — 2026-09-02 ### Fixed - **`job ping` and the CLI error boundary now name _which_ network failure happened.** DNS, connection refused, connect timeout, read timeout and TLS were all collapsed into a single `unreachable` string, so the two cases with opposite remedies were indistinguishable: a refused connection means the path is fine and the broker is not running, while a dropped connection means the broker is probably healthy and a firewall or tailnet ACL is in the way. Each now reports its own `kind` plus a hint pointing at the layer that can actually be fixed. The error boundary's old advice, "is the broker running?", was actively wrong for a dropped packet. - **`job ping` no longer calls a broker that answered "unreachable".** A 401, a 5xx, or a non-ok body now render as `reachable, but not healthy` (the exit code is still 2). - **The MCP server passes the same diagnosis to agents.** A transport error now reads `jobd transport error (timeout): ...` followed by the hint, instead of a bare exception string. An agent cannot check the host itself, so it is more dependent on that steer than a human is: from "timed out" alone the obvious inference is "the broker is down", which is wrong for a dropped packet. The JSONL call log records the specific `transport_` rather than a flat `transport`. - **`job ping --json` gains `kind` and `hint`.** Its `reachable` field now means "the broker sent us bytes" rather than "nothing went wrong", so an authentication failure is no longer reported as a network failure. ## [0.5.43] — 2026-09-02 ### Fixed - **`job list --project` and `/events?project=` now fold the name the way submit does.** Submit stores a job under its registered spelling (`BETA` prices as, and is recorded as, `beta`), but both read filters compared the raw string, so no single spelling returned all of a project's jobs and a query for `Project_A` found no event written after the fold landed. The filter now matches the folded identity, the typed spelling, and the typed run label. `project_label` gains an index (created by `migrate()` on in-place upgrades too) since the list filter ORs on it. - **A project declared with `roots:` but no `priority:` is a load error, not a silent drop.** `load_projects` skipped any entry lacking `priority` before the roots validation ran, so `name: {roots: [...]}` vanished with no log line — the invisible removal the roots validation promises to prevent. `_default` may no longer declare roots either (they were validated, accepted, and never consulted). A leading `//` in a root or cwd is collapsed onto `/`; previously `//home/x` was a valid root that could never match. - **`/reload` takes the projects mutation lock.** A `set`/`nudge` that landed between reload's re-read and its swap of the projects table was persisted from the new table and lost. - **`job submit --explain` names the typed spelling on a pure fold, `job projects set/nudge` percent-encode the project name, and `GET /projects` / `job projects list` show each project's `roots`.** The dry run printed the typed name only when cwd had supplied the identity, so `--project PROJECT-C` rendered the registered `project-c` and never what was typed. The write commands spliced the name into the URL raw, so `job projects set ../reload 5` normalised to `POST /reload`, ran a config reload, and then reported a write that never happened. And no runtime surface listed the roots a broker had actually loaded; the only way to learn one was a dry run from inside it. The runtime overlay still persists priorities alone, so exposing roots on the read surface cannot leak them into it. - **The MCP server no longer hides a substituted project identity, crashes on a missing argument, or stalls the session behind one slow call.** `jobd_list` rows and the `jobd_submit` result now carry `project_label` (the name as typed, `null` when it equals `project`), so an agent can see that the broker scheduled a job under a different identity than it asked for; previously only `jobd_status` exposed it. A missing or mis-typed argument (`jobd_status {}`) is returned as an `is_error` result of kind `invalid_arguments` with a hint, instead of a protocol-level "Internal server error" with a traceback on stderr. Tool dispatch runs off the event loop, so a `jobd_submit wait=true` (up to 270 s) no longer blocks every other request on the stdio session. The `project` schema description now describes the cwd-roots fallback rather than the old `_default`-only one. - **A worker running two jobs at once can no longer kill the second job's fresh child during the first job's cleanup.** The reparented-orphan `/proc` sweep at job finalize gated on a point read of the in-flight count and then scanned with the lock released. Between the two, the poll loop could register and start job B, whose new child has this worker as parent and is not in job A's tracked set, so the sweep SIGTERMed it. The check and the scan now run under the same lock registration takes before any `Popen`, so B cannot appear inside the window and, if it registered first, the sweep declines. Only reachable with `max_concurrent > 1`. ### Security - **`config/projects.local.yaml` is gitignored, and a test now keeps it so.** The rule added in #129 landed as a single line containing literal `\n` sequences, so the pattern matched nothing and the file that exists to hold the operator's real project roots and names showed up as untracked, one `git add -A` away from being published. The rule is corrected in the companion privacy fix; `tests/test_no_private_paths.py` now asserts `git check-ignore` accepts the path, so it cannot silently regress. - **Tracked config no longer carries real paths or private names.** `config/projects.local.yaml` (gitignored, beside `projects.yaml`) now holds machine-specific `roots:` and private project names; each entry replaces the same-named tracked entry wholesale. Operators must create it on deploy or lose cwd-derived identity for those roots (docs/projects-yaml.md). The tracked `projects.yaml`, the replay corpus, tests, plans and docs are pseudonymised, and `tests/test_no_private_paths.py` fails the suite if a private path, host, address or session trailer is ever tracked again. ### Changed - **Docs and metadata caught up with what shipped.** `CITATION.cff` records only the concept DOI: the "DOI for this specific release" entry named the v0.5.38 Zenodo record beside `version: 0.5.42`, and it is stale by construction because Zenodo mints a version DOI only after the GitHub release exists. The cwd-identity design and plan under `docs/plans/` carry "shipped in v0.5.42" banners (the plan's 44 unchecked boxes were never status), `docs/projects-yaml.md` is retitled from "Plan" and says which sections are historical, and the README's link text matches its target. New `docs/events.md` catalogs every event name in `KNOWN_EVENTS` with a one-line meaning, pinned two-way by a test. `config/projects.yaml`'s header says plainly that every name and path in it is a pseudonym, where the real ones go (`projects.local.yaml`), and what a fresh clone must replace. A real tailnet address and login that the privacy scrub missed in the plan doc, and the maintainer's database path hard-coded in the corpus export script, are gone; the script now requires `JOBD_DB`. ## [0.5.42] — 2026-09-01 ### Added - **The cwd-derived project identity feature is no longer inert: `alpha`, `beta`, `gamma`, and `delta` now declare `roots:` in `config/projects.yaml`.** The matching machinery and its acceptance gate against 3,608 rows of real job history (`tests/test_corpus_replay.py`) landed first, but no project had ever declared a root, so a job typed with an unregistered run label (`stage2_sweep`, `leaf-assay`, `gamma-rerun`, ...) still fell through to `_default` regardless. Each root was picked by replaying `tests/data/project_cwd_corpus.csv` grouped by cwd and keeping only directories where every project name ever typed from them reads as a variant of one project — deliberately excluding `/home/user/trellis2` and `.../hunyuan3d` (>=97% alpha-typed in the corpus, trellis2 the single largest cwd in it) because both are generic third-party tool checkouts a differently-owned future clone could silently inherit. `docs/projects-yaml.md` gains a `roots:` section covering the schema, the three-rule resolution order (an explicit, registered `--project` always wins over cwd — the safety property the other two rules sit behind), longest-root-wins, component-wise (non-symlink-resolving) path matching, and the no-identity-plus-warning outcome when two projects' roots both claim a cwd. `--project` remains required; a cwd-derived identity changes what its value _means_, not whether it must be given. Shipping in the same release, the user-visible surface that makes a substituted identity findable and explainable rather than merely correct: - `job list --project NAME` now matches **either** the scheduling identity or the typed run label. Without this a job submitted as `stage2_sweep` and priced as `beta` was unfindable under the only name its submitter ever knew it by. - `JobInfo` gains `project_label`: the name as typed, `null` when it agrees with `project`, so the field reads as "something was substituted here". - `POST /resolve` gains `project_label` and `matched_root` (the root that supplied the identity, `null` when cwd was not consulted), and `job submit --explain` renders both — the dry run is where an operator asks "why did this get that priority?", and the root is the answer. - A new `cwd_identity_applied` event is recorded per job whenever a cwd-derived identity is applied, carrying the project, the typed label, and the matched root. It is not a warning: nothing went wrong, and a substitution that is invisible in the event stream is one nobody can audit after the fact. - The `job submit` substitution note goes to **stderr** (stdout carries the submit JSON alone, so `job submit ... | jq .id` parses). It reports what it can observe — that the typed name and the scheduling one fold together, or that they do not and cwd supplied the identity — rather than naming a mechanism it cannot verify: when two registered projects fold onto the same key the fold is refused and cwd decides, yet the names still fold together. `job submit --explain` carries `matched_root` and answers it. - Path matching now collapses `..` lexically before comparing components. It previously did not, so `/home/user/beta/../../tmp` — a job actually running in `/tmp` — matched a root of `/home/user/beta` and was priced at that project's `78`. `cwd` is free text on the wire, so this was caller-reachable. A root containing a `..` component is now a load error, in the same raise-don't-drop style as the other root validations. Symlinks and bind mounts remain unresolved by design: the broker cannot see the worker's filesystem. ## [0.5.41] — 2026-08-31 ### Fixed - **`job projects set/nudge` no longer crashes when the name folds.** v0.5.40 taught the write path to fold a spelling onto the registered project, but `POST /projects/{name}` kept returning the bare table, so the CLI went on indexing it by the name the user typed — `job projects set leaf_assay 65` wrote `leaf-assay` and then died with `KeyError` on the echo, reporting a traceback for a write that had already succeeded. Both write endpoints now report the project they landed on (`{"project": ..., "projects": {...}}`), matching what `/resolve` already did for the read path, and the CLI says when a name was folded rather than printing the typed spelling beside another project's priority. A new CLI is tolerant of an older broker's bare-table reply. ## [0.5.40] — 2026-08-30 ### Fixed - **A registered project no longer loses its priority to how the name was typed.** Project names are free text chosen at submit time and were matched with a bare `name in projects`, so `epsilon` ran at `_default` 40 while `epsilon` sat deliberately registered at 65 — a difference of case alone, raising no error and visible only in a warning nothing consumed. Case and `-`/`_` are now folded when matching, the resolved name is what `/submit` stores and `/resolve` previews, and an exact hit always wins. Folding is deliberately **not** fuzzy: `kappa` is not folded onto `kappa-fm`, because a suffix difference is a registration decision, and guessing there would run work at another project's priority — the same bug inverted. Two registered names that fold together are reported at load and fall back to exact matching rather than one being picked arbitrarily. ## [0.5.39] — 2026-08-30 ### Added - **Each submit warning now emits its own event, so a rate alert can name a cause.** `submit_warning` fires once per job whatever warned, and five independent causes — unknown project, preflight, cwd route, serialization, gpu contention — were joined into one string before emission, so the event stream could say THAT something warned and never WHICH. `unknown_project`, `preflight_warning`, `cwd_route_warning`, `serialization_warning` and `gpu_contention_warning` are now emitted alongside it and are all in `KNOWN_EVENTS`, so each gets its own `jobd_events_total` label rather than the `other` bucket. `submit_warning` itself is unchanged — still one per job, whatever warned — so its calibrated baselines stay comparable, and the `warnings` list published by dry-run and array submit is byte-identical. ## [0.5.38] — 2026-08-06 ### Changed - **`jobd[mcp]` now requires mcp 2.x (`mcp>=2,<3`), and the stdio server is migrated to its low-level API.** This is the "separate work" 0.5.37 deferred: rather than keep pinning below 2.0, `src/jobd/mcp/server.py` moved off the removed `@server.list_tools()` / `@server.call_tool()` decorators onto `Server(name, on_list_tools=..., on_call_tool=...)`. Handlers take `(ctx, params)` and return `ListToolsResult` / `CallToolResult` — a bare list is rejected — and `inputSchema` became `input_schema`. The extra's range **moves** rather than widens. The migrated code cannot import under 1.x at all (`ServerRequestContext` does not exist there), so a range spanning both majors could resolve to a version this package can no longer run — the same failure mode 0.5.37's pin was defending against, just inverted. ### Fixed - **A broker outage is an error result again, not a protocol crash.** mcp 1.x converted an exception raised inside `call_tool` into `isError=true`; 2.x removed that conversion. Left alone, `BrokerUnreachable` would have escaped as a protocol-level failure instead of the `is_error` response `tests/mcp/walkthrough.md` step 11 documents, so the handler now builds the result itself. This was invisible to both mypy and the existing suite. ### Added - **`tests/mcp/test_protocol_roundtrip.py`** — the whole 1220-test suite passed the migration without exercising it, because every test reached _past_ the protocol (calling `jobd.mcp.tools` directly, or via the `_jobd_dispatch` escape hatch). A completely broken handler registration would have failed none of them. These four tests drive `mcp.client.Client` against the real `Server` object and were confirmed to fail before the migration. ## [0.5.37] — 2026-07-28 ### Fixed - **`jobd[mcp]` no longer installs an SDK it cannot run against.** The MCP specification revision `2026-07-28` shipped alongside `mcp` 2.0.0 on the same day, and the extra's floor-only `mcp>=1.0` resolved straight to it. 2.0.0 is a major rework: the low-level `Server` class still imports, but its `@server.list_tools()` / `@server.call_tool()` decorators are gone (replaced by `add_request_handler` / `MCPServer`), so `build_server()` raises `AttributeError: 'Server' object has no attribute 'list_tools'` and `jobd-mcp` dies on startup. Verified against PyPI on 2026-07-28: a clean `pip install 'jobd[mcp]'` pulled `mcp-2.0.0` and the entry point failed; with `mcp>=1.0,<2` the same install resolves `mcp-1.29.0` and prints `jobd-mcp ready`. Note that the failure is at _startup_, not import — `import jobd.mcp.server` succeeds under 2.0.0 because the decorators run inside `build_server()`, so an import smoke test would have reported this green. The existing `tests/mcp` suite does call `build_server()` and would have caught it, but only if CI had ever installed 2.x; the pin is what keeps that from happening by accident. Migrating to the 2.x API is separate work — the deprecations in the new spec carry a twelve-month minimum window, and every jobd MCP surface is stdio, which the stateless-transport changes do not touch. ## [0.5.36] — 2026-07-28 ### Fixed - **Dependabot no longer opens PRs against the generated Docker requirements export.** `docker/requirements-docker.txt` is a `uv export` of `uv.lock`, not a manifest, but Dependabot's pip ecosystem kept discovering it and opening PRs that edited it directly — desyncing the export from its own source while leaving `uv.lock`, which CI installs from, untouched. Such a PR is red by construction (the lockstep lint compares the committed file against a live export of the PR's own `uv.lock`, so a rebase cannot fix it), and merging one would ship dependency versions no test has ever run into an image whose `--require-hashes` install exists to prevent exactly that. It is now excluded via `exclude-paths`. The previous mitigation — relocating the file under `docker/` — rested on the belief that a pip entry at `directory: "/"` scans only the root; that belief was wrong, and its deploy-lint could not detect the fact because it asserted the workaround was _in place_ rather than that it _worked_. The lint now pins the exclusion itself, with its own matcher tests. ### Added - **jobd is citable.** `CITATION.cff` (CFF 1.2.0, schema-validated, wired to ORCID [0009-0003-4055-5238](https://orcid.org/0009-0003-4055-5238)) and `.zenodo.json` are now in the repository, so GitHub renders a **Cite this repository** panel and Zenodo mints a DOI from pinned metadata rather than inferring it. A `citation` CI job fails any change where `CITATION.cff`'s version disagrees with `pyproject.toml`: the file records the version in a second place, and nothing was keeping the two in agreement — `ghostcite` shipped `0.5.0` while its `CITATION.cff` still said `0.4.0`, and `plant-genomics-mcp` drifted within two hours of the file landing. The gate mirrors the existing "tag matches pyproject version" release check in `release.yml`, which exists to refuse a mislabeled artifact; this is the same invariant one layer earlier, on every pull request. Stale citation metadata is worse than none — it is machine-readable, and a wrong version propagates into other people's bibliographies where nobody re-checks it against the tag. - **`max_concurrent > 1` now has real concurrency tests.** The setting was covered only by parsing assertions (`_max_concurrent_jobs()` defaults to 1, the API round-trips the field) and by nothing that ran two jobs at once — the last open finding from the 2026-07-25 adverse-conditions audit, and the behaviour the README's pueue/task-spooler migration table points migrants at for "parallelism limits". `tests/test_worker_concurrency.py` pins the ordering that actually bounds concurrency (`_reserve_and_dispatch` records the reservation synchronously in the poll thread, before it returns), that N jobs genuinely overlap at a cap of N, that the gate stops at the cap, that the heartbeat's `running`/`in_flight_job_ids` track live reservations, and that `_allocations_total` sums every pending claim rather than the newest. A guard also pins the real poll-loop gate expression, since the tests mirror it rather than driving the HTTP loop. Note for future test authors: the first version of these tests passed even with the historical bug reinstated — asserting on a count after `Thread.start()` is a race that usually resolves in the buggy code's favour — so the ordering tests now prevent the worker thread from running at all, which makes the two implementations distinguishable with certainty. ## [0.5.35] — 2026-07-26 ### Fixed - **`pip install jobd && jobd` now actually boots.** `JOBD_DB_URL` defaulted to `sqlite:////app/data/jobd.db` — the _Docker container's_ path — so the broker died on startup with SQLite's opaque "unable to open database file" on any machine that was not the image. That default had been present since the first commit, meaning the README quickstart's very first command had never worked for a bare pip install on any released version. It now defaults to `$XDG_DATA_HOME/jobd/jobd.db` (falling back to `~/.local/share/jobd/jobd.db`) and creates the directory on boot; if it cannot, the error names the path and the environment variable instead of surfacing a SQLAlchemy traceback. Deployments are unaffected: the Dockerfile sets `JOBD_DB_URL` explicitly, and nobody running from pip could have been relying on a default that could not work. Found by re-running the launch dry-run against a real `pip install` rather than the Docker image — the previous dry-runs, CI, and `test_broker_boots_and_runs_a_job_with_no_config_at_all` all supplied the DB path themselves, which is precisely why none of them saw it. - **The pueue/task-spooler migration table renders correctly again.** The `simple_gpu_scheduler` row carried an unescaped `|` inside a code span, and GitHub-flavoured Markdown splits table cells on pipes even inside backticks — so the renderer dropped the entire right-hand cell and showed a dangling `` `... `` where the migration command should have been. The source looks fine and every local previewer renders it fine, which is why it survived on the front page; it was caught by pushing the table through GitHub's own `/markdown` API. `tests/test_markdown_tables.py` now sweeps every tracked Markdown file for the same mistake rather than just the file that happened to be wrong. ### Security - **The README now discloses the unauthenticated surface, not just the two walls.** Its Security section listed interface binding and the bearer token and stopped there, without saying that `/livez`, `/readyz` and `/metrics` answer with neither — and that `/metrics` publishes the broker version, job counts and every worker's hostname and version. `docs/security.md` already documented this; the README, which is what most readers judge the security posture by, did not. `tests/test_security_doc_parity.py` was added to prevent exactly this drift but covered only `docs/security.md`, so it now derives the same assertions against the README in both directions. ## [0.5.34] — 2026-07-25 ### Fixed - **The sweeper no longer reclaims across an interval it did not observe.** Every reclaim decision is a wall-clock delta against a stored timestamp — "this worker last heartbeat 400s ago, so it is dead" — which is only a valid inference if the broker was _watching_ those 400s. It had two ways not to be: it had just started (the first sweep runs immediately at boot, before any worker can re-heartbeat), or it had been frozen by a host suspend or a stalled loop. Either way the first post-gap sweep saw the whole fleet as dead and every in-flight job as reclaimable, and acted on all of it at once — including the `wall_clock_exceeded` branch, whose terminal `resurrect` does **not** undo, so healthy long-running jobs died permanently and cascade-cancelled their dependents. `sweep_once` now records each pass and suppresses the time-based terminal phases (scheduling-timeout expiry, ASSIGNED reclaim, RUNNING wall-clock backstop and dead-worker reclaim/orphan) for one cycle when process uptime is under the reclaim window or the gap since the previous pass far exceeds the sweep interval. Worker offline/stale marking, retention, env scrub, version drift and the queue warnings keep running — none of them ends a job, and all self-correct on the next heartbeat. Suppression is logged and emits a `reclaim_suppressed` event, so a sweep that declines to reclaim is never mistaken for one that found nothing to reclaim. - **A full disk no longer turns every job-log chunk into a 500.** `POST /jobs/{id}/log` wrote the chunk with no guard, so `ENOSPC` propagated: the worker swallowed the error and kept going — the job survived but its output vanished silently — while the broker wrote a stack trace per chunk to the same full disk. The write is now guarded and reports the drop in the response body (`{"bytes": 0, "dropped": N, "reason": ...}`) rather than raising, matching the principle `_emit_event` already applied one module over: observability must never break broker liveness. - **A finished job no longer displays a queue warning that resolved when it started running.** `warning` is one column serving two purposes — durable explanations of a terminal state (`parent_failed:`, `auto-preempted in favor of`) and transient advisories about the queue a job is sitting in (`will queue behind job N`, `blocked:`, `no matching worker —`, the submit-time projects.yaml notice). The sweeper only clears the second kind on rows that are still QUEUED, so a job that left the queue kept its advisory for the life of the row: `job list` was rendering `⚠ will queue behind job 3037 on laptop` under a job that had completed eleven hours earlier, in the present tense. Read surfaces now suppress queue advisories once a job is terminal, while keeping the warnings that explain _why_ it is cancelled or preempted. Applied in `_to_info`, where `GET /jobs`, `GET /jobs/{id}`, the submit/cancel/preempt responses and the MCP tools all converge — previously `job status` omitted the warning while `job list` showed it, so the two surfaces disagreed about the same row. The stored value is untouched; only the presentation stops asserting a resolved prediction as current. - **A worker the broker refuses now says so, instead of polling forever in silence.** The poll loop treated every non-200 from `/next-job` as "no job", so a worker whose payload the broker rejects — most plausibly a `422` because the worker predates a field the broker now requires — backed off and re-polled indefinitely while logging nothing. It now warns on the first refusal and at most once a minute thereafter, with the suppressed count and the broker's response, naming the likely cause. That matters because mixed versions are a designed state: installs pin a worker to the broker's version, `update-worker.sh` defers upgrades while a worker is busy, and `version_drift` exists because prolonged mismatch is expected. - **The worker↔broker request contract now has a compatibility floor in CI.** `NextJobQuery` and `WorkerHeartbeat` declare required fields with no defaults, so adding one silently locks out every older worker — and nothing caught it. `tests/test_worker_payload_compat.py` freezes the minimum payload an older worker sends and asserts it still registers, still claims, and still receives a real job; a companion test proves every key in that payload is load-bearing, so the floor cannot rot into a bloated dict that passes vacuously. Adding a required field now fails CI with an explanation instead of stranding the fleet. ### Changed - **The hashed Docker requirements export moved to `docker/requirements-docker.txt`.** It is a generated `uv export` of `uv.lock`, but at the repo root Dependabot's pip ecosystem (`directory: "/"`) discovered it as a _manifest_ and opened PRs editing it in place — which desyncs it from the source it is generated from, so every such PR failed the lockstep lint by construction, and merging one would have shipped dependency versions CI never ran (`uv.lock`, what the test job installs from, was left untouched). Dependabot has no per-file exclude, so relocating it out of the scanned directory is the supported fix. A new deploy-lint pins the placement in both directions: the export must not be at the root, and no dependabot pip entry may scan the directory it moved to. Regenerate with `uv export --frozen --no-dev --no-emit-project --format requirements-txt -o docker/requirements-docker.txt`. ## [0.5.33] — 2026-07-25 ### Security - **`/readyz` no longer returns the database exception text to unauthenticated callers.** It answers a fixed `"reason": "database_unavailable"` and logs the exception broker-side instead. `/readyz` is exempt from _both_ the bearer token and the tailnet source-IP ACL, so its body is readable by anyone who can reach the port; on the default SQLite backend the old message leaked little, but `JOBD_DB_URL` accepts any SQLAlchemy URL and a network driver puts host, port and username into it. The endpoint stays deliberately uncached — a cached readiness probe reports "ready" for the length of its TTL after the database has gone, which is wrong at the only moment the probe matters. - **The tailnet source-IP ACL can no longer be influenced by a forwarded header.** uvicorn is now started with `proxy_headers=False`. Its default (on) lets a request from a trusted peer rewrite `request.client.host` out of `X-Forwarded-For` — the exact value the ACL decides on. This was inert in the shipped deployment (uvicorn only trusts the header from a `127.0.0.1` peer, and the broker binds directly to a tailscale IP under `network_mode: host`), and would have stopped being inert the moment a TLS-terminating reverse proxy was put on localhost in front of it. - **A non-ASCII bearer token now returns 401 instead of 500.** `hmac.compare_digest` raises `TypeError` on a `str` holding non-ASCII, so the caller's own bytes decided the error shape; tokens are now compared as bytes. No token material was ever exposed. - **A bearer token with surrounding whitespace now authenticates.** The expected value was already `.strip()`ed — `.env` files and systemd `Environment=` lines routinely carry a trailing newline — while the presented value was not, so a client echoing the operator's own trailing byte was rejected. - **`docs/security.md` now documents the unauthenticated surface it previously denied.** The page claimed every request needs a bearer token; three endpoints (`/livez`, `/readyz`, `/metrics`) are exempt from both walls, and `/metrics` publishes the broker version, job counts, and every worker's hostname and version. A new `tests/test_security_doc_parity.py` derives that list from the live constants in both directions, so the page cannot drift again — it also caught a reference to `jobd_job_get`, an MCP tool removed on 2026-07-12. ## [0.5.32] — 2026-07-25 ### Fixed - **`job submit`'s worker-health banner reports the broker's verdict instead of re-deriving it.** The CLI carried its own `STALE_HEARTBEAT_SECONDS = 60`, duplicating the _default_ of the broker's `JOBD_STALE_WORKER_THRESHOLD_S` — so an operator who tuned that threshold got a banner contradicting `job workers`. (Same defect as the MCP server's `STALE_AFTER_S`, removed in 0.5.26; this was the second call site.) - **`job submit --stdin` reports a mid-batch refusal instead of raising `KeyError`.** An admission block, an unknown project, or a broker restart used to abort the batch with a bare traceback and no account of what had already been submitted. The broker's own reason is now printed against the offending line, along with how many lines went unsubmitted, and the exit code is 1. The summary also stops claiming an `a..b` id range when the ids are not contiguous. - **`job logs -f` on an unknown job id fails instead of exiting 0.** It previously streamed nothing and succeeded, so a typo looked like a job with no output. - **Worker: no lingering SIGKILL-escalation timer when a job exits in the same instant a cancel/preempt lands.** `wait_with_grace` now closes the timer list when it drains it, and a racing `initiate_termination` declines to start a timer against a closed list. The leak was benign (the timer's callback found the process already gone) but left a timer thread alive for up to the grace window after `run_job` returned — and made `test_run_job_cancels_kill_timer_on_exit` flaky (CI red on the v0.5.31 main push). New deterministic regression test forces the losing ordering. - **Worker: a cancel arriving as the workload exits no longer relabels a clean run `cancelled`.** v0.5.31 stopped the late signal from leaking a kill timer but still let it write `got_signal`, and `run_job` does not read `final_state()` until after straggler reaping and the `/checkpoint-complete` POST — so a `poll_signals` thread parked in its 5s `GET /signal` for that window could report a job that exited 0 as `cancelled` with `exit_code: 0`. `initiate_termination` now declines outright once the timer list is closed, which is precisely the moment `wait_with_grace` has the exit code; a signal that lands while the workload is genuinely still running is unaffected. ### Security - **`job fleet add` no longer lets the broker's reported version reach the remote shell unquoted.** The install arguments are spliced into a script piped to `ssh bash -s`, and `--version` defaults to whatever `GET /health` returns — so a compromised broker, or a MITM on a plain-http `JOBD_URL`, could run arbitrary commands as the ssh user on every host an operator bootstrapped. `install-worker.sh`'s own `X.Y.Z` check was no defense: it runs downstream of the splice. Every interpolated value is now `shlex.quote`d, and the resolved version is validated against `X.Y.Z` client-side before any ssh happens. `--dry-run` additionally stages under `mktemp -d` instead of a fixed `/tmp/.jobd-fleet-preflight.sh`, closing a symlink-plant that turned the preflight write into an arbitrary file write on the target. Both are covered by tests that execute the composed script for real. ### Changed - **Dependency refresh (17 minor/patch bumps), done at the source.** fastapi 0.136.1→0.140.0, sqlalchemy 2.0.49→2.0.51, uvicorn 0.46.0→0.51.0, pydantic 2.13.3→2.13.4, typer 0.24.2→0.27.0 and twelve more. Bumped in `uv.lock` with `requirements-docker.txt` regenerated from it, rather than editing the generated export directly — the export is derived, and editing it in place ships versions no test ran against. - **Route enumeration in the test guards survives FastAPI 0.140.** `include_router` no longer flattens a sub-router's endpoints into `app.routes`; it leaves an opaque `_IncludedRouter` whose `path`, `methods` and `routes` are all absent. Every broker endpoint has lived behind a router since the Stage-3 split, so the guards that walked `app.routes` saw _nothing_. One failed loudly; two — including the auth wall's "every route except the probes must 401" check — compare `live - expected` and so kept passing while verifying nothing at all. Enumeration now reads the OpenAPI schema (prefix-resolved, and a documented API) from one shared helper, both guards assert a non-trivial route count so an empty enumeration can never be silent again, and `tests/test_route_table.py` pins the helper itself. - **The image build no longer downloads an unpinned pip.** `pip install -U pip` ran before the `--require-hashes` dependency install — the one unverified fetch in a build whose premise is hash pinning, and the most privileged code in it. The digest-pinned base image pins pip too; bump the digest to move it. - **The env-at-rest scrub is batched (500 rows per sweep).** It is the only sweeper query that reads terminal rows, and those are retained forever by default, so an unbounded `SELECT` loaded the whole eligible backlog as ORM objects in one transaction on the first sweep after an upgrade. `env_scrubbed_at` already makes the scan monotonic, so a backlog simply drains over consecutive passes. - **`profiles.yaml` / `classifier.yaml` that exist but cannot be read no longer crash the broker at startup.** 0.5.31 made a _missing_ file optional; a permission error or a directory in the file's place still aborted boot — the same failure class. The reason is logged rather than swallowed. - **MCP `jobd_list` reports zero counts.** States with no matching jobs were omitted from `counts`, leaving an agent unable to distinguish "nothing queued" from "queued was never queried". - **The release workflow's test leg waits on the tag/pyproject check** rather than spending a full run discovering a mismatch it was going to refuse anyway. ## [0.5.31] — 2026-07-16 ### Fixed - **The README quickstart actually works on a pristine machine now.** A launch-prep dry-run in a clean container ran the quickstart's exact commands and found `pip install jobd && JOBD_ALLOW_NO_AUTH=1 jobd` **crashed on startup**: the README has always promised the three config files are optional, but only `load_projects` honored it — `load_profiles` and `load_classifier_rules` hard-failed on the missing `/app/config/*.yaml` defaults. Invisible for months because CI exports `JOBD_CONFIG_DIR` and production runs in Docker where the path exists; fatal for every new user's first command. Both loaders now return empty defaults with an info log, and a regression test boots a broker with a nonexistent config dir and runs a job through it. Second find from the same dry-run: a root-run worker (every "try it in docker" user) refused submits from its own `$HOME` because `/root` is outside the mount-root candidates — the worker now always advertises its own home when nothing else covers it. ### Changed - **The README demo is re-recorded against the live fleet** and now shows the current surface: `job fleet status` (four workers, versions, a drift warning), a GPU job routing to the box with enough free VRAM and streaming back, a `--stdin` two-job batch, and `job logs -f` following one to completion. Also fixed a broken code fence from the fleet-add docs edit that was swallowing the following paragraph. ## [0.5.30] — 2026-07-15 ### Fixed - **The v0.5.29 GHCR image publish failed — the Dockerfile didn't COPY the fleet assets the wheel force-includes.** hatchling's metadata generation hard-fails on a missing force-include source, and `scripts/install-worker.sh` wasn't in the image build context when `pip install --no-deps .` ran. Invisible to the PR suite by construction (the wheel-packaging test builds from a full checkout), so it fired at the one moment it could: on the tag. Fixed with the COPY, verified by a real `docker build` + in-image asset check, and a new deploy-lint requires every pyproject force-include source to be COPY'd before the project install — mutation-verified against the broken Dockerfile. (v0.5.29 shipped fine to PyPI; only its container image is missing, and this release supersedes it.) ### Added - **`docs/agent-cookbook.md` — the worked tour of driving jobd from an LLM agent.** Five patterns with the real tool calls: fire-and-babysit on the ETA's own cadence (and why `idle_timeout_s` is non-negotiable for agent-submitted work), surviving preemption via the `install_preemption_handler` three-liner + resubmit-from-checkpoint, sweeps with dependency-chained collect jobs, asking the broker _why_ a job won't schedule (warnings, live capacity, `dispatch_skip` events), and events-as-flight-recorder post-mortems — plus ground rules worth pasting into an agent's instructions. Writing it caught a README drift: the front page advertised `jobd_job_get` (long gone) and omitted `jobd_events`; fixed, and a new parity test pins the README's and cookbook's tool mentions to the registered `_TOOLS` list so neither can go stale again. ## [0.5.29] — 2026-07-15 ### Changed - **README: the comparison table now covers the neighbors people actually ask about, accurately.** Added Pueue (whose own README declares distributed execution out of scope — jobd is that missing layer) and HyperQueue (the closest multi-machine rival: single binary, but no VRAM tracking, no preemption/checkpoint, no agent interface); corrected the SkyPilot row to say what its "existing machines" mode really does (installs a k3s cluster on your boxes) and dstack's real host prerequisites (Docker + passwordless sudo). New "Coming from pueue or task-spooler?" section maps the verbs one-to-one (`pueue follow` → `job wait`, groups → projects) — a single-machine jobd deployment behaves like a network-reachable pueue, so the migration costs nothing. - **Repo visuals: renderer-proof architecture diagram + social preview.** The README architecture image was a mermaid pre-render that drew its labels with `foreignObject` HTML — fine in a browser, blank or garbled anywhere that treats SVG as a static image — and its trebuchet styling clashed with the terminal demo card above it. Replaced with a hand-authored, self-contained SVG (plain SVG text only, same dark palette as the demo terminal) that renders identically on GitHub light/dark and PyPI, and now also shows the tailnet boundary, the broker's storage/observability chips, and the long-poll/CD dispatch loop. Added a matching 1280×640 social-preview card (`docs/assets/social-preview.png`) and a GHCR container badge. ### Added - **`job fleet add user@host` — a new worker in one command.** The manual closing instructions of `install-worker.sh` (copy units, edit ExecStart, wire the env file, enable the timer) are now automated over a single ssh connection: the installer _and_ `update-worker.sh` are pushed on stdin (the API token never touches an argv, local or remote — proven by the ssh-stub tests), the install is pinned to the broker's running version, the generated systemd units share one `worker.env` (0600) and point at the pushed updater — not the `%h/jobd/scripts/…` git-clone path the committed templates assume — and the command then waits for the worker to actually register with the broker before declaring success. `--dry-run` runs only the remote detection preflight; `--no-systemd` installs the venv and config without units. **`job fleet status`** shows every worker's version against the broker's with a drift marker. Drift guards: the generated unit's load-bearing directives (KillMode=mixed, TimeoutStopSec, Restart) are test-pinned to `scripts/job-worker.service`, and a packaging test builds a real wheel and asserts the fleet assets landed — a silently-dropped force-include would otherwise strand every pip-installed CLI. - **`job submit --stdin` and `job logs --follow/-f`.** `--stdin` reads one shell command per line and submits each as an independent job sharing the invocation's project/profile/resource options (run via `bash -c`, so pipes and redirects survive; blank lines and `#`-comments skipped; one `idcommand` per line on stdout for composability) — simple_gpu_scheduler's batch idiom, fleet-wide. `logs -f` replays a job's log from the start and streams until terminal, exiting with the job's own exit code — `pueue follow` muscle memory over the existing `/wait` SSE stream. ## [0.5.28] — 2026-07-15 ### Changed - **Stage 3 of the app.py split landed — the endpoint closures moved onto APIRouters.** Parked twice as "all-downside" back when it meant a ~90-reference rewrite; the intervening service-layer extractions shrank what endpoints actually capture to five names, so the move became mechanical: `build_app` is now ~250 lines of wiring (engine, config, auth, sweep loop, long-poll plumbing, test seams) and the 30 endpoints live in `jobd.broker.routes.{probes,jobs,workers,events,config}`, each module's `build_router(deps)` unpacking `BrokerDeps` into the exact local names the bodies always captured — the bodies moved verbatim, and the route table (paths, methods, response models) is proven byte-identical against the pre-split tree. Test seams that were patched on `jobd.app` (`_cas_state`, `WAIT_STREAM_CHUNK_BYTES`) are patched on the routes module that resolves them now, the same precedent as the submit-service split; the names are gone from `jobd.app`, so a stale patch fails loudly instead of silently no-opping. ### Added - **Every `JOBD_*` variable is now documented in one place — enforced, not aspired.** ~35 env vars were read ad-hoc across broker, worker, CLI, and MCP with no way to discover them short of grepping. `docs/configuration.md` catalogs them all (defaults, purpose, component), and `tests/test_config_catalog.py` keeps it honest in **both directions** via an AST literal sweep: a variable added to the source without a catalog row fails CI, and a catalog row whose variable no longer exists anywhere fails CI (the 07-01 audit once chased a documented `--env` flag that had never existed — stale docs are worse than none). Deliberately _not_ a frozen settings object: knobs are read at the point of use on purpose (sweep passes re-read per tick, keeping them runtime-tunable), and centralizing the reads would trade that away for nothing. - **A worker silently stuck on an old release now announces itself.** The metric half existed (`jobd_worker_version_info` vs `jobd_build_info`), but only helped someone watching a dashboard — and the fleet's history says nobody is: the cwd_missing fix shipped in v0.5.10 and did nothing for twelve days because the workers were still on 0.5.3. The sweeper now emits one `version_drift` event per episode when an **online** worker's version has mismatched the broker's continuously for `JOBD_VERSION_DRIFT_WARN_HOURS` (default 24; negative disables). Short-lived mismatch stays silent on purpose — the worker CD defers while a job runs and self-heals on the next idle tick; the event is for the episode that never self-heals. Episode state lives on the worker row (`version_mismatch_since` / `version_drift_warned_at`, additive migration), clears when versions align, and re-warns on the next release's episode. A versionless worker counts as drifted by definition; offline workers are excluded (the offline warning covers them). ## [0.5.27] — 2026-07-15 ### Security - **A secret submitted via `env` no longer sits plaintext in the SQLite file forever.** Job rows are kept indefinitely by default (`JOBD_JOB_RETENTION_DAYS=0`), so every `HF_TOKEN`/`WANDB_API_KEY` ever submitted lived in `jobs.env_json` — and in every backup — for the life of the deployment, readable by anyone with access to the DB file. The sweeper now masks a terminal job's env **values** (keys kept, same `{"KEY": "***"}` shape every read surface already returns) once it has been finished for `JOBD_ENV_SCRUB_HOURS` (default 1; negative disables). Nothing observable changes — reads were always masked and only the `/next-job` claim ever consumes real values, which no terminal row can reach again. The one terminal→QUEUED restore path (a cascade-cancelled child a parent resurrect can un-cancel) is excluded via its `parent_failed:` warning stamp — now a shared constant with the un-cascade's matcher so the two predicates cannot drift — and a test pins that a restorable child keeps its real env no matter how old. - **Workloads no longer inherit the broker credential.** The worker launched every job with a full copy of its own environment, `JOBD_API_TOKEN` included — any script, framework, or crash reporter inside a workload that dumps its env (many do) exfiltrated the shared bearer token into logs, checkpoints, and third-party telemetry. The worker now drops `JOBD_API_TOKEN` before layering the submitted env; the jobd facilities a workload legitimately uses (`JOBD_CHECKPOINT_DIR`, `JOBD_CHECKPOINT_GRACE_S`) were always delivered separately. A job that genuinely needs broker API access opts in explicitly with `env: {"JOBD_API_TOKEN": ...}` at submit, which layers over the scrub. ### Changed - **`run_job` decomposed (pure refactor).** The worker's 585-line dispatch function — where the last three audits' worker bugs all lived — is now 148 lines of orchestration over seven early-return pre-flight helpers and a `_WorkloadRun` object holding the state its threads share (signal routing, SIGKILL escalation, the three watchdogs, drain hook, stdout streaming, grace wait, straggler reaping, final-state mapping). Same data, same lock discipline, no behavior change; the H-2 guaranteed-terminal `try/finally` shape is preserved exactly, and the three watchdog kill branches share one tail instead of three verbatim copies. One micro-alignment: the first-output-watchdog disarm now actually happens under `last_log_lock`, which is what its own comment always claimed. - **Test/CI hygiene sweep.** The CI coverage gate now measures **branch** coverage (line coverage alone lets an untested `else` arm count as covered because its `if` line ran) with the floor recalibrated to 83 (85 measured), and a deploy-lint keeps both the floor and `--cov-branch` from silently disappearing. The one standard broker-fixture body that had been pasted into 24 test files is a single set of `app`/`client`/`client_logs` fixtures in `tests/conftest.py`. A drain test that gave a cold `python3` workload a fixed 0.5s to install its SIGTERM-ignoring handler now waits for the workload's READY byte — firing early made the SIGKILL-cap path under test pass vacuously. The README platform table now says what CI actually verifies (Linux) instead of implying macOS/Windows are tested. - `/wait`'s per-tick job-state read runs off the event loop (`asyncio.to_thread`) — it was the last synchronous SQLite call issued directly on the async loop, where a contended read would stall every in-flight request in the process, once per waiting client per 0.5s tick. ### Added - **Per-PR changelog fragments (`changelog.d/`) replace editing `[Unreleased]` directly.** Five parallel PRs in the v0.5.26 merge train conflicted on the same `CHANGELOG.md` lines at every single merge — the shared-insertion-point design guaranteed it. Each PR now adds its own `changelog.d/..md` file (no shared lines, no conflicts); `scripts/roll-changelog.py ` assembles them into the released section at release time and deletes the consumed fragments. The scheme is enforced by real-execution tests: the roll script runs against fixture repos (happy path, unknown category, malformed bullet, stray `[Unreleased]` content, duplicate version — all refused with the changelog untouched), and `[Unreleased]` staying empty is a standing assertion, so an entry pasted the old way fails CI with a pointer to the new home. ## [0.5.26] — 2026-07-15 ### Fixed - **A malformed MCP tool body crashed the error mapper instead of explaining itself (found by the new real-broker contract test).** A real FastAPI 422 carries `detail` as a _list_ of error dicts; `map_broker_refusal` regex-searched it and raised TypeError, surfacing a transport error instead of `kind="invalid_arguments"`. The synthetic error tests never caught it because they only ever fed the mapper strings — the exact drift class the audit flagged (T-LOW-5). The mapper now coerces non-string details, and `tests/mcp/test_errors_real_broker_contract.py` drives the REAL app to emit every mapped refusal (missing parent, terminal parent, 404, 422), so a reworded broker message fails CI instead of silently degrading agents to `bad_request`. - **`jobd_list` no longer fetches the entire job history on every call (audit L-8).** It issued at most one `state_filter` and pulled the _unfiltered_ list to compute counts — the transfer the broker's pagination (v0.5.22) was built to end, still happening on every default call. It now issues one **bounded** request per requested state and takes exact per-state counts from `X-Total-Count` (new `client.list_jobs_with_total`); the default active-state set is **derived** from `JobState`/`TERMINAL_STATES` instead of hand-listed, so a future non-terminal state can't silently vanish from the default view. - **`jobd_workers` trusts the broker's staleness verdict (audit L-7).** It recomputed staleness against a local `STALE_AFTER_S = 60` — a copy of the broker default that ignored `JOBD_STALE_WORKER_THRESHOLD_S`, so a tuned broker produced contradictory verdicts between `job workers` and `jobd_workers`. It now reads `WorkerInfo.state`. - **The MCP call log can no longer fail the tool call it logs (audit Q-2).** `_log_call` runs in a `finally` on the hot path with no guard; a read-only HOME or full disk turned every _successful_ broker result into an OSError. Filesystem errors are swallowed — telemetry never outranks the payload. - **The `jobd_events` schema is derived from the real event vocabulary (audit Q-1).** The hand-typed list was stale within a day of shipping — 7 real event types (`job_resurrected` among them) weren't advertised, so agents were told those filter values didn't exist. The description now derives from `models.KNOWN_EVENTS` (itself AST-sweep-tested for completeness), the `source` enum covers `hook`/`mcp` (the old enum _forbade_ filtering to events the broker records fine), and parity tests hold both. - **A cancel issued while a worker was frozen was silently dropped across an orphan→resurrect round trip (audit 2026-07-15 F1, HIGH).** The resurrect (v0.5.25) promises a user cancel is honored "through `GET /jobs/{id}/signal`" — but both worker-gone orphan writes (`worker_died` in the sweeper, `worker_restarted` in the reconcile) erased the pending signal with `signal=None`, so by resurrect time there was nothing left to poll. Exact scenario: worker freezes, user cancels the hung job, sweeper orphans it, worker thaws, job resurrects and **runs to completion** — durable user intent gone. Both writes now preserve the signal (inert on a terminal row if the worker never returns; a resurrected worker's next poll delivers the kill). A pending _preempt_ survives the round trip too — a resurrect is the same incarnation, not a fresh one, so the M1 stale-signal hazard does not apply. - **The un-cascade restored children with a stale queue clock (audit F5).** A child restored to QUEUED by a parent's resurrect kept its pre-outage `last_enqueued_at`, so one with `scheduling_timeout_s` shorter than the outage went straight back to `SCHEDULING_TIMEOUT` on the next sweep — the interval it spent _cancelled_ counted as time "waiting to be scheduled". The restore CAS now stamps a fresh `last_enqueued_at`. - **The dependency cascade's child write is now a CAS on QUEUED (audit F5).** It was a plain ORM assignment against a QUEUED snapshot; a user cancel landing in the window was overwritten with a `parent_failed` warning — same final state, but the stamp made the human's cancel wrongly restorable by a later parent resurrect's un-cascade. The CAS loses cleanly: the cancel stays untouched and unrestorable. - **A resurrect flap on a stale in-flight report is pinned as self-healing (audit F4).** A heartbeat snapshot taken just before a worker finished a job can resurrect a row nothing is running; the forward reconcile re-orphans it within `RECONCILE_MISS_THRESHOLD` heartbeats. Behavior was already correct — now there is a regression test holding it. - **The worker-CD idle gate read a heartbeat-lagged gauge, and its recovery comment was wrong (audit 2026-07-15 F2).** `busy()` consulted the `/workers` registry's `running` count, which only this worker's own heartbeat refreshes — a job claimed via long-poll right after a heartbeat was invisible to the gate for up to ~5s. The gate now counts **job rows** (`/jobs?state_filter=assigned|running`, owned by this host), which the `/next-job` claim writes synchronously. The comment claiming a lost race is fine because "the broker resurrects it on the next heartbeat" was false — a restart _drains_, and `worker_shutdown` is a deliberate terminal the resurrect never undoes; dependents cascade-cancel. The comment now states the real stakes. - **`update-worker.sh`'s fail-safe branches were unreachable under `set -e` (audit L-2).** A broker-unreachable `$(api | python3)` assignment killed the script before its own diagnostic; the `${running:-1}` "unknown means busy" default could never engage. Both assignments now tolerate failure, so the written fallbacks actually run — proven by a real-execution test that stubs curl dead and asserts the diagnostic prints. - **`install-worker.sh` installed unpinned PyPI latest — the exact violation `update-worker.sh` exists to prevent (audit L-1).** A fresh worker could start _ahead_ of its broker, the schema authority. The installer now pins to the broker's own version (asked via `/health`) or an explicit `--version`, with no unpinned fallback, and its closing instructions install the update timer so the host self-heals thereafter. - **A projects.yaml `defaults:` value now has to be sane before it can reach a Job row (audit 2026-07-15 F3).** YAML bypasses Pydantic, and the split was live: `max_wall_s: 0` reads as _unset_ to the worker (`or`-falsy, never enforced) but _set_ to the sweeper (`is not None`) — which then orphans the healthy running job as `wall_clock_exceeded`, a non-resurrectable terminal, and discards its `/complete`. A **string** value (`max_wall_s: "8h"`) TypeErrors the sweeper's arithmetic and aborts the ENTIRE sweep pass (scheduling timeouts, reclaims, offline marking, retention) every 30 seconds. `_parse_defaults` now enforces the same bounds as `JobSubmit` and drops bad values loudly. YAML `true` is caught too (`bool` is an `int` subclass — it would have landed as `max_wall_s=1`). - **`dispatch_skip_state` eviction snapshots the dict before iterating (audit F6).** Concurrent `/next-job` attempts share it; iterate-while-insert raises "dictionary changed size during iteration" and 500s the poll. - **The `/projects` mutation endpoints are serialized and the overrides overlay is written atomically (audit F7).** set/nudge mutate-then-iterate a shared dict from threadpool threads (dict-changed-size 500; last-write-wins overlay), and a crash mid-write could leave half a YAML document where the runtime priorities live. Lock + tmp-file + `os.replace`. - **A typo'd `JOBD_LOG_LEVEL` no longer crash-loops the worker (audit L-3).** `setLevel("DEUBG")` raises `ValueError` at logging setup, and `Restart=on-failure` turns one bad env var into a crash loop. Unknown levels fall back to INFO with a warning. ### Security - **The shipped image now installs exactly the dependency versions CI tested (audit 2026-07-15 Sec-B).** `pip install .` re-resolved everything fresh from PyPI at build time, so every release image carried versions no test had ever seen — and was exposed to a dependency hijack on each build. The Dockerfile now installs `requirements-docker.txt` (exported from uv.lock, hashes included) with `--require-hashes`, kept in lockstep by a deploy-lint check; the base image is digest-pinned; project installs are `--no-deps`. Verified by a real build + import smoke test. The stub-package phase-1 hack became unnecessary and is gone. - **The release pipeline gates on tests and version lockstep (audit T-2).** `release.yml` never ran a single test, and unlike `publish-image.yml` it never checked the tag against pyproject — a tag pushed on a stale pyproject published a GHCR image whose name lies about its contents (a poisoned rollback target, forever). New `verify` (tag↔pyproject) and `test` jobs gate both publishers; a workflow-shape lint keeps them there. - **Buildx provenance attestations are ON in both image workflows (audit Sec-C)** — the deploy pulls by mutable tag from an automated timer, and the attestation is the counter-signal tracing a pulled image back to the workflow run that built it. - **`publish-image.yml` no longer splices the workflow input into its shell script (audit LOW-1)** — it's attacker-shaped text in a job holding `packages:write`; it now flows through `env:` only, the job asserts HEAD is actually sitting on the tag, and a lint bans GitHub-expression splices in `run:` blocks across all workflows. - **Bearer tokens are no longer passed on curl's argv (audit 2026-07-15).** `-H "Authorization: Bearer $TOKEN"` is world-readable in `/proc/*/cmdline` for the lifetime of every timer-driven request. All scripts now pass the header via a `/dev/fd` curl config (`-K <(printf ...)`), and a lint keeps argv clean fleet-wide. - **`deploy-broker.sh` validates its target before `sed`-ing it into the `.env` that holds the API token** — the target arrives from argv or the GitHub API, and anything but a plain `X.Y.Z` now dies before touching state. The `.env` is also `chmod 600`-enforced, and the health gate honors a configured `JOBD_PORT` instead of hardcoding `:8765` (a non-default port made the gate unpassable — and the deploy un-rollback-able — by construction). - **The Prometheus event counter's `event` label is allowlisted (audit 2026-07-15 Sec-A).** `POST /events` takes free-form event names (hooks name their own events — by design), and `EVENTS_TOTAL.labels(event=...)` minted a **permanent time series per distinct name**: any token holder, or one buggy hook loop, could grow the broker's memory and every `/metrics` scrape without bound. Names outside the new `models.KNOWN_EVENTS` vocabulary now share one `event="other"` bucket (events.jsonl keeps the real name at full fidelity), `EventIngest.event` is length-bounded (64), and an AST sweep test proves every name emitted anywhere in the source is in the vocabulary — a broker event missing from it would silently alert-rate as "other". ### Changed - The "is this job idempotent?" policy parse is now one helper (`broker.state._job_requires_idempotent`) instead of three pasted copies across the sweeper and reconcile (audit L-5); the dead `max_concurrent` parameter fell off `_reserve_and_dispatch` (audit L-6 — the real gate reads `len(_in_flight)` in the poll loop). ### Added - **Real-execution tests for the CD pipeline (`tests/test_cd_scripts_execution.py`, audit T-LOW-2/3).** deploy-broker.sh runs against a stubbed `docker`/`curl` PATH: happy path (pull-before-pin ordering, pin written, metrics, 0600), **the rollback path** (health gate fails → previous pin restored → second `compose up` → exit 1) — logic that otherwise first executes during a failed production deploy — plus injection/`latest` refusal and `DRY_RUN` inertness. update-worker.sh: pinned dry-run, busy-defer on a real job row, other-hosts-ignored, and the unreachable-broker diagnostic. `healthcheck.py` is finally _executed_, not grepped: green on a real jobd payload, red on an alien daemon, an HTTP error, and a dead socket. The systemd `Environment=` lint now covers `*.timer` units too. - **Worker CD (`scripts/update-worker.sh` + a 15-minute user timer) — the workers had no deployment story at all.** The broker self-deploys; the workers were upgraded only when a human remembered to restart them, so they drifted. That is not cosmetic: the `cwd_missing` pre-dispatch check shipped in **v0.5.10 on 2026-06-30 and did nothing for twelve days**, because the workers were still on 0.5.3/0.5.7. Nine jobs died of the exact fault it had been written to prevent, and CI was green the whole time. **A fix that never reaches a worker is not a fix.** The one dangerous power the updater has is `systemctl restart`, which SIGTERMs any in-flight job (60s drain, then `worker_shutdown`) — that is how 13 jobs died on 07-13/14, every one of them killed by a human upgrading a worker. So **it never restarts a busy worker**: it asks the broker (the only thing that knows) for its own `running` count and defers to the next tick, and re-checks _after_ the install, because the worker long-polls and can claim a job during it. It targets the **broker's** version rather than PyPI latest — a worker must never run ahead of its broker, which is the schema authority. Guarded by mutation-verified lint (removing the gate, dropping the post-install re-check, and unpinning the version all fail CI), plus `bash -n` over every script in `scripts/` — because the text-matching lints would happily pass a file that is not a valid shell script at all, and did. - **Fixed: the worker updater crashlooped on every host (exit 127), invisibly.** Two wrong assumptions, each caught only by _running_ it — the text lints in `test_deploy_lint.py` read the script and never execute it, so both shipped green: 1. **pip everywhere.** The install used `"$VENV/bin/pip"`, but the laptop's venv is uv-managed and has _no pip binary_ — `command not found`, 127, on every tick. 2. **uv everywhere.** The first fix switched to `uv`, but three of four hosts have a plain pip venv and _no uv installed at all_ — so a uv-only install 127s on exactly the hosts pip worked on. The installer now **prefers the venv's own pip and falls back to uv** (resolved to an absolute path — `~/.local/bin` is not on the systemd user-service PATH), so it self-adapts to both venv shapes with no per-host config. Separately, a deploy helper had injected `Environment=JOBD_WORKER_VENV=$HOME/jobd-worker/.venv` into three update units; **systemd does not expand `$HOME`**, so the script looked for jobd under a literal `$HOME` path and found nothing (`installed=none`). Fixed to absolute paths and guarded: a new lint rejects any shell variable in a committed unit's `Environment=`, and a new lint requires both the pip and uv install paths to be present. **The lesson is real-execution testing: a guard that never engages the real environment cannot see a missing binary.** ## [0.5.25] — 2026-07-14 ### Fixed - **A live worker's job ran four hours past its own obituary, and its result was thrown away.** The laptop worker's heartbeat lapsed (a WSL freeze). The sweeper did exactly its job and orphaned the in-flight job as `worker_died`. But the worker had not died — it came back, and it had never stopped running the job. It then reported that job in `in_flight_job_ids` on **every heartbeat for three hours**, and the broker ignored it every time. `_reconcile_worker_in_flight` only ever queried rows it already believed were `ASSIGNED`/`RUNNING`. Terminal rows were outside its `WHERE` clause, so reported ids matching no such row were dropped on the floor. It answered _"the broker claims this is running — is it?"_ and never the mirror question, _"the WORKER claims this is running — is it?"_ Four hours of GPU folds ran to completion and the `/complete` was discarded by the terminal-is-terminal CAS; the ledger recorded a death at 17:56 while the GPU was at 99%. The reconcile now runs **both** directions. A job is resurrected only when the worker reports it in flight _now_, the broker has it as `ORPHANED` (never completed/failed/cancelled), the reason is one whose entire premise is "the worker is gone" (`worker_died`/`worker_restarted`), and it is still owned by that worker — never one re-dispatched elsewhere, which would put two live copies in flight. A user's cancel is deliberately **not** undoable: the worker learns of it via `GET /jobs/{id}/signal` and kills the workload. Resurrecting a parent also **undoes the cascade its orphaning caused** — otherwise the parent completes while the children cancelled _because it died_ stay dead forever. `_cascade_on_parent_terminal` stamps each child with `parent_failed: {id} → {state}`, so its work is identifiable and invertible (a user's cancel has no such stamp and is untouched). Transitive, and a child with another genuinely-failed parent stays cancelled. Emits `job_resurrected` / `job_uncancelled`. Mutation-verified against 5 mutations, including restoring the original bug. ## [0.5.24] — 2026-07-14 ### Fixed - **`_default.defaults` was a fallback, not a floor — so configuring a project silently disarmed its zombie hang-guard.** `resolve_project_defaults` did `projects.get(name) or projects.get("_default")`, an either/or: a project _with_ an entry never saw `_default.defaults` at all. That block is the fleet hang-guard — `idle_timeout_s` and `max_wall_s`, the reaper written after a silent job held a desktop GPU for six days — and `config/projects.yaml` described it as "the FLEET-WIDE hang-guard: any unlisted project inherits them". It reached exactly the projects nobody had configured, and fell off the moment anyone gave a project a priority: `job projects set NAME 70` writes a bare `{priority: 70}`, and that alone was enough. On 2026-07-14, registering 32 real projects to set their priorities disarmed the hang-guard on all 32 — including `project-x`, the project whose 170-hour zombie _created_ the guard. Nothing failed and no test went red; the guard was simply gone from everything that mattered, while the config file went on claiming it was fleet-wide. `_default.defaults` is now a **floor**: merged under every project, overridden one key at a time (setting `idle_timeout_s` no longer drops `max_wall_s`). The merge is derived from `dataclasses.fields`, so a field added later is inherited without anyone remembering to add it. `ProjectDefaults.escalate_to_arc` changes from `bool = False` to `bool | None = None`: every field now uses `None` as its unset sentinel, because a merge cannot otherwise tell "the project said nothing" from "the project said false" — and a project must be able to opt out of a floor. Guarded by `tests/test_project_defaults_floor.py`, mutation-verified against the old either/or. - **`test_uv_lock_in_sync_with_pyproject` could only ever pass in CI, and the committed lock was stale through seven releases.** `uv run` syncs before it runs: handed a stale `uv.lock` it rewrites the file, _then_ starts pytest. So the guard shelled out to `uv lock --check` against a lockfile its own runner had just repaired. `uv.lock` sat at `jobd 0.5.16` while `pyproject.toml` said `0.5.23`, with CI green the whole way; the staleness only surfaced running pytest directly. Fixed by setting `UV_NO_SYNC: "1"` once at the workflow level in `ci.yml` — not on individual `uv run` lines, because the next `uv run` someone adds would not have it — and re-locking. Verified both directions on a pristine checkout: with the rewrite a stale lock passes, with `UV_NO_SYNC` it fails as it always should have. ## [0.5.23] — 2026-07-14 ### Fixed - **The `/livez` and `/readyz` probes added in 0.5.22 cleared the bearer-token wall but not the tailnet ACL — and there are two walls.** They were exempt from the token and declared usable by external monitors; they were not. The blackbox exporter, a bridge container with source `172.20.0.4`, got `403 Forbidden`, so the probes were useless for the monitoring that was their entire justification. A probe that clears one of two walls is exactly as unreachable as one that clears neither. Every unit test passed, because the TestClient's source IP is allow-listed and the ACL never engages in the suite; only a real container revealed it. The tell was a contrast: Prometheus, on the _same_ docker network, scrapes `/metrics` happily — because `/metrics` had been exempted from **both** walls. The fix is **one list consulted by both walls** (`_PUBLIC_PROBE_PATHS`), not a second parallel exemption — which would have re-created the same defect one refactor later. Safe by precedent and by content: `/metrics` is already ACL-exempt and exposes far more (job counts, worker versions, per-state gauges) than these two, which are mute. The listener is still bound to the tailscale IP, so reachability is unchanged; this only stops the ACL rejecting monitors that can already reach the port. Guarded by a test asserting a probe is reachable from a source that is neither tailnet nor loopback — because a monitor is, definitionally, not on the tailnet. ## [0.5.22] — 2026-07-14 ### Added - **`GET /livez` and `GET /readyz` — unauthenticated liveness and readiness probes.** Not a convention import: **Uptime Kuma monitors twelve homelab services and jobd was the only one it did not**, because every route sits behind a bearer token and a generic HTTP monitor cannot send one. The broker's log had zero 401s — nothing was even trying. `/livez` says the process is up and deliberately does not touch the database; `/readyz` round-trips a query to SQLite and returns 503 if it cannot. Two rather than one, because the distinction earns its keep when things break: `/livez` green with `/readyz` red is "the process is fine, the database is wedged" — fix the DB rather than restart the process, since restarting destroys the evidence. These are the **only** unauthenticated routes on the broker and they are deliberately mute — alive-or-not, ready-or-not, with no version, counts, or job data. The tailnet ACL still gates the source IP exactly as it does for the (also unauthenticated) `/metrics`, so the trust boundary is unchanged. `/health` keeps both its auth and its version, because the container healthcheck must _prove_ it is talking to jobd and not to whatever else may be listening on the port. The exemption matches by **exact path, never a prefix**, and a derived guard enumerates the live route table to assert every other route still requires a token. ### Notes - **Composite indexes were considered and rejected, on measurement rather than instinct.** Against the live 2,947-row database, the retention prune runs in 1.5 ms, a project+state job list in 1.0 ms, and the dispatch scan in 0.005 ms — all already index-backed by `ix_jobs_state`. A composite index would add write cost to _every_ job transition to save under a millisecond on queries that run every 30 seconds or on user request. ## [0.5.21] — 2026-07-14 ### Added - **`jobd_events` MCP tool.** The broker's event stream is the only surface that explains _why_ rather than _what_: `/jobs` says a job is queued; only the events say it has been skipped 400 times because no worker advertises `cuda-32gb`. It was reachable from the CLI and the HTTP API but not from MCP. Nine tools now. ### Fixed - **The MCP error model had a hole an agent falls into.** `JobdClient` raises `BrokerRefusal` for _any_ 4xx, but the rule table covered only 400/404/409 — so **401** (bad token), **403** (tailnet ACL) and **422** (malformed body) fell through to `kind: "unknown"` with the hint _"Unmapped broker 401: …"_. An LLM handed that has nothing to act on and, worse, no way to know the failure is _permanent_, leaving it free to retry a bad token forever. All three now map to a real kind whose hint says so explicitly. A 400 with an unrecognised message maps to `bad_request` for the same reason: the message may be novel, but "re-sending these arguments will not work" is known. Verified against a live broker — a genuinely bad token now returns `kind: auth_failed`. - **The worker logged 41 diagnostics with `print(..., file=sys.stderr)`, and systemd filed every one of them as INFO.** A stderr stream gets a single journald priority (`SyslogLevel=info` by default), so `PRIORITY=6 (info) [worker] heartbeat error: [Errno 111] Connection refused` is what was actually in the journal. **`journalctl -p err` returned nothing while the worker was failing to reach the broker at all**, and no severity-based alert could ever have seen it. Now `log.info`/`warning`/`error` with levels chosen per message (8/10/23), and a formatter emitting systemd's `` priority prefix — real journald priorities with no new dependency, and only when systemd is consuming the stream. Verified on a live worker: a genuine "cwd missing here; refusing admission" is now `PRIORITY=4 (warning)` and `journalctl -p warning` surfaces it. ### Changed - **MCP↔broker surface parity is now DERIVED rather than remembered.** `tests/mcp/test_surface_parity.py` reads the route table off the live FastAPI app and asserts every route is either mapped to a tool or named in `_NOT_ON_MCP` **with the reason it should not be an agent action**. A route added tomorrow that nobody exposes now _fails a test_ instead of quietly living on one surface — the recurring drift the 2026-07-12 audit kept finding. The converse is enforced too: a registered tool covering no route fails, because every tool costs model attention (a duplicate `jobd_job_get` was deleted in 0.5.16 for exactly that). Same shape as the `_SUBMIT_SYNTHESIZED` deny-list: coverage derived, exclusions named, **forgetting is what fails**. ## [0.5.20] — 2026-07-13 ### Fixed - **The blocked-warning was its own dedup key, and it carried a clock.** The sweeper re-emits a warning only when it differs from the one stored on the job (`if j.warning != new_w`) — but the blocked-warning rendered as `queue-age 47m: blocked by non-preemptible job N on ...`, and that `47m` ticks by itself. So the comparison was true on _every_ sweep, forever: a job blocked behind a non-preemptible neighbour re-emitted a `sweep_warning` event **and rewrote its DB row every minute it stayed blocked**. In production that was 974 events over 7 days, one job accounting for 498 of them. The diagnosis is self-proving: the _unmatcheable_ warning runs through the same dedup code three lines away, carries no clock, and deduped perfectly — one event, not 561. Identical mechanism, opposite outcome, and the only difference is the ticking value in the key. The queue age is not lost: it is derivable from `submitted_at`, and rendering it at read time is strictly _more_ accurate than a number frozen at whenever the last sweep ran. What the warning is _for_ is the blocker's identity and the remedy, and neither changes minute to minute. The prefix moves `"queue-age "` → `"blocked: "` accordingly. This is the third instance of one class — a guard whose key cannot capture what it must distinguish (see also the `dispatch_skip` dedup in 0.5.17, and the healthcheck in 0.5.17 that could not tell _which_ daemon answered). A sweep of the whole event surface by repeat-count per (event, job) confirms there is no fourth: every other event fires at most twice for a given job, i.e. once per state transition. ## [0.5.19] — 2026-07-13 ### Changed - **Services, not routes.** The two fat route closures are now callable services: `jobd.broker.submit.submit_job` (~295 lines — validation, the effective-config cascade, array fan-out, persistence, the H-1 TOCTOU cascade sweep, event emission) and `jobd.broker.admission.refuse_admission` (~250 lines — the admission decision tree: stale-worker rejection, the ASSIGNED guard, pending-cancel honouring, and the requeue / terminal-fail / cwd-exclusion / auto-preempt branches). The routes are thin adapters. **`app.py`: 1,873 → 1,351 lines (−28%).** No behavior change. Worth recording _how_ this was verified, because a passing suite could not have shown it. The tests guarding the two worst bugs this project has shipped work by monkeypatching module-level names **through the module that reads them** (`_serialization_warning`, `_FAILED_SIDE_TERMINAL`). Moving `submit` out of `jobd.app` breaks those patches — and the failure mode is not a red suite, it is **both tests continuing to pass while testing nothing**. That is exactly what the 2026-07-12 audit found: an H-1 fix that was a silent no-op in production while its test stayed green (0.5.13). So the refactor was gated on mutation instead: re-inject each historical bug and require the named test to _fail_, both before the move (proving the guards were alive) and after (proving they still are). `tests/unit/test_submit_service_guards.py` makes that permanent — the patch targets must demonstrably intercept, and `jobd.app` must not re-expose the stale names, since a direct assignment there would silently create an attribute nothing reads. ### Fixed - **One safe way to build a job log path** (`py/path-injection`, high). Four places were composing `logs_dir / f"{job_id}.log"` by hand and had already drifted — two raw interpolations, one ad-hoc `int()` cast added to quiet this very warning, and a differently-named variable in the sweeper. The traversal was not reachable (FastAPI types `job_id` as `int`, so `../` is rejected with a 422 several frames earlier), but that made the safety a property of an annotation in another module and invisible at the filesystem call. `job_log_path()` is now the only way to name a job's log file, and coercion inside it makes the guarantee local and total. ## [0.5.18] — 2026-07-13 ### Added - **The broker image is now published to GHCR** (`ghcr.io/musharna/jobd:X.Y.Z`) on every `vX.Y.Z` tag, and `docker-compose.yml` pulls a **version-pinned** image instead of building in place. This removes the `--build` footgun rather than guarding against it. Compose used to have `build: .` + `image: jobd:latest` with **no registry behind that tag**, which meant `docker compose pull` was a silent no-op and a bare `docker compose up -d` after a `git pull` found a `jobd:latest` already present and reused it — restarting the _old_ code, with nothing to say so. The mitigation was to remember `--build` forever. Now `pull` means something, the running version is knowable and pinned, and **rollback is possible at all** (you cannot roll back to a moving tag). Building from source is still supported, deliberately, via `docker-compose.build.yml`. - **Pull-based continuous deployment** (`scripts/deploy-broker.sh` + `jobd-deploy.timer`). A timer on the broker host checks every 5 minutes for a newly published release and deploys it. Pull-based on purpose: the broker host has no public ingress, so push-based CD would mean storing a tailnet auth key and an SSH key in GitHub and letting CI reach into the network. **No secrets in GitHub, no inbound access**, and the check is nearly free when there is nothing to do. The deploy **verifies itself**, and each property is a bug this project has actually shipped: it pins an exact version rather than following `latest`; it confirms success by asking the _broker_ its version rather than treating a clean `docker compose up` as proof the new code is serving; and it health-gates on `/health` rather than a socket connect. If the broker does not report the deployed version within 90s, **the previous pin is restored automatically**. Deploy success and the running version are exported as node-exporter textfile metrics, so a failed deploy is visible in Prometheus instead of only in a journal nobody reads. `DRY_RUN=1` reports what it would do and changes nothing. Rollback is now `sudo jobd-deploy 0.5.17`. ### Notes - The GHCR package publishes **public**, so the broker host pulls it with no credentials at all — verified: the host has no `~/.docker/config.json` and ghcr.io issues an anonymous pull token for the image. Nothing in the deploy path holds a secret. ## [0.5.17] — 2026-07-13 ### Fixed - **The `dispatch_skip` event storm: 54,118 events to serve 139 dispatches.** Framed in the backlog as a thundering-herd problem, it was really a dedup-key defect. `explain_skip(jobs, w)` computes its reason against a _specific worker_, so one queued job legitimately has different reasons on different hosts — but the dedup state was keyed by `job_id` **alone** and shared across all workers. Each worker's answer overwrote the previous one, so the "has the reason changed?" guard was true on _every_ poll and the event fired every time any worker polled. Two permanently-unplaceable jobs produced 3,908 of 5,340 skip events in one sample, each an append to `events.jsonl`. Now keyed on `(job_id, worker_host)`. The GC had the same shape of bug — `queued` is already filtered per-worker, so evicting entries for jobs merely invisible to _this_ worker discarded other hosts' state and made them re-emit — and is now scoped to the polling worker. - **The container healthcheck never probed the broker, and passed anyway.** It TCP-connected to a hardcoded `127.0.0.1`, but the broker runs `network_mode: host` and binds `$JOBD_HOST`; it has never listened on loopback. It reported green because an unrelated container held `127.0.0.1:8765` and **a bare TCP connect cannot tell which daemon accepted the socket**. Moving that container off the port exposed it: the probe began failing against a broker that was serving perfectly. It now makes an authenticated HTTP request to `$JOBD_HOST` and requires jobd's own `/health` payload back, so a squatting daemon fails the check rather than satisfying it. - **N+1 dependency reads on every claim attempt** — one point-read per parent, per queued job, per attempt, per parked worker. Now a single batched `SELECT id, state ... WHERE id IN (...)`. `_deps_satisfied_bulk` is _fresher_ than the loop it replaces, not merely cheaper: it bypasses the identity map, whereas `session.get()` under `expire_on_commit=False` can return a cached pre-commit state — precisely how the original H-1 cascade fix became a silent no-op (see 0.5.13). `_deps_satisfied` now delegates to it, so there is one copy of the terminal-state policy rather than two that can drift apart. ### Changed - **The sweeper no longer wakes every parked worker every 30 seconds regardless of whether anything happened.** It broadcast a wake on every pass; each parked `/next-job` then ran a full claim attempt against an unchanged queue, and all but at most one returned nothing. It now wakes only when the sweep actually changed state. Safe by construction: parked workers re-attempt every `_LONGPOLL_RECHECK_S` (10s) as a backstop, so an over-narrow predicate costs bounded latency, never a stuck job. ### Notes - Deliberately **not** done: a composite `(state, priority, submitted_at)` index, and a rewrite of the wake mechanism to wake exactly _K_ workers. At a queue depth of 1–2 with four workers, the index would add write cost to every job transition to speed up a scan that is not slow, and coalescing would risk dispatch stalls to solve a fan-out that does not hurt at this scale. The storm was the pathology, and it is fixed at the cause. ## [0.5.16] — 2026-07-13 ### Fixed - **The MCP surface silently dropped `scheduling_timeout_s`.** It shipped on the broker and the CLI; jobs submitted through MCP could not be given a scheduling deadline, and nothing said so. The guard built to catch exactly this — a test named "round trips every JobSubmit field" — _could not fire_: it compared a hand-written passthrough tuple in `translate.py` against a hand-written expected set in the test, so both halves had to be remembered when `JobSubmit` grew a field, and neither was. The test only ever checked the fields someone had already remembered. The fix inverts the direction. The MCP submit surface is now **deny-listed, not allow-listed**: `_SUBMIT_PASSTHROUGH` is _derived_ from `JobSubmit.model_fields` minus a documented `_SUBMIT_SYNTHESIZED` list, each entry carrying the reason it is synthesized rather than forwarded. A new broker field is now either forwarded automatically or it must be named with a justification — **forgetting is what fails**. Forwarding `scheduling_timeout_s` and collapsing `arch`/`os` into `requires` both fall out of that change rather than needing their own patches. - **The systemd unit deployed stale code on boot.** `docker-compose.yml` pairs `build: .` with `image: jobd:latest` and there is no registry behind that tag, so `docker compose pull` is a no-op and `up -d` after a `git pull` re-runs the _old_ image. `ExecStart` now passes `--build`; the runbook says why, and names the tell (a `job ping` version that disagrees with the checked-out tag). ### Added - **Workers report their version in the heartbeat.** The fleet is upgraded host-by-host over SSH, so it routinely runs mixed versions — and the broker had no way to know: a worker three releases stale was indistinguishable from a current one. `Worker.version` (additive migration) now surfaces on `/workers` and as `jobd_worker_version_info{host,version}`. Paired with the existing `jobd_build_info`, drift becomes _alertable_, not merely visible: ```promql count(jobd_worker_version_info unless on(version) jobd_build_info) > 0 # a worker is adrift count(count by (version)(jobd_worker_version_info)) > 1 # fleet is not uniform ``` A worker too old to report reads as `null` / `version="unknown"` rather than inheriting the broker's version or vanishing from the series — the absence is itself the answer, and a missing series would shrink the drift count instead of raising it. The version is also **not** pinned to its last-known value on downgrade: a registry that makes the fleet look newer than it is would be worse than not knowing. ### Removed - **`jobd_job_get` (MCP tool) and `client.job_get()`.** `jobd_job_get` was byte-identical to `jobd_status`'s no-wait path — both `GET /jobs/{id}` — while its description advertised fields it implied `jobd_status` lacked. It spent 1/9 of the MCP tool budget actively misdirecting tool selection. `jobd_status`'s description now states the scheduling internals it has always returned. Use `jobd_status` / `status()`. ## [0.5.15] — 2026-07-12 ### Added - **Split retention: job rows and job logs now prune on independent clocks.** Retention had been off entirely, and for a good reason — one shared clock forced a false choice. Measured on the live broker: 2,875 job **rows** = **6.4 MB** (~7 MB/yr), but 2,605 job **logs** = **2.0 GB** (~0.7 GB/month). A row is ~2 KB and feeds the ETA estimator's per-project p50/p90, so it is cheap and gets _more_ useful with age; a log is ~800 KB and is write-once-read-maybe. Pruning them together meant discarding cheap, valuable history purely to reclaim expensive disk. Now: - **`JOBD_LOG_RETENTION_DAYS`** (default **60**) unlinks the `.log` of a terminal job finished more than N days ago while **keeping the row** — this is what bounds disk. Emits `logs_pruned`. `0` disables. - **`JOBD_JOB_RETENTION_DAYS`** (default **0**, unchanged) deletes the terminal row itself. Still opt-in, because rows are nearly free and the estimator wants the history. - **`/jobs/{id}/output` reports `pruned`.** A pruned log and a never-written one both leave no file on disk, but they mean opposite things — reporting a pruned log as empty output would make a job that emitted megabytes look like it produced nothing. `job logs` now says the log was pruned by retention and points at `job status`. ### Changed - Jobs carry a `log_pruned_at` stamp (additive migration), so the log-prune scan shrinks monotonically instead of re-stat'ing every historical log on every 30s sweep. A log whose unlink genuinely fails is left unstamped and retried. ## [0.5.14] — 2026-07-12 First batch from the 2026-07-12 improvement audit. Headline: `job projects set` / `nudge` were returning HTTP 500 in production — a documented feature was dead. ### Fixed - **`job projects set` / `job projects nudge` returned HTTP 500 (config was mounted read-only).** `_persist_projects` rewrote `config/projects.yaml` in place, but that file is git-owned _and_ bind-mounted `:ro` into the broker container, so every priority mutation raised `OSError: Read-only file system`. Verified against the live broker. Config ownership is now split: `config/projects.yaml` stays the **git-owned, read-only baseline** (projects, their baseline priority, and their `defaults:`), while runtime priority changes persist to a **writable overlay** at `$JOBD_STATE_DIR/project-priorities.yaml` (defaulting to the SQLite DB's directory, so it is covered by the DB backup and never conflicts with a `git merge` on redeploy). The overlay stores only _deltas_ from the baseline, so a later config-as-code priority change still lands for any project nobody nudged. Because the endpoints only ever touch `priority`, `defaults:` blocks are now git-only — making the old "one nudge silently erases every `defaults:` block" round-trip hazard structurally impossible rather than merely tested against. See the new "Config vs state" section in `docs/runbook.md`. - **Fleet-wide hang-guard `_default` defaults are now committed.** `idle_timeout_s: 3600` / `max_wall_s: 172800` existed only as an uncommitted edit on the broker host — one `git clone` away from being silently lost. ### Added - **`GET /jobs` pagination.** The endpoint used to return _every row ever_ — on a broker with retention off, the entire history. It now accepts `limit` (1..1000) and `offset`, and always reports the full filtered count in the `X-Total-Count` header. `limit` is deliberately opt-in (absent = all), because `graph` and `--array` build over the complete set and a silent default cap at the API layer would quietly corrupt them. - **`job list --limit/-n` (default 50) and `--all`.** `job list` used to dump the broker's entire job history. It is now bounded by default and prints `… showing 50 of N` rather than truncating silently. `--array` still shows every member of an array (a truncated array misrepresents its shape). The MCP surface had already capped its own output; the human CLI had not. - **`job --version` / `-V`**, and a bare `job` now prints the full help (with the command list) instead of a terse usage error. `JOBD_URL` and `JOBD_API_TOKEN` are documented in `job --help`, and the six commands that rendered as blank rows (`cancel`, `wait`, `classify`, `projects list/set/nudge`) now have help text. ### Changed - **SQLAlchemy connection pool sized above the threadpools that feed it.** The engine used QueuePool's defaults (`pool_size=5` + `max_overflow=10` = 15 connections) while _two_ threadpools open sessions against it — anyio's (40 tokens; every sync endpoint) and `asyncio.to_thread`'s (the `/next-job` dispatch scan). Past 15 concurrent DB-touching requests, the surplus threads blocked on pool checkout for up to `pool_timeout` (30s) — a cliff reached exactly when the dispatch fan-out wakes every parked worker at once. Now 20/60, tunable via `JOBD_DB_POOL_SIZE` / `JOBD_DB_MAX_OVERFLOW`. ### Removed - **`test_full_suite_green`**, a meta-test that shelled out to re-run the entire suite as a subprocess. It contributed zero coverage, ignored the outer `-m`/`-k`/`-x` filters, doubled the cost of every new test, and its own docstring scoped it to a migration that finished when `mcp-v1` was tagged. CI already runs the full suite. **Suite runtime: ~136s → ~66s.** ## [0.5.13] — 2026-07-12 Fixes from the 2026-07-12 focused re-audit of the v0.5.12 delta. Headline: the v0.5.12 H-1 fix was a no-op in production and is now actually closed. ### Fixed - **Submit `depends_on` TOCTOU fix was a no-op (H-1, regression of the v0.5.12 fix).** The v0.5.12 post-commit cascade re-check read the parent through a cached identity-map object; with `expire_on_commit=False` the intervening commit never refreshed it, so the cascade gate saw a stale non-terminal state and never fired — a child concurrently orphaned by its parent still stranded QUEUED forever. `/submit` now `session.refresh()`es each parent before the cascade re-check (matching the idiom every other `_cascade_on_parent_terminal` caller already uses), guarded for a parent pruned mid-submit. New real-race regression test drives the actual `/submit` endpoint with a concurrent-session `/complete` and fails on the pre-fix code. - **`scheduling_timeout` queue-clock reset on refuse-admission re-routes (M-1 follow-up).** A refuse-admission requeue funnels through the same helper that resets `last_enqueued_at`, so a job that never ran but kept getting offered-and-refused (e.g. oscillating `gpu_contention`, which does not grow the exclusion set) reset its timeout clock every bounce and could evade `scheduling_timeout` forever. Refuse-admission re-routes now preserve the clock; only requeues of a job that actually dispatched reset it. - **Worker `run_job` exception path leaked a signal-poll thread and a fast_path child (F1/F2).** On a raise past the terminal path, the outer `finally` now stops the signal-poll thread (it would otherwise spin on `GET /signal` forever for a terminal job) and SIGKILLs a `fast_path` child that has no scope cgroup to reap. ### CI - **Property-based fuzzing of the broker state machine (H-4, audit 2026-07-10).** A `hypothesis` `RuleBasedStateMachine` (`tests/property/`) generates arbitrary interleavings of submit / claim / start / complete / cancel / worker-death-reclaim / scheduling-timeout / sweep against the real broker (in-process FastAPI + SQLite + sweeper) and asserts the invariants the CAS discipline guarantees: no terminal state is ever clobbered, a job is terminally cancelled at most once, a reclaim never silently drops a pending user cancel, and (added 2026-07-12) a default-policy dependent of a failed parent is never stranded on a runnable path. The harness now generates `depends_on` edges so it actually exercises the cascade (previously it created none, leaving the cancel-at-most-once invariant vacuous). Runs bounded in CI (`JOBD_H4_EXAMPLES` / `JOBD_H4_STEPS` env overrides for a deep run). - **The `live` CI job silently skipped 5 of 9 marked-live tests (H-3 follow-up).** It exported only `JOBD_LIVE`, while the e2e / MCP / project-defaults live tests gate on `JOBD_E2E` / `RUN_LIVE_JOBD` and require an external broker — so they skipped, and skips don't fail pytest. The job now stands up a real broker + worker from the committed `config/`, exports all three opt-in gates, and guards against a broken gate silently running zero tests. ## [0.5.12] — 2026-07-10 Fixes from the 2026-07-10 v0.5.11 4-reviewer audit (security / broker-correctness / worker-reliability / hygiene). ### Fixed - **Submit `depends_on` TOCTOU → permanent QUEUED strand (H-1).** The failed-side reject in `/submit` is a point read of `parent.state`; between it and the child INSERT+commit a parent could reach a failed-side terminal and run its cascade over a QUEUED set that didn't yet include the child, stranding a default-policy child QUEUED forever (no future transition re-fires the parent's cascade). `/submit` now re-runs the cascade for each parent after the child is committed and visible — a no-op unless the parent is already failed-side terminal. - **`run_job` terminal path unguarded → strand + tracked-pid leak + un-reaped scope (H-2).** A `proc.stdout.read()` pipe/closed-file race or a signal-thread start failure raised past the terminal path; the outer dispatch handler only logged. The post-Popen body is now wrapped so a terminal `/complete(failed, worker_exec_error)` + tracked-pid discard + best-effort scope kill always run when the body doesn't reach its own terminal post. - **Reclaimed long-running job killed by `scheduling_timeout` (M-1).** The sweeper keyed the deadline on `submitted_at`, which requeue paths copied forward — a job that dispatched, ran for hours, then had its worker die was immediately SCHEDULING_TIMEOUT'd and its dependents cascade-cancelled. A new `last_enqueued_at` column (additive migration) is reset on every requeue and now keys the timeout; `submitted_at` (which orders the queue) is untouched. NULL falls back to `submitted_at`. ### Security - **Interactive docs disabled.** `/docs`, `/redoc`, and `/openapi.json` are Starlette-mounted and bypassed the app-level `require_token`, letting a tokenless tailnet peer enumerate the API. Now off (`docs_url`/`redoc_url`/`openapi_url` = `None`). ### Added - **Prometheus `jobd_events_total{event,source}` counter (M-3).** The gauges were point-in-time; cumulative failure/throughput signals (cancellations, cascades, preempts, refusals) are now rate-alertable, incremented at the single `_emit_event` choke point. ### CI - **The live worker↔broker suite gates merges (H-3).** A new `live` job (`JOBD_LIVE=1`) runs the real 2-process contract tests, previously deselected on every merge — the gap that let H-1/H-2 ship. ## [0.5.11] — 2026-07-05 ### Changed - **Quality batch from the 2026-07-05 audit (A4/A5 + LOWs).** (a) The dependency-cascade terminal sets in `broker/constants.py` are now _derived_ from the `models.py` single-source sets instead of hand-re-enumerated — a new terminal state can no longer silently miss the cascade (the H2 stranded-children class). (b) `build_app` no longer injects its test seams into module `globals()` (where the last-built app silently won module-wide and pinned its engine alive); they live on `app.state` (`sweep_once` / `prune_old_jobs` / `engine`) and all ~100 test references were updated. (c) `POST /jobs/{id}/complete` takes a typed `CompletePayload` (an untyped dict let a malformed worker payload store a non-int `exit_code`). (d) Dead `JobState.PREEMPT_REQUESTED` removed (nothing ever set it, but it sat inside `NON_TERMINAL_STATES`). (e) `jobd_workers{state="stale"}` is a fixed metrics series instead of appearing/disappearing. (f) CLI: `preempt-blockers` exits 3 (not 2, which means broker-unreachable) for the benign "no blocker signaled" outcome; the exit-code map is documented in `main()`; `list` loses its `-w` short flag (`-w` means wait/watch elsewhere); `cancel`/`workers`/`gpu-holders` use the `_client()` helper. (g) Dead `logs_dir` param dropped from `_cascade_on_parent_terminal`/`_reconcile_worker_in_flight`; stale docstrings (app.py "split at 500 lines" header, missing `sweeper` in the broker package map, `sweep_once` one-liner) and false "string value (not the enum)" StrEnum comments corrected. ### Fixed - **Preempt-signal writes are now CAS-guarded, and the `/next-job` claim clears any leftover signal (audit 2026-07-05, A1).** The sweeper's auto-preempt and `POST /jobs/{id}/preempt-blockers` stamped `signal='preempt'` with plain ORM writes — no guard on the state the candidate was selected in — so a `/complete` or reconcile-requeue landing in the window could carry a stale preempt onto a requeued job; and since the atomic queued→assigned claim never touched `signal`, the fresh worker would read it on its first poll and kill the brand-new run seconds in. Both writers now CAS on `(assigned, running)` (a lost race returns `signaled: null` from `/preempt-blockers` instead of a false success), and the claim UPDATE clears `signal`, killing the stale-signal class outright. - **A pending user cancel is honored, not erased, when a claim is torn down (audit 2026-07-05, A2).** Cancelling an `assigned` job sets `signal='cancel'` and tells the user "cancelling" — but every requeue path (admission refusal ×2, sweeper dead-worker reclaims, heartbeat reconcile) cleared `signal` unconditionally (the M1 fix, aimed at stale _preempt_ signals), so the cancel vanished and the job silently re-ran, possibly on another host. Teardown now goes through a signal-qualified CAS (`_requeue_or_honor_cancel`): a pending cancel transitions the job to `cancelled` (with dependency cascade + `job_cancelled` event, `via=refuse_admission|sweeper_reclaim|reconcile`), while non-cancel signals are still cleared on requeue. `/refuse-admission` also checks first, so a cancelled job can't be mislabeled `failed/cwd_unreachable` on the no-eligible-worker branch. - **`cwd_unreachable` failures now stamp `finished_at` (audit 2026-07-05, F-3).** The terminal transition omitted it, and retention pruning filters on `finished_at` — so these rows (and their logs) were never pruned. Also clears any pending signal, matching `/complete`. - **Sweeper phase-2 events (`auto_preempt`, `sweep_warning`) are emitted only after their commit (audit 2026-07-05, F-4).** They fired mid-transaction, so a failed commit could leave `events.jsonl` claiming a preempt signal or warning the DB never recorded — the same emit-after-commit invariant the M3 fix established for phase 1. - **Worker offline/stale marking re-checks `last_heartbeat` in the write (audit 2026-07-05, F-7).** The sweeper's SELECT-then-set could clobber a heartbeat that committed in between, briefly taking a live worker out of matching and emitting a spurious `worker_offline`/`worker_stale` event. - **Heartbeat reconcile wakes long-pollers when it orphans or cancels jobs (audit 2026-07-05, F-6).** It only woke on requeues; orphans/cancels are failed-side terminals that can unblock `depends_on_any_exit` dependents, which previously waited out the ~10s recheck backstop. - **Concurrent `/next-job` attempts can no longer 500 on the skip-dedup cleanup (audit 2026-07-05, F-5).** Two attempts share the dedup dict; `del` raced (`KeyError`) — now `pop(…, None)`. - **`POST /events` returns 422 (not 500) when a payload key collides with an envelope field (audit 2026-07-05).** A payload containing `source`/`job_id`/`project`/`event`/`ts` blew up the emit call with `TypeError: got multiple values for keyword argument`; it's now rejected at the Pydantic boundary (which also closes the only route to forging envelope fields). - **MCP `jobd_list` now does what its schema says (audit 2026-07-05, A6).** The advertised `limit` (default 50) was never applied anywhere — `GET /jobs` has no window, so on a long-lived broker with retention off (the default) a bare `jobd_list` dumped the entire job history into the model's context — and the documented state default (`queued/assigned/running`) wasn't implemented either (all states returned when omitted; only the _first_ element applied when a multi-state filter was passed). Omitting `state` now returns the active set (explicit `[]` opts into all states), multi-state filters apply client-side, and `limit` caps the jobs array to the newest N (clamped `[1,200]`) with `counts` still covering the full filtered set and a `truncated` field reporting the overflow. - **MCP error mapping matches the broker's real detail strings (audit 2026-07-05, A7).** Two rules were dead — "unknown parent job" and "no eligible worker" match nothing the broker ever emits — while the most common submit hard-fails (the matcher's mount-root cwd rejections) fell through to kind `"unknown"` with no hint. The missing-parent rule now matches the actual `depends_on refers to missing job: N`, a new `parent_failed` rule covers the terminal-parent reject, mount-root failures map to `cwd_outside_mount_roots` with a routing hint, and the dead no-eligible-worker rule is gone. The tests now drive the mapping with details copied from the emitting source, so a broker wording change breaks them. - **The committed `uv.lock` was stale — CI has been running without `pytest-timeout` since it "landed" (audit 2026-07-05).** #16 added `pytest-timeout>=2.2` to the dev extra and set `timeout = 300`, but the lockfile was never regenerated; `uv sync --frozen` (CI) installs the lock verbatim and does **not** validate it against `pyproject.toml`, so the plugin was silently absent and the ini option ignored — no hang protection, the exact gap Tests-1 was meant to close. The lock is regenerated, and two `test_deploy_lint` guards make the failure mode structural: the timeout plugin must be _loaded_ in the running test env, and `uv lock --check` must pass. - **Live/e2e suites now carry the `live` marker, not just an env `skipif` (audit 2026-07-05, L2).** CI's `-m "not live"` was mostly decorative — only 2 of the live tests were marked; the rest relied on per-module env checks, where one typo'd variable name would run a live suite in CI. `tests/test_e2e.py`, `tests/integration/test_broker_concurrency_live.py`, and `test_project_defaults_live.py` are now structurally deselected (capability-gated tests like `test_cancel_latency`/`test_hook_forward` deliberately keep only their skipif — they legitimately run where the capability exists). - **`publish-registry.yml` verifies the `mcp-publisher` download against a pinned sha256 (audit 2026-07-05, L4)** — release tags/assets are mutable, unlike the SHA-pinned actions elsewhere — **and the wait-for-PyPI poll is 10 minutes (L5)** so a slow upload (or a future required reviewer on the `pypi` environment) can't fail the registry publish after the tag is already cut. Also: the metrics cache now has a TTL-_expiry_ regression test (hit-within-TTL and TTL=0 were covered; expiry wasn't), and a stale `# v6.0.3` comment on a v7.0.0 action pin is corrected. ### Security - **`POST /jobs/{id}/log` no longer buffers an oversized body before the size check (audit 2026-07-05, LOW).** `await request.body()` pulled the whole request into memory and only then compared against the 10 MiB cap, so a token-holder could drive broker memory arbitrarily high per request. The body is now rejected from the `Content-Length` header when one is declared, and read in bounded stream chunks with the cap enforced mid-read either way. ## [0.5.10] — 2026-07-03 ### Fixed - **`GET /wait/{id}` streams the log in bounded slices instead of reading the whole backlog into memory (audit 2026-07-01, LOW).** The SSE generator opened the log and did `f.read()` from the last position; on the first iteration `position` is `0`, so attaching `/wait` to a job that had already produced a large log pulled the entire file into one allocation (plus a second full copy on `.decode()`) — for the exact case `/wait` is built for (babysitting a long-running job with lots of output), each connection could balloon broker memory. It now reads in 64 KiB slices (`WAIT_STREAM_CHUNK_BYTES`), reopening the file per slice so no fd is held across an SSE yield, and decodes through an incremental UTF-8 decoder so a multibyte char split across a slice boundary isn't corrupted into replacement chars. A final drain on terminal-state detection also closes a latent race where bytes the worker wrote as it exited (between the read and the state flip) could be dropped. ## [0.5.9] — 2026-07-03 ### Fixed - **Config-mutation endpoints return 422, not 500, on a malformed body (audit 2026-07-01, LOW).** `POST /projects/{name}` and `POST /projects/{name}/nudge` took a raw `dict` and did `int(payload["priority"])` / `int(payload["delta"])`, so a missing key or non-integer value raised `KeyError`/`ValueError` inside the handler → an opaque 500. They now take pydantic request models (`SetPriorityRequest`/`NudgePriorityRequest`), so FastAPI rejects a bad body with a descriptive 422 before the handler runs. The `[0,100]` clamp stays in the handler (out-of-range is clamped, not rejected — unchanged behavior). - **`job` CLI no longer dumps a Python traceback when the broker is down (audit 2026-07-01, LOW).** Most subcommands wrapped their broker call with no error handling, so a `BrokerUnreachable` (broker down / wrong `JOBD_URL`), `BrokerServerError` (5xx), or `BrokerRefusal` (4xx) surfaced as a raw stack trace; only a few (`preempt`, `preempt-blockers`, `delete-worker`, `ping`) handled any of them. The console entry point is now `job_cli.cli:main`, which wraps the Typer app in a broker-error boundary: a one-line stderr diagnostic + a clean exit code (2 for unreachable/5xx, 1 for a 4xx refusal), never a traceback. It's a backstop — per-command handlers still run first and keep their tailored messages; only exceptions they don't catch reach the boundary, and normal `SystemExit`/`typer.Exit` pass through untouched. ### Security - **`/metrics` caches its DB aggregation to blunt unauthenticated scrape amplification (audit 2026-07-01, LOW).** `/metrics` is unauthenticated and tailnet-ACL-exempt by design (an in-cluster Prometheus scrapes from the docker-bridge IP, not a tailnet address), so its per-scrape `GROUP BY state` query was an amplification vector — an unauthenticated caller could drive one DB round-trip per request in a tight loop. The collector now caches the aggregate counts for a short TTL (default 5s, `JOBD_METRICS_CACHE_TTL_S`, set `0` to disable), querying under a lock so a burst that all misses the cache still issues a single query (dogpile-proof). Normal Prometheus scraping (15–30s intervals) is unaffected; the ACL exemption is retained since it's load-bearing for scraping. - **Submitted `env` values are now masked on observability reads (audit 2026-07-01, LOW-Sec).** A job's `env` was stored plaintext and echoed verbatim on every job-info read — `GET /jobs`, `GET /jobs/{id}`, the submit/cancel/preempt responses, and the MCP `jobd_status`/`jobd_job_get` tools (all via `broker.jobinfo._to_info`) — so a token someone passed in `env` leaked to any other token-holder and, worse, into an agent's context on a routine status read. Those surfaces now return `{"KEY": "***"}` (keys preserved so operators can see _which_ vars are set; values hidden). The masking is fail-safe by default in `_to_info`; only the worker-claim `/next-job` path opts out (`redact_env=False`) so the job still runs with the real values. **Not** covered: env is still stored plaintext at rest in `jobs.env_json` (DB-file/backup exposure is out of scope for this change) — see `docs/security.md`. Note: there is no `--env` CLI flag on `job submit` in this tree; env enters only via MCP or a direct `POST /submit`. ### Changed - **Docker build no longer re-downloads dependencies on every source edit (audit 2026-07-01, Tests-2).** The Dockerfile copied `src` above the single `pip install .`, so any code change busted the layer that resolves and downloads all third-party deps. The install is now two-phase: phase 1 installs only the dependencies (against a stub package, keyed on `pyproject.toml`+`README.md`) and phase 2 copies the real source and reinstalls just the package (`--no-deps --force-reinstall`). A source edit now reruns only the fast phase-2 layer; the dependency layer stays cached. Verified by a double-build (phase-1 `CACHED` after a source change) plus an import/entry-point smoke check of the resulting image. A new `.dockerignore` keeps local artifacts (`__pycache__`, `.venv`, caches, `tests/`, `data/`) out of the build context so they can't bust the `COPY src` layer or bloat the image. - **mypy now checks the bodies of un-annotated functions (`check_untyped_defs = true`; audit 2026-07-01, Quality-4).** The `mypy src/jobd src/job_cli` gate was passing but largely vacuous over the hot path: mypy skips the bodies of every def lacking a full signature by default, and the broker/worker (`app.py`, `job_worker.py`) are mostly un-annotated endpoint closures and helpers, so their bodies were never type-checked. Enabling `check_untyped_defs` type-checks those bodies (params default to `Any`, so it catches internal inconsistencies, not missing annotations) and produced **zero** new errors — the code was already clean under it. A `tests/test_deploy_lint.py::test_mypy_checks_untyped_defs` guard keeps the flag from being silently dropped. - **`/submit` and `/resolve` now share one field-resolution cascade (audit 2026-07-01, Quality-3).** Both endpoints hand-encoded the same `CLI > project_default > profile > global` precedence (docs/projects-yaml.md §3) — `/submit` for values, `/resolve` for value+source — and had already drifted: `/resolve`'s profile `host_pin` branch carried a `!= "any"` guard (so a profile `host_hint: any` is not mistaken for a real pin) that `/submit` lacked. The divergence was latent (source-label only; no resolved-value difference today), but any future edit to one copy would silently diverge the other — exactly the class of bug where a dry-run preview stops matching the real submit. The precedence now lives once in `jobd.config.resolve_effective_config`, returning a `(value, source)` per field; `/resolve` surfaces it verbatim and `/submit` reads `.value` (submit-only `vram_gb`/`ram_gb`/`cpus`/`fast_path`, which have no source label and a single consumer, stay inline). A new `tests/test_submit_resolve_agreement.py` locks the invariant: for a matrix of project/profile/CLI inputs, every field `/submit` persists equals what `/resolve` previewed. ## [0.5.8] — 2026-07-02 ### Fixed - **Watchdog kills now escalate to SIGKILL (audit 2026-07-01, H1).** The wall / idle / first-output watchdogs in `job_worker.py` sent a single `SIGTERM` and returned from `poll_signals` without arming a kill timer. A workload that ignores SIGTERM while holding its stdout open (native code in a critical section, or a child holding the pipe) blocked the stdout-read loop forever, so the post-loop `proc.wait(timeout=grace)` escalation was never reached — the job pinned its slot (e.g. a GPU) indefinitely and later cancels went unpolled. All three watchdogs now route through `_initiate_termination`, the same idempotent path the broker cancel/preempt signal uses, which SIGTERMs and schedules the SIGKILL escalation (grace `WATCHDOG_KILL_GRACE_S`, default 60s, now tunable via `JOBD_WORKER_WATCHDOG_KILL_GRACE_S`). Validated end-to-end by a real broker+worker harness (`tests/integration/test_broker_concurrency_live.py`, `JOBD_LIVE=1`) that force-kills an actual `trap '' TERM` workload. - **`job wait` / `job status --watch` / MCP `wait=true` no longer hang on preempted/orphaned/scheduling_timeout jobs (audit 2026-07-01, Quality-1).** The terminal-state set was defined five times across the broker, CLI, and MCP surfaces, and three copies (`cli.py` single-job `TERMINAL_STATES`, `mcp/tools.py` `_TERMINAL`) were the narrow `{completed, failed, cancelled}` — so any wait loop spun until its timeout when a job reached one of the three newer terminal states. All copies now import a single canonical `TERMINAL_STATES` / `TERMINAL_FAIL_STATES` from `models.py` (frozensets of `JobState`; StrEnum membership works on both raw API status strings and enum values). - **State transitions are now compare-and-swap guarded (audit 2026-07-01, H3).** Only the matcher's `queued->assigned` claim was atomic; every other broker transition was an ORM read-check-write. The worst case: `cancel_job`'s queued path read `QUEUED`, and if a worker's claim committed before its write, it overwrote `ASSIGNED` back to `CANCELLED` _without_ setting a kill `signal` — the worker then ran the job to completion while the user was told it was cancelled, and the resulting `/complete` was swallowed as a terminal-idempotent no-op. A new `_cas_state` helper (mirroring the claim) now guards `cancel` (both the queued->cancelled write and the running/assigned signal write), `complete` (wins only from a non-terminal state, so a late `/complete` can't clobber a sweeper-set `ORPHANED` and vice versa), `started` (assigned->running), `preempt` (signal write), and — after an independent review found the initial pass had left them unguarded — the sweeper's `scheduling_timeout`, ASSIGNED reclaim, wall-clock and dead-worker orphan writes, plus the heartbeat reconcile. Each is a compare-and-swap on the state it read, so a `/complete` a briefly-revived worker commits between the sweep's SELECT and its trailing commit can no longer be clobbered back to `QUEUED`/`ORPHANED` (which would lose the result and re-dispatch). The worker-liveness check gates the reclaim _decision_; the CAS gates the _write_. - **Dependency cascade is now transitive and fires on every failed-side terminal (audit 2026-07-01, H2).** Three gaps stranded dependents in `QUEUED` forever. (1) `_cascade_on_parent_terminal` was single-level: in `A<-B<-C`, `A` failing cancelled `B` but left `C` queued — it now traverses transitively (a cancelled child is itself a failed-side terminal, so its own default-policy children cascade). (2) `/submit` only checked that a `depends_on` parent existed; a default-policy dep on an already failed-side-terminal parent (which can never reach `COMPLETED`, and whose cascade already fired) is now rejected with `400` instead of stranding the child. (3) The `scheduling_timeout` sweep and the `cwd_unreachable` refuse-admission path both transitioned the parent to a failed-side terminal without cascading; both now cancel their default-policy dependents (the `scheduling_timeout` deps test previously only passed because it invoked the cascade hook by hand). any-exit dependents are unaffected — any terminal parent satisfies their dep. - **Requeue paths now clear the pending kill signal (audit 2026-07-01, M1).** A job carrying a `cancel`/`preempt` signal that gets re-queued (both `refuse-admission` branches, the sweeper's ASSIGNED and idempotent-RUNNING reclaims, and the heartbeat reconcile) kept `job.signal` set, so the fresh worker it re-dispatched to read the stale signal on its first `/signal` poll and SIGTERMed the re-run. All requeue sites now reset `signal` to `None` (as the terminal `/complete` path already did). - **The broker refuses `/log`, `/started`, and `/complete` from a stale worker (audit 2026-07-01, M2).** After a partition, a genuinely-running job could be reclaimed and re-dispatched to a second worker while the first kept running; the first worker's late `/complete` then terminal-ized the job under the wrong worker and its `/log` chunks interleaved into the new run's log. The worker now tags every request with an `X-Jobd-Worker: ` header (the same value it sends as `host` in `/next-job`, which becomes `job.worker`), and the broker returns `409` when the reporting worker isn't the job's current owner. The same guard covers `/refuse-admission` (added after an independent review found it missing — a stale worker's delayed refusal could otherwise requeue/exclude a job now running on a different worker), whose state writes are also CAS-guarded. Pre-header workers (no header) are unaffected — the check is skipped, preserving the old best-effort behavior. - **`cwd_refused` is emitted after commit (audit 2026-07-01, M3).** The `cwd_missing` refuse-admission path emitted its `cwd_refused` event before the transaction committed — the lone violator of the emit-after-commit invariant, so a failed commit could leave `events.jsonl` claiming a refusal the DB never recorded. It now emits after the commit in both the re-route and `cwd_unreachable` branches. - **Worker VRAM ads no longer double-count a running job (audit 2026-07-01, M6).** The heartbeat subtracted the full sum of in-flight VRAM reservations from the live NVML free reading — but NVML free already reflects the VRAM those jobs have allocated, so a steady multi-slot worker understated its free VRAM by Σ(ads) and idled capacity it actually had. The worker now reserves only the _unallocated_ part of each ad, `max(0, ad − already_allocated)` (computed from per-pid NVML usage of the worker's own job pids, the mirror of the foreign-VRAM accounting): a job still in its CUDA-init startup window gets the full reservation (so a second big job can't overcommit), while a job that has allocated its VRAM reserves ~0. The live `/next-job` admission gate remains the safety net. - **`/next-job` long-poll no longer parks a threadpool thread, and the sweep no longer freezes the event loop (audit 2026-07-01, M4/M5).** `/next-job` was a sync endpoint that held one anyio threadpool token (default ~40) for its entire long-poll wait; a fleet of ~40 long-pollers starved `/heartbeat` and the other sync endpoints, so workers were spuriously marked stale/orphaned — self-amplifying. It is now `async`: the blocking claim runs via `asyncio.to_thread` (asyncio's executor, separate from the anyio pool) and the wait suspends on an `asyncio.Event` on the loop, so a parked long-poll holds no threadpool token. Wake-on-submit is preserved via a cross-thread `call_soon_threadsafe` with a swap-on-wake event (a wake that fires during a claim attempt can't be lost). Separately, `_sweep_once` now runs via `asyncio.to_thread` from the sweep loop instead of directly on the event loop, so a sweep (incl. its up-to-5s SQLite `busy_timeout` stalls) no longer freezes every async endpoint. ### Changed - **CI: added `pytest-timeout` with a 300s per-test cap.** A hung subprocess/socket/watchdog test previously stalled CI to GitHub's 6h job kill; each test now fails after 5 minutes instead. Override per test with `@pytest.mark.timeout(N)`. ## [0.5.7] — 2026-07-01 ### Added - **Unauthenticated Prometheus `/metrics` endpoint.** The broker exposes aggregate state for scraping — `jobd_jobs{state}`, `jobd_workers{state}`, and `jobd_build_info{version}` — computed from the DB on each scrape via a `prometheus_client` custom collector (private registry; only `jobd_*` series, no default process collectors). It is mounted as an ASGI sub-app so it bypasses the global bearer-token dependency (mounts don't inherit router deps), and its `/metrics` path is exempted from the tailnet-IP ACL so an in-cluster Prometheus — whose source IP is the docker bridge, not a tailnet address — can scrape the broker's tailnet-bound port. Only non-sensitive aggregate counts are exposed, and the endpoint stays reachable only on the broker's bound interface. New dependency: `prometheus-client`. ## [0.5.6] — 2026-06-30 ### Fixed - **Host-local cwds no longer silently misroute to `exit 127`.** A job whose `cwd` exists only on some hosts (e.g. a git worktree under `/home`, which **every** worker advertises in `mount_roots`) used to pass the broker's coarse `/next-job` prefix filter, route to a host that lacked the path, and die `[worker setup error] No such file or directory` / `exit 127` minutes later. Two layers now prevent this. **(A) Submit-time probe:** `cwd_routability` (`matcher.py`) checks the cwd against known workers' `mount_roots` — a pinned host that advertises no covering root — or a `host_pin=any` cwd that **no** known worker covers — gets a `400` with a routable hint (generalizing the prior `/mnt/c` guard), since the job can't run anywhere. Workers with empty `mount_roots` are treated as "unknown" so it never false-rejects. **(B) Worker-side re-queue:** before running, a worker verifies `os.path.isdir(cwd)`; if absent it POSTs `refuse-admission` `reason=cwd_missing` instead of `exit 127`. The broker records the host in a new per-job `jobs.excluded_workers_json` (auto-migrated), drops the job for that host at `/next-job`, and re-routes — or fails it `termination_reason="cwd_unreachable"` with an explanatory log line when no eligible worker remains (never a silent queue, never a hot loop: each worker refuses a given job at most once). `AdmissionRefusal` gains `reason`/`cwd` and relaxes `required_gb`/`free_gb` to optional; the GPU-contention admission path is unchanged. Old workers (no cwd check, empty `mount_roots`) keep working. Motivating incident: a project-y GPU test submitted from a worktree cwd with `--gpu` (2026-06-30). Design + plan: `docs/plans/2026-06-30-submit-cwd-probe-design.md`, `…-submit-cwd-probe.md`; deploy: `…-submit-cwd-probe-deploy.md`. ## [0.5.5] — 2026-06-11 ### Added - **`/gpu-holders` now attributes GPU-holding PIDs to jobd jobs (#7).** Workers report a per-job PID inventory in every heartbeat (`in_flight_pids`: the job's top-level pid plus its live scope-cgroup pids — the same ownership boundary the foreign-VRAM accounting uses). The broker stores the latest map per worker (`workers.in_flight_pids_json`, auto-migrated) and `/gpu-holders` resolves each probed pid against it: `known` becomes meaningful and rows gain `job_id` + `worker` attribution. PIDs are host-local, so `?host=` scopes the lookup to one worker's inventory; the default consults every online worker (documented pid-collision caveat on multi-host fleets). Old workers that don't report the field behave exactly as before. ### Fixed - **Worker SIGTERM now drains in-flight jobs instead of stranding them.** The old handler called `sys.exit(0)` from the signal handler: daemon job threads died without signaling their workloads or posting `/complete`, the orphaned subprocess kept running, and — because `Restart=on-failure` brought the worker back within seconds, refreshing the heartbeat the dead-worker reaper keys on — the job could stay `RUNNING` in the broker forever. The handler also posted an httpx event from inside the signal context, which can deadlock on the client's pool lock. Now the handler only sets the stop flag; after the poll loop exits, the worker preempts every in-flight job through its normal cancel/preempt machinery (checkpoint window honored, SIGKILL escalation after `min(checkpoint_grace_s, JOBD_WORKER_DRAIN_GRACE_S)`, default 60s), joins job threads against a bounded deadline, and posts a `worker_shutdown` event with a `signaled`/`aborted` summary. Drained jobs land as `preempted` with `termination_reason="worker_shutdown"`. Jobs claimed mid-drain are refused before Popen and completed as preempted. Single-slot workers now run jobs in a thread like multi-slot ones (the inline path parked the main thread for the whole job, making a drain impossible). `job-worker.service` gains `KillMode=mixed` + `TimeoutStopSec=150`. Design: docs/plans/sigterm-drain.md. - **Broker reconcile backstop for workers that die without draining (Phase 2).** A SIGKILLed/crashed/power-lost worker restarts within seconds under `Restart=on-failure` and heartbeats again — refreshing the liveness clock the dead-worker reaper keys on — while knowing nothing about the jobs it was running, stranding them in `RUNNING`/`ASSIGNED` forever. Workers now report their in-flight job ids in every heartbeat (`in_flight_job_ids`; old workers that omit the field are skipped entirely), and the broker gives any claim older than 60s that is absent from 2 consecutive reports the worker-died disposition: `ASSIGNED` jobs requeue (also recovering a lost `/next-job` response, which the heartbeat-keyed reaper never caught), idempotent `RUNNING` jobs requeue, non-idempotent `RUNNING` jobs go `ORPHANED` with `termination_reason="worker_restarted"` and cascade to dependents. New `jobs.reconcile_misses` column (auto-migrated). Both deploy orders are safe — the heartbeat field is additive. - **Startup sweep of stale `jobd-*.scope` units (Phase 3).** Scope-wrapped workloads survive an undrained worker death (scopes live outside the worker service's cgroup), so a job requeued by the Phase 2 reconcile could re-dispatch and double-execute against the still-running old workload. The worker now kills every leftover `jobd-.scope` at startup — before its first poll — via cgroup-walk (or `systemctl --user kill` when the cgroup path can't be resolved) and emits a `stale_scope_sweep` event listing the units. Assumes one jobd worker per user session, which is the deployment model. - **2026-06-10 audit batch.** Children of a `scheduling_timeout` parent no longer strand in QUEUED forever (the state was missing from the dependency-terminal and cascade-trigger sets); `POST /jobs/{id}/log` now 404s on unknown jobs and 413s chunks over 10 MiB; cascade-cancel events are emitted after the transaction commits instead of inside it; the worker retries the `/complete` POST with backoff so a transient broker blip can't lose a job result; SIGKILL-escalation timers are cancelled on process exit; the post-SIGKILL `proc.wait()` is bounded (30s, rc −9); `tracked_pids` reads/writes are lock-guarded; the broker strips `JOBD_API_TOKEN` like clients already did; and `JOBD_ALLOW_NO_AUTH=1` + `JOBD_DISABLE_TAILNET_ACL=1` on a non-loopback bind now refuses to start instead of warning. ### Changed - **mypy is now a blocking CI gate.** Added a `[tool.mypy]` config and dropped `continue-on-error` from the type-check step, so type regressions fail the build. Cleared the pre-existing type debt with real fixes (a `BrokerState` TypedDict, `Literal` resolution-source annotations, `None`-narrowing) and targeted `# type: ignore[attr-defined]` only for genuine SQLAlchemy/mypy false-positives. No runtime change. ## [0.5.4] — 2026-06-07 ### Fixed - **Broker-side wall-clock backstop in the RUNNING reaper.** A job whose worker crashed mid-run and then restarted _within_ `DEAD_WORKER_SECONDS` (300s) could be stranded in `RUNNING` forever: `max_wall_s` was only enforced worker-side (`job_worker.poll_signals` SIGTERMs at `max_wall_s`), and that per-job monitor dies with the worker. The restarted worker has no memory of the job, so it never kills it and never posts `/complete`; the broker's reaper saw a live heartbeat again and left the job running. The reaper now also orphans any RUNNING job that has blown past `max_wall_s + checkpoint_grace_s + 120s` regardless of worker liveness (`termination_reason="wall_clock_exceeded"`), cascading to dependents like the existing `worker_died` path. The 120s grace keeps the broker strictly looser than a healthy worker (which terminates + reports within seconds of `max_wall_s`), so it never races one. Jobs submitted without `max_wall_s` are unaffected by design. ## [0.5.3] — 2026-06-07 ### Added - **Listed on the official MCP registry.** Added `server.json` plus a `publish-registry.yml` workflow that submits it via GitHub Actions OIDC (`mcp-publisher login github-oidc`) on every published release — no stored credentials, mirroring the existing PyPI trusted-publishing. An `mcp-name: io.github.musharna/jobd` marker in the README lets the registry verify the PyPI package belongs to this server. Because the MCP server is the `jobd-mcp` entry point behind the `[mcp]` extra (not the default `jobd` broker entry point), the registry entry runs it as `uvx --from jobd[mcp] jobd-mcp`. Auto-propagates the listing to downstream directories (PulseMCP, etc.). No code or runtime change. ## [0.5.2] — 2026-06-03 ### Fixed - **Worker no longer warns + buffers job output.** The job subprocess was opened with `bufsize=1` (line buffering) in binary mode (`text=False`), which Python doesn't support — it emitted a `RuntimeWarning` on every job and fell back to a `BufferedReader` whose `read(4096)` blocks until 4096 bytes accumulate, delaying streamed logs. Now `bufsize=0` (unbuffered): the warning is gone and each chunk read returns on first available data, so logs stream promptly. - **Docker image now builds.** The Dockerfile only `COPY`ed `pyproject.toml` + `src`, but `pyproject.toml` declares `readme = "README.md"`, which hatchling reads during metadata generation at `pip install .` → build failed with `OSError: Readme file does not exist`. Added `README.md` to the `COPY`. (The public Docker build path isn't exercised by CI, so this was latent.) ### Documentation - **Non-root image + host-owned bind-mount gotcha.** The image runs as uid 10001; a bind-mounted `data/`/`logs/` owned by a different host uid makes the broker crash-loop on `attempt to write a readonly database`. Documented the two fixes (chown to 10001, or run the container as your host user via `user:`) in `docker-compose.yml` and the runbook, plus a runbook note on repointing a stale worker-unit `ExecStart` at the `jobd-worker` console script. ## [0.5.1] — 2026-06-02 ### Documentation - **README "Retention" section** documenting `JOBD_JOB_RETENTION_DAYS` — the opt-in knob added in 0.5.0 was only described in the runbook, not the main README's Configuration area (the 0.3.0 multislotting knob set the precedent of documenting user-facing config in the README). No behavior change. ## [0.5.0] — 2026-06-02 ### Added — job/log retention - **`JOBD_JOB_RETENTION_DAYS` prunes old terminal jobs.** When set to a positive number, the sweeper deletes jobs in a terminal state (`completed`/`failed`/`cancelled`/`preempted`/`orphaned`/`scheduling_timeout`) whose `finished_at` is older than that many days, along with their per-job `.log` files, and emits a `jobs_pruned` event with the count. This bounds jobs-table and log-dir growth on a long-running broker (the `events.jsonl` stream is already bounded by size-rotation). Opt-in: the default (`0`) keeps history forever, so existing deployments are unchanged. Freed SQLite pages are reused under WAL, so the DB file stays bounded without a global-locking `VACUUM`. Pruning old terminal parents is safe for any still-pending dependents — they're already treated as dependency-satisfied. ## [0.4.0] — 2026-06-02 ### Changed — server-side long-poll for `/next-job` - **The broker now holds `/next-job` until a job is dispatchable.** Previously the endpoint returned `null` instantly and the worker re-polled every 2s, so each idle worker drove ~15× the intended request rate (a full queued-scan + per-dep lookups every 2s) even with nothing to do — the worker's own 30s client timeout showed a long-poll was the original design intent, never built. The endpoint now blocks up to `wait_s` (the worker sends its `POLL_TIMEOUT_S`, 30s), re-attempting whenever a `/submit`, terminal transition, or requeue wakes it (a `threading.Condition`), and otherwise rechecking at most every 10s as a backstop. Result: an idle worker makes ~0 requests while waiting, and a freshly submitted job dispatches near-instantly instead of after up to 2s. - **Backward-compatible.** `wait_s` defaults to `0` (legacy instant return), so older workers — and any client that omits it — are unaffected. A worker talking to an older broker that ignores `wait_s` gets an immediate `null`; the worker detects the fast return (elapsed < half its timeout) and keeps the 2s backoff, so it never hot-loops. ### Added — events.jsonl rotation + bounded reverse-read - **Size-based rotation.** The broker's `events.jsonl` is rotated to `events.jsonl.1` (one backup, overwriting any prior) once it crosses `JOBD_EVENTS_MAX_BYTES` (default 50 MB), bounding on-disk retention to ~2× the threshold. `job audit` / `GET /events` read both files, so a query spanning a rotation returns continuous history within the window. New `jobd.events` module is the single write choke-point (serialized by a lock so a concurrent rotate+append can't race). - **Bounded reverse-read.** `GET /events` now reads newest→oldest and, with a `--since` cutoff, stops at the first row older than it instead of scanning the whole file — events are append-ordered by the broker's server-side timestamp, so the early-stop is exact. Output order, filters, and `limit` semantics are unchanged. ### Fixed - **Removed a stale runbook "Known issue"** describing the self-GPU-as-foreign-VRAM reporting bug, which was fixed in 0.3.0. ## [0.3.0] — 2026-06-01 ### Fixed — Worker's own multi-process GPU job mis-counted as foreign VRAM - **GPU foreign-VRAM accounting now uses the scope cgroup, not a single pid.** A GPU job's CUDA contexts can live in forked children (DDP / accelerate / dataloader workers, or a `bash -c` that spawns the trainer) under pids different from the one the worker tracked. NVML reports whichever pid holds the VRAM, so the worker counted its _own_ children as foreign — inflating heartbeat `unregistered_vram_gb` (spurious broker contention) and, at `JOBD_WORKER_MAX_CONCURRENT_JOBS>1`, refusing the worker's own second job. `compute_unregistered_vram` and the live admission gate now exclude `_effective_owned_pids()` = the tracked pid unioned with every live pid in each in-flight job's `jobd-.scope` cgroup (the same ownership boundary the cancel/reap paths already use). Verified on an RTX 5090: a parent that forks a CUDA-holding child shows NVML reporting the child pid, which is absent from the tracked set but present in `cgroup.procs`. ### Added — Multislotting observability - **Slot usage in `job workers`.** Each worker's heartbeat now carries `max_concurrent` (its `JOBD_WORKER_MAX_CONCURRENT_JOBS`) and `running` (live in-flight job count), persisted on the worker row (additive migration) and surfaced on `WorkerInfo` / `job workers` as `running`/`max_concurrent`. Pre-field workers read as `1`/`0`. Lets you see at a glance how full each worker's slots are when bin-packing CPU + GPU jobs. - **README "Concurrency (multislotting)" section** documenting `JOBD_WORKER_MAX_CONCURRENT_JOBS`: the matcher is resource-aware, so raising the limit bin-packs jobs that fit side by side (a CPU-only job co-runs with a GPU job; two GPU jobs co-run only if both fit live VRAM) rather than blind oversubscription. ### Added — Parameter sweeps - **`job submit --sweep KEY=v1,v2,v3`** (repeatable) fans out a job array over the cartesian product of named axes — a grid search in one call. Each member substitutes `{KEY}` → its value in the command and env; `{i}` (the flat 0-based member index) is available alongside the named keys. `--sweep lr=0.1,0.01 --sweep seed=1,2,3` yields 6 members. `--sweep` and `--count` are mutually exclusive, the product is capped at 1000, and `i` is reserved as an axis key. Reuses the array machinery (shared `array_id`, `job list --array`, `job status A`) and the literal-`{key}`-replace substitution from `--count`. The `jobd_submit` MCP tool accepts `sweep` (list of `{key, values}`); the broker `JobSubmit` model gains a `sweep: list[SweepAxis]` field. ## [0.2.0] — 2026-06-01 ### Added — Job arrays - **`job submit --count N`** fans out one command template into N array members in a single call. Each member is an ordinary job (independent routing, preemption, checkpointing); `{i}` in the command — and in env values via the API/MCP — is replaced by the member's 0-based index. The substitution is a literal `{key}` replace (not `str.format`), so commands containing JSON literals or shell braces pass through untouched. The engine is generic over named keys, leaving room for a future `--sweep` form. - **Array identity + inspection.** Members share an `array_id` (the first member's job id), plus `array_index` / `array_size`, surfaced on `JobInfo`. New `job list --array A` filters to one array; `job status A` prints an aggregate state tally + per-member rollup and exits non-zero if any member ended non-completed. The broker `/submit` returns `{array_id, count, job_ids, warnings}` for `count>1` (a single `JobInfo` for `count==1`, unchanged); `GET /jobs` gains an `array_id` filter. The `jobd_submit` MCP tool accepts `count` and returns the array summary. ### Changed — Worker is now a packaged, installable component - **`jobd-worker` console script.** The worker daemon and its capability detection moved into the `jobd` package (`jobd.worker.job_worker`, `jobd.worker.capabilities`) and now install as a `jobd-worker` entry point via `pip install "jobd[worker]"`. No clone, no manual file copy, no `python worker/job_worker.py`. The `[worker]` extra carries the runtime deps (httpx, psutil, pyyaml, nvidia-ml-py). - **Simplified worker setup.** `scripts/install-worker.sh` now `pip install`s `jobd[worker]` into `~/jobd-worker/.venv` instead of copying source files from a checkout. The two systemd templates collapsed into one `scripts/job-worker.service` (`ExecStart=…/jobd-worker`) — the repo-checkout vs standalone-copy distinction is gone now that imports resolve from the installed package. ## [0.1.0] — 2026-05-31 Initial public release. The bullets below summarize the capabilities built during pre-release development; parenthetical short hashes reference that internal history and are not present in this repository's squashed tree. ### Added — Auto-preempt protocol - **#8 path A — auto-preempt on queue-age** (`6db2712`): broker sweeper auto-preempts running jobs when their host has a higher-priority candidate waiting and queue age crosses threshold. - **#28/#29 manual preempt-blockers + warnings-only filter** (`b06e78d`): `--no-preempt` submit flag and warnings-only `job list --warnings` filter. - **#32 `events.jsonl` auto-preempt event** (`e719842`): broker emits a structured event each time a sweep fires, for observability. - **#34 `/preempt` accepts ASSIGNED-but-not-started state** (`22a3791`): pinned by test. - **#35 `depends_on` cascade on PREEMPTED parent** (`26e56ac`): pinned by test. - **#8 path B — checkpoint protocol** (`e7593b5`): SIGTERM grace window with `JOBD_CHECKPOINT_GRACE_S` knob; terminal `preempted` state with `RESUME_FROM` operator-driven resume contract. - **`JOBD_CHECKPOINT_DIR` env var** (`f0cb754`): exposed to every job, pointing at a per-job directory `${JOBD_WORKER_CHECKPOINT_ROOT:-$XDG_DATA_HOME/jobd/checkpoints}//` (default `~/.local/share/jobd/checkpoints//`). Workloads write durable preempt-time checkpoints here; broker does not sweep contents. - **`JOBD_WORKER_CHECKPOINT_ROOT` operator override** (`71ae23b`, fix-forward `bf195bc`): worker-level env var redirects checkpoint root (e.g., to a faster filesystem); `os.path.expanduser` applied. ### Added — Resource-aware admission & GPU matching - **#42 heartbeat-aware GPU matcher** (`a3509c9`): matcher consults live worker heartbeats (`unregistered_vram_gb`) and drops saturated hosts; 2 GB VRAM floor on `--gpu`. - **#42 multi-tenant per-host co-routing** (`f9e6152`): `JOBD_WORKER_MAX_CONCURRENT_JOBS` enables threaded per-host dispatch with per-job thread tracking. - **`unregistered_vram_gb` in `/workers`** (`93795b0`): exposed for matcher + observability. - **#41 resource-aware admission at dispatch** (`53ffc09`): NVML probe → admission gate → `/jobs/{id}/refuse-admission` re-queue path; tier-tag inference; bypass marker. - **#41d `vram_gb` on `JobSubmit`** (`ed154f5`): CLI `--vram-required` flag now persists end-to-end. - **#43 unsatisfiable-placement preflight** (`bf15ffa`): submit-time check rejects jobs whose `needs:` tags or VRAM ask cannot be satisfied by any registered worker. - **CUDA VRAM tier tags** (`860fdf6`): `cuda-32gb`, `cuda-12gb`, etc. so `needs:cuda-32gb` routes only to matching hosts. - **`cuda-8gb` tier** (`a8ed96d`): added so RTX 2080-class GPUs advertise a discoverable tier. ### Added — Scheduling, watchdogs, MCP - **Per-job idle-output watchdog + max-wall timeout** (`5f074c0`): broker-side wall enforcement; idle-output watchdog terminates silent-hung jobs. - **`--max-wall` and `--idle-timeout` flags on `job submit`** (`f105724`). - **Scheduler-awareness warnings** (`0ef4497`): single-slot stall + queue-age-blocked-by-load surface as `warnings:` on `/jobs/{id}`. - **Per-job time estimation v1** (`1642d5d`): `eta_*` fields on `Job` and `--eta` flag on `job list`. - **Default-on ETA banner on `job submit`**: `--eta/--no-eta` defaults on; prints `Estimated wall p50 X, p90 Y (n=N prior runs)` or `ColdStart` line to stderr after submit. Closes BACKLOG "Auto-surface ETA on submit" Part 1. - **ETA Part 2 — ctest-aware sub-job parsing**: opt-in via `JOBD_CTEST_PARSE=1`. New module `src/jobd/ctest_eta.py` parses `/build*/Testing/Temporary/CTestCostData.txt`, filters tests by the `ctest -R ` arg, and sums avg-cost values. Broker reports `eta_basis="ctest-cost-K="` ahead of history-based prediction; CLI banner renders `Estimated wall ~Xs (ctest cost-data, k=K tests)`. Falls through to history when env unset, regex misses, or cost file absent. Closes BACKLOG "Auto-surface ETA on submit" Part 2. - **First-byte smoke watchdog — pieces 2+3**: worker-side, in `worker/job_worker.py`. Piece 3 (pre-dispatch launcher-existence check) verifies `cmd[0]` exists and is executable when it looks like a path (`/`, `./`, `../`); on miss, POSTs `/complete` with `termination_reason="launcher_missing"` instead of exec-ing into a silent exit-127. Piece 2 (first-output watchdog) adds env var `JOBD_WORKER_FIRST_OUTPUT_TIMEOUT_S`: fires once if no stdout byte lands within N seconds of job start, disarms permanently on first byte. Both surface as `final_state="failed"` for depends_on cascades. Closes BACKLOG "First-byte smoke" pieces 2+3; piece 1 (push-on-terminal-failure) deferred. - **Per-project defaults block in `projects.yaml`** (`34c2a30`): keys at submit time inherit from project block when omitted. - **`fast_path` field honored in `JobSubmit`** (`557a0e8`). - **#51 `submitted_via` marker** (`636969c`): `JobSubmit.submitted_via: Literal["cli", "mcp"]` round-trips through translation layer; structural test pinned. ### Added — Worker management & deploy - **DELETE `/workers/{host}` for purging stale registrations** (`ea04d6a`). - **`job delete-worker` + `jobd_worker_delete` MCP tool** (`ccdf35c`): exposes the DELETE endpoint via CLI and MCP. - **`jobd-broker.service` systemd unit** (`5334a5c`, #52): with tailscale IP wait. - **`job-worker.laptop.service` variant** (`c4591b0`): for full-repo-checkout hosts (vs. the server standalone install). - **Nightly live integration test cron wrapper** (`ef24309`): runs `tests/mcp/test_live.py` against the live broker. - **Audit instrumentation pass** (`07aa2cf`): coverage inventory + gap report against the existing observability surface. ### Fixed - **`jobd --help` / `--version` no longer crash with SQLite OperationalError**: entry point now parses argv before `build_app()`, so `--help`/`--version` short-circuit cleanly without touching the database. Help text documents `JOBD_CONFIG_DIR` / `JOBD_DB_URL` / `JOBD_PORT` / `JOBD_LOGS_DIR`. - **`#51` install-worker no longer writes static `tags:` to `worker.yaml`** (`5c01882`): tags now come from runtime probe. - **`#73eaa46` cancel via `systemctl kill` on named scope** rather than `Popen.pid`: ensures the entire scope tree dies, not just the tracked PID. - **`59561aa` ASSIGNED → RUNNING transition** so cancel reports SIGTERM correctly. - **`969abc3` MCP `log_tail` and `depends_on` field-name alignment** (mcp-v1 field-test fallout). - **`353a4fb`** clear pyright noise in `_print_resolved`/`_row`. ### Documentation - **Checkpoint directory contract** in `docs/preemption.md` (`8b5e339`, fix-forward `7ebffca`, final-review polish `2eec63b`): canonical surface for workload authors; covers env vars, default root, mode-0700 + cross-user-resume caveat. - **Auto-preempt default-flip design spec** (`37e2dc4`) + amendment (`31e843d`). - **Auto-preempt jobd-side implementation plan** (`915970f`). - **Backlog reconciliation** (`ddca65e`, `d34f3b0`, `6d0fbaf`, `b257cca`, `354e043`, `ab8f7f7`, `7eb1203`, `394269c`, `b8de199`): Open vs Done sweep after the 2026-04-27..04-29 ship train; field-test backlog items filed. - **Projects.yaml defaults — implementation blueprint** (`c98cf26`). - **CHANGELOG.md created** (`86b4880`); this entry backfills the gap from `mcp-v1` to current tip. --- ## [mcp-v1] — 2026-04-26 Translation-layer/MCP shipping point. Earlier history is not catalogued here; see `git log mcp-v1` for detail.