# Changelog

All notable changes to this project will be documented in this file.

Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

## [Unreleased]

### Changed
- **Clarified public claims across README, documentation, and website copy (DOC/DIST-1).** Reconciled licensing copy to a single permanent Apache-2.0 open-source core statement without separate commercial license implications; bounded cold-start latency wording to avoiding latency on warm hits; qualified session affinity as opt-in (disabled by default); updated per-key rate-limiting and auth descriptions to remove multi-tenant terminology; accurately reflected SQLite-restored hourly analytics in documentation; attributed fleet savings projections as illustrative models; clarified installer auto-discovery coverage and manual registration for MLX on Apple Silicon; and documented realistic encryption key loss failure behavior.
- **Requests that no local node can serve now go to your cloud fallback immediately instead of waiting out the queue timeout.** When every node is down or draining, or no node serves the model, and a cloud provider is configured, the key may use cloud and the spend cap is not reached, the request skips the queue. A request for a healthy but busy node still waits, now up to the overflow SLA (default 2000 ms on installs that never saved the setting; 0 keeps meaning wait the full queue timeout) and then goes to cloud; later requests return to local as soon as a slot frees. Without a cloud provider, with a local-only key, or with the spend cap reached, every request waits the full queue timeout as before. The routing decision for a request that found no node now carries a `noRoute` class (no_node_serves_model, all_nodes_unavailable, at_capacity_healthy or unknown_state); the request log and explain output surface it in the next change.

### Added
- **Documented scraping marbor-agent with Prometheus (HTTPS, bearer token, CA trust).** Added guidance covering agent scrape configuration (:9200), TLS by default, HTTP 400 plaintext refusal vs HTTPS, bearer token extraction from `/etc/marbor-agent.env`, CA certificate trust (`tls_config.ca_file`), SAN matching, post-rotation reload (`marbor-agent service regen-cert`), and exported GPU and host metrics table.
- **Dedicated runtime entry paths for Ollama, vLLM, and MLX fleets.** Added documentation pages (`docs/ollama.md`, `docs/vllm.md`, `docs/mlx.md`) covering operational pain points, runtime-specific quick starts, shipped operational outcomes, and when-not-to-use guidance for operators running fleets on each backend. Added a Runtimes group to the documentation sidebar.
- **Every cloud-routed request now says why it went to cloud and how long each part took.** The request log, `GET /admin/requests`, `/admin/audit`, `marbor requests`, `marbor requests live` and `marbor audit` (new WAIT MS and CLOUD MS columns) and the dashboard Requests page show a reason code (no_node_serves_model, all_nodes_unavailable, saturated_sla_expired, queue_timeout_cloud, queue_full_cloud, local_upstream_failed) plus the measured queue wait and cloud time, or `-` when not measured; `marbor requests explain` prints the reason, a one-sentence explanation and the no-route class. A `local_only` key refused with 503 now leaves a `local_only_blocked` row. New metric `marbor_cloud_overflow_reason_total{reason}`. Existing rows show `-`.
- **Drain and runtime restart can now be confirmed against the exact list of models you reviewed.** Add one `acknowledge_sole=<model>` per model to `force=true` on `POST /admin/nodes/{name}/drain` or `POST /admin/nodes/{name}/runtime/restart`. If any model the node is now the only server of is missing from that list, marbor changes nothing and answers 409 (`code: sole_replica`) with the current lists, an `unacknowledged` array and a message saying the list changed since you reviewed it. The dashboard confirm and the interactive CLI confirm send the list they showed (an empty `acknowledge_sole=` when it was empty). The CLI prints the new list to stderr, exits 1 and does not retry.
- **`force=true` on its own is unchanged.** It still overrides without checking the list, and an explicit `--force` stays that way. Script recipe: `GET /admin/nodes/{name}/sole-replica`, then POST with `force=true` and one `acknowledge_sole` per model, and on a 409 review the new list and repeat.
- **How the list is matched.** Models match by exact name, so one entry covers that model on every runtime; names that are only cold-loadable or unverifiable never need an entry. The comparison and the change happen under one lock, so it is atomic against marbor's own changes only, not against changes made elsewhere on the fleet.
- **Query-string limits.** The list is sent in the URL: the dashboard and CLI refuse above 6000 encoded bytes and point to `--force` or a script. A query string that does not parse is ignored when no model would be left without a server, and answered 400 (nothing changed) when one would, including with a bare `force=true`. An audit entry for a forced drain or restart that was sent with a list ends with `Reviewed: true`; the model names in the request are never written to it.

### Fixed
- **Saving the overflow SLA, queue timeout or queue depth on the Settings page now takes effect on the next request, without a restart.** Before, the values were stored but the running process kept the ones it started with, so a saved `overflow_sla_ms` did nothing until marbor was restarted. A request already waiting keeps the timer it started with. A negative value is now refused with 400. The Settings page labels the SLA field as 0 = wait the full queue timeout and says these three apply immediately.
- **Streaming `/api/generate` through a cloud provider no longer returns an empty answer.** The stream translator only read chat-style `delta.content`; OpenAI completions-style chunks carry `choices[].text`, so every chunk came out blank. Both shapes are now read.
- **Long benchmarks no longer fail partway with 401.** The temporary key a benchmark uses expired after a fixed 15 minutes; its lifetime now scales with the number of samples (capped at 9 hours), is limited to local nodes, and a key left behind by a crash is removed at startup.
- **Key names starting with `benchmark-` or `pull-verify-` are now reserved.** They are rejected with 400 when creating an API key and when approving a user with a new key (including the default `<username>-key` name, where the message says how to choose another). At startup an existing key with such a name is treated as a leftover probe key and removed only if it is not revoked, is local-only, expires, is limited to one model, has no usage limits and does not allow local degradation (the startup log names each one). A revoked key that otherwise matches is revoked in auth and its row stays in the store; it is not removed.
- **A NaN or infinite reading from an engine or GPU tool no longer blanks a node's whole status.** The marbor agent now treats non-finite and out-of-range readings (percentages outside 0-100) as unavailable instead of passing them through, skips them in `/metrics`, and answers `/v1/status` with an error instead of an empty 200 if a snapshot still cannot be encoded.
- **The Settings page no longer shows an authentication on/off control, because it never changed anything.** Saving it did not reload the running authentication or survive a restart. `auth.enabled` is now ignored on `PUT /admin/settings`, so the running configuration always matches what is enforced; `GET /admin/settings` still reports it read-only.
- **The marbor agent now rejects a Docker container identifier that is not a valid container name or ID** (for example one starting with `-`) before running `docker exec`, `inspect`, `logs` or the disk-stats probe, so a malformed identifier can no longer be read by the docker CLI as a flag.
- **Saving Settings no longer resets a custom proxy port, and the Settings response no longer includes API key secrets.** `GET /admin/settings` now names `proxy.port`, `auth.enabled` and `auth.keys` in lowercase, which is what the dashboard already read, so the dashboard showed the default port and a routine save wrote it back. The response also no longer carries API keys (they were returned in clear text) and a read no longer masks cloud provider keys in the running configuration. A script that read the old capitalised names (`Port`, `Enabled`, `Keys`) must switch to the lowercase ones.
- **API key changes are saved before they take effect, and a failed save is reported.** Create, rotate, patch and the key made when a user is approved write to the database first and answer 500 if that fails, instead of changing live auth and losing the change on restart. Revoking a key (including for a suspended or deleted user) revokes it live first, then saves; if the save fails the answer is 500 saying the key may return after a restart. Key changes are serialised. Creating a key now also honours the snake_case names `rate_limit`, `expires_at`, `daily_limit`, `monthly_limit`, `daily_usd_cap`, `monthly_usd_cap`, `local_only` and `allow_local_degradation`, which the dashboard sends; before, the rate limit and expiry were silently dropped. A model pull verification now fails if its temporary key cannot be saved, and creating a key through approval refuses a name that already exists (409).
- **Bare model names (`llama3`) now match their `:latest` resident on Ollama nodes** for in-flight eviction protection, warm checks and warm scoring and predictive warm-up. Other runtimes still match exact names only.
- **Eviction no longer unloads a model that started serving after it picked its victims**, and a model that can never fit on a node is skipped instead of loaded anyway.
- **Changing a node's URL keeps its drain state and no longer races with requests being routed.** `UpdateNodeURL` and an `AddNode` with a new URL swap in a fresh node record that keeps drain, grace period, warm-up disabled flag and VRAM totals, and clear the cached model tags and saved warm state for the old URL. Re-adding a node with an unchanged URL no longer rewrites live fields that did not change.
- **A panic while predictively warming a model is recovered** instead of stopping marbor, and the list of missing tensor-parallel ranks in a replica suggestion is capped at 32 entries followed by a count.
- **A runtime restart whose agent reply is a 5xx or unclear no longer clears the restart record.** Only a 4xx answer (other than 408 and 499, which a proxy can send after passing the request on, a known gap) counts as the agent refusing before it ran the restart. The error then says the outcome is unclear and that the restart record clears on its own after the next poll or after 15 minutes, so a drain of a peer is still checked against a node that may be restarting.
- **Disabled confirm buttons in the drain and runtime dialogs show the not-allowed cursor again.**

## [0.25.2] - 2026-10-28

### Added
- **The backup directory setting is now checked when it is saved, and a planted or oversized backup status file is ignored.** `backup.target_dir` must be an absolute path with no `..` element, control character or NUL byte, and must not be a filesystem root; on Windows it must also not be a device or extended-length path, a reserved device name, a path with a colon after the drive letter or a forbidden character, or an element ending in a dot or space. The saved value is cleaned. A Settings change that fails the check is refused with a 400 and the live directory is unchanged. A relative path from an older install or the `MARBOR_BACKUP_DIR` variable is resolved against the start-up directory once, as before; a stored value that fails the check is ignored with a log line and the default directory is used, so marbor still starts. The saved backup status file is only read if it is a plain file (not a symlink) up to 1 MiB. On Linux and macOS, the backup directory warning now also reports a directory owned by a different non-root user than the one marbor runs as. The start-up sweep of staged restore files now removes only the exact names a restore creates next to the database (it matched on the prefix alone before), and saving the backup status no longer deletes the file it just wrote when the backup directory is another name for the database directory.
- **Backups are now gzip-compressed, and can be listed, downloaded and restored from the dashboard, the CLI and the Admin API.** Scheduled and on-demand backups are written as `marbor-backup-<UTC timestamp>.db.gz` (plain `.db` files from earlier releases stay listed, restorable and counted by retention). Restore and upload accept both by content, with the same integrity check, a cap on the unpacked size, and a failure before the live database is touched. New: `GET /admin/backup/download?name=`, `marbor backup download <name> [--output file]`, a download button in Settings > Backup & restore, and `marbor backup list` / `GET /admin/backup/list` now report whether each file is compressed plus the last backup result (success, failed or refused, with the reason) and any warning about the backup directory. `marbor backup now` and `marbor backup upload` stream to and from disk instead of buffering and no longer hit the CLI's 10 second request timeout on large databases. A backup is refused up front, with how much space is needed and free, when the volume cannot hold the staged snapshot. A scheduled run that fails or is refused retries after an hour, an unchanged database is not backed up twice (compared by a fingerprint stored in the file, no extra copy of the database), only one backup runs at a time, leftover temporary files from an interrupted run are removed at startup, and backup files are created `0600` in a `0700` directory (an existing wider directory is reported, not changed). Every backup is now read back in full and verified before it is kept (gzip checksums, database header and content fingerprint); a file that fails is discarded and the run reported as failed. A scheduled run skips a duplicate only after reading the candidate in full, so a damaged file is never mistaken for a good copy. The scheduled and on-demand outcomes are tracked and shown separately, along with the last restore, whether the backup directory exists, a failure to prune, leftover files and entries that could not be listed. Free space is checked on every volume a backup, upload or restore writes to, with a margin and room for write-ahead-log growth. Uploaded files are named `-upload`, have their own retention budget, are flagged in the list as ok, unverified or invalid (an invalid file is flagged and kept, never deleted), and are limited to 2 GiB. A restore decompresses and checks the backup before answering the request, so a bad or too-large file is rejected on the card, and a stale staged file is removed at the next start. Transfers that stall are cut off after two minutes of no progress. `marbor backup now` and `marbor backup download` never replace an existing file unless `--force` is given, write through a temporary file, and report a full local disk as a local error. In the dashboard, both downloads now go through the API instead of a bare link, so an expired session, a refused backup or a missing file shows an error instead of saving an error page as the backup; a stored backup streams straight to a file chosen in a save dialog where the browser has one, and otherwise (and always for Download Backup Now, which is prepared on the server first and would outlive the click the dialog needs) the file is collected in memory up to 512 MiB, checked from the declared size before anything is read; larger databases use `marbor backup now`. The card also shows an unreadable status, an unknown result or a failure with no reason instead of a blank or a dangling colon, refreshes while a backup runs, and fits a 375 px screen. Measured on a synthetic request-log database, one run per cell, backup on the same disk, Windows and Linux (WSL2): 1,000,000 rows (162 MiB) take 5.9 to 9.8 s in total of which the read-back is 0.5 to 0.6 s, 23.9 MiB compressed (6.6x); 10,000,000 rows (1.60 GiB) take 70 to 111 s of which the read-back is 5.2 to 6.3 s, 242.5 MiB compressed (6.55x). With a second connection writing a row every few milliseconds throughout, writes took a median of 0.2 ms (below the Windows clock tick), 99th percentile 0.4 to 1.6 ms, and slowest 10 to 267 ms; the write-ahead log peaked at 3.9 MiB (1M rows) and 32 to 52 MiB (10M rows). The test process peaked at 33 to 38 MiB of memory at both sizes, and the staged snapshot plus the compressed file peaked at 1.12 times the database size on disk. Restoring the 10M-row backup took 14 to 68 s to unpack and validate (it varied with disk cache state) and a full re-check of the staged file at swap time costs 4 to 6 s. The pre-run space check now plans the compressed copy at the full snapshot size, the worst case for data that does not compress. An earlier run on a 2.48 GiB database with a concurrent writer (before the read-back existed) took 4 min 31 s, 0.55 GiB compressed (4.5x), write-ahead log peak 171 MiB, and writes at p50 3 ms, p99 33 ms and worst 717 ms. Compression is gzip's default level, so it is CPU bound; real data compresses differently from this synthetic set. Plain `VACUUM INTO` holds one read transaction for the whole snapshot, so the write-ahead log cannot shrink while a backup runs; it is truncated right after. On Windows the `0600` and `0700` permission bits have no effect, so restrict the backup directory with NTFS permissions.
- **Backup hardening: restore safety, honest "ok", shared space accounting and CLI exit codes.** Only one restore can be staged at a time (a second gets 409), restoring an in-memory database is refused with 501, and after the live database closes the staged file is validated in full again, an old write-ahead log that still holds data blocks the swap, and sidecar files are removed only after the rename. A pending marker and an outcome marker survive the restart; a failed or ambiguous restore stays on the card and in `marbor backup list` until acknowledged (dashboard dismiss button, `marbor backup ack-restore`). In the list, "ok" means gzip checksums, SQLite header and fingerprint agree; results are remembered for 24 hours in a small file in the backup directory (keyed on file identity) and re-verified daily; a damaged file is flagged, kept and never counted toward retention. Concurrent backups, uploads and restores reserve disk room against one shared ledger, and the preflight plans the compressed output as half the snapshot. Uploaded files are a separate pool with the same retention count (so up to twice as many files can be on disk, and the Settings card says so), at most 2 manual backups run at once, and an upload below 64 KiB/s or past its total deadline is cut off. `marbor backup list --check` exits 3 when anything needs attention. CLI downloads clean up their partial file on Ctrl-C, and a failed final directory sync is a warning with exit 0. The recover wrapper now passes through connection deadline controls so large backup transfers are not killed by the server-wide write timeout, and a test streams a backup through the real admin chain to keep it that way. The error-leak hygiene test now also catches a correlated reference printed next to a raw error in the same log statement.
- **Backups: clearer restore follow-up, delete, and stricter checks.** `/health` carries a `started_at` process identifier (a 503 from it is a degraded but running marbor), and `marbor backup restore --wait` and the dashboard use it to tell a restarted marbor from the old one, then read the stored result of that restore (its 16-character reference) before saying it worked; the dashboard watcher survives leaving the page, and if the session is gone after the restart it says the outcome could not be confirmed. The dashboard dismiss button names what it does (clears the result for every operator and the CLI) and sends the reference it displayed; the server refuses a mismatch with 409 and writes an audit entry. A stored backup can be deleted with `DELETE /admin/backup?name=`, `marbor backup delete`, and a confirmed button in the dashboard. The status carries `schedule_enabled`, `schedule_interval_hours`, `running_since` and `dir_scan_error`, and `marbor backup list --check` exits 3 for a missing directory, no backups, a scheduled backup that never succeeded or is overdue, a backup running over an hour, notes and notices, entries or a directory that could not be read, and more (each listed in `docs/cli.md`). Uploads reserve space as bytes arrive, report a commit `warning`, and the CLI sizes its waits to the file (`backup now` 30 minutes for the first byte, stored download 2 minutes, upload verdict `max(15 minutes, size / 8 MiB/s)`) and refuses a response that is not a backup (login page, wrong type) instead of saving it. At most 2 manual backups run at once (`429` with `Retry-After`), and a request that arrives during housekeeping or a restore gets `503` with `Retry-After`. The in-memory verification cache is no longer persisted, so every file is re-read after a restart. The Settings card stops polling after five failed refreshes, shows the unsaved state of a cancelled on-demand download, blocks an upload while a stored download runs and the reverse, and keeps every control at 44 px on a phone.
- **`GET /admin/audit` and `marbor audit` can page through the request log with a cursor.** Pass `before_id` (older rows) or `after_id` (newer rows) taken from `next_before_id` and `prev_after_id` in the previous response; each entry now carries its `id`. The cursor is a row id, so a page does not shift while new requests arrive, and a missing `next_before_id` or `prev_after_id` means there is no older or newer matching row. Filters combine with the cursor, `limit` still defaults to 100 and is capped at 1000, and a request without cursor parameters behaves as before apart from the new fields. `marbor audit --before-id N` and `--after-id N` do the same and print the cursor to use next, and the table output gains a leading ID column. Filtering by model, key or node is a substring scan of the whole log, so a filter that matches few rows is slow on a very large log.

### Changed
- **The Activity page's refresh no longer drops events you loaded with Load more.** Neither the 30-second auto-refresh nor the Refresh button does: the refresh now adds new events at the top and keeps the older pages already on screen, so the "N loaded" count no longer drops back to one page every 30 seconds. If more than a page of new events arrive between refreshes, the list restarts from the newest page and says so. Pages loaded with Load more stay in memory until the page is fully reloaded.
- **Create API key, Create user and Add GPU node now show errors in one place: directly above the submit button.** A missing name, host or username is reported when you press the button, which stays enabled (Add GPU node used to grey out with a hint beside it). The message is announced to screen readers, focus moves once to the first field that needs fixing, and the message clears as soon as you edit any field in the form. The server's own error (a duplicate name, an unreachable host) appears in the same spot, and Create user now uses the same alert styling as the other two. Required fields keep their asterisk.
- **The create forms are stricter about what they send.** Names, hosts and usernames are trimmed, the API key rate limit must be a whole number, and a second click while a request is running is ignored (the button shows "Creating..." or "Adding..." and stays enabled).
- **Closing the Create key or Add node form, with Cancel, the close button, Escape or a click on the backdrop, keeps what you typed; a successful create clears it.** The error message is cleared on close.
- **Add GPU node checks the port and host before sending.** The port must be a number from 1 to 65535, and the host must be a bare name or address (no `http://`, port or path; a bracketed IPv6 address is fine). Create user trims the email. A secret from a second API key created before the first is copied now waits its turn instead of replacing it, and reopening a create form while an earlier request is still running no longer clears what you typed.
- **Closing a create form while a request is still running no longer loses the result.** A new API key's secret and a new user's initial password, both shown only once, appear even if you closed the form first. A failure for a form you already closed is dropped, since nothing was created. A failed list refresh after a successful create is reported by the page, not as a failed create.
- **The Requests page now pages through the request log instead of loading the newest 1000 rows.** It shows 100 requests by default (50, 100, 250 or 500 per page) with Newer and Older buttons, so any request in the log can be reached, and the page keeps a bounded number of rows instead of re-rendering a thousand every 3 seconds. The newest page still refreshes every 3 seconds; when you page back, refresh pauses so the rows you are reading do not move, and a "Live paused, Resume" chip returns you to the newest requests. The server reports no total, so the page says "At least N requests loaded" while older rows exist. Filters, quick range, page size and the current page are kept in the URL, so a view can be bookmarked or shared; active filters show as chips you can remove one by one or clear together, and an empty result names the filters that produced it. The CLI equivalent is `marbor audit --before-id` / `--after-id`.
- **The Activity page now shows every audit event by default.** Sign-ins, failed sign-ins, API key changes, user changes and settings changes were hidden under a default filter. The Kind filter now has Access, Keys, Users and Settings, and Access highlights failed sign-ins. The stat cards state what they count (events shown of events loaded, operators compared without regard to case), and the footer says when a filter is active with a one-click clear. The Admin API and CLI are unchanged; the CLI still reports these events as `config`, and the exact route to failed sign-ins is `marbor activity --action login_failure`.
- **`marbor keys` prints `-` for a key with no hourly rate limit instead of `0`.** Zero has always meant no cap; the daily and monthly columns already showed `-`. `--json` output is unchanged.

### Fixed
- **A restore always validates the staged database in full before swapping it in, and backup checks are stricter and more accurate.** The swap no longer skips the full integrity check when the staged file looks unchanged (a few seconds of extra downtime on a large database), and a symbolic link in place of the staged file is refused. A plain backup is now checked for integrity before it counts toward retention, an intact gzip backup that has no fingerprint record is accepted as verified, and 409 responses from the backup endpoints carry `Retry-After`.
- **The panic-recovery wrapper around every response no longer hides the connection's deadline controls.** Handlers could not extend or clear their own read and write deadlines through it, so long streaming responses were still subject to the server-wide write timeout. It now passes them through, which large backup downloads and uploads rely on, and a regression test covers it.
- **The cloud spend figure on Analytics and `GET /admin/metrics/savings` no longer shows `-` after every restart.** The count of cloud tokens behind the spend was kept only in memory, so a restart left a stored spend with no token count and the figure was hidden until new cloud traffic arrived. The count is now saved with the other counters (about every 30 seconds, only when it changed, and at shutdown) and restored at startup. A database from before this change has no saved count, so its spend stays `-` until new cloud tokens are recorded; nothing is estimated. If a crash lands between the two writes, the spend can show `-` or a slightly low token figure, never an invented number. The `since` timestamp on `/admin/analytics` and `/admin/metrics/savings` is now always UTC (`Z`) instead of the server's local zone; no other timestamp in the Admin API changed. When an Analytics refresh fails, the page now keeps the last figures and says why it failed and when they were last updated, and the Cloud spill table keeps its last rows with the same warning instead of silently emptying. A slow earlier response can no longer overwrite newer data.
- **Analytics and the saved-vs-cloud cards now say which period their numbers cover.** The Analytics subtitle claimed everything was the last 24 hours, but the three summary cards and the Dashboard and Routing saved-vs-cloud card are all-time totals and only the charts cover 24 hours. The cards are now labelled "All time", and the Requests by model table says it counts requests since the last restart (it starts over when marbor restarts) and shows when that was. The API keys list reads "tokens this month". `marbor savings` now reads "(all time)" instead of printing the process start time as if it were the start of the counters; `--json` is unchanged and still carries `since`. The demo Analytics page now agrees with the demo savings figures.
- **The Requests page "Why" column is visible without scrolling sideways.** The table defined widths for 8 of its 9 columns, so the routing reason was pushed off the right edge. All nine columns now fit at desktop widths, and on wide screens the spare room goes to the key, model and node columns instead of being left empty.
- **The dashboard no longer logs a Content Security Policy error on every page load, and a dark-mode user no longer sees a light flash on a hard reload.** The script that applies the saved theme before the first paint was inline, which the dashboard's `script-src 'self'` policy blocks, so the theme was only applied after the page painted. It now loads from a same-origin file, and the policy is unchanged.
- **The Warmup page no longer tells you to restart marbor to turn on prefix locality hints.** The setting applies immediately once saved, as Settings already said. The Prefix Locality Hit Rate card now names the real section, Settings > Routing & Reliability, and links to it.
- **A node added with `--vram-total-mb` (or `POST /admin/nodes`) now shows its declared VRAM total.** The value was saved but never reached the running node, so `marbor nodes` and the dashboard showed `0 MB / 0 MB` until a restart. `marbor nodes` also stops printing `0 MB` for used VRAM when it is unknown: a declared total shows `- / 24000 MB (declared)`, and a node with no VRAM information shows `- / -`. The dashboard's fleet capacity card no longer counts a declared-only node as using 0 GB.
- **An API key with no rate limit shows Unlimited instead of 0/hr.** Zero always meant no cap, and the Edit key form now says so.
- **Required fields are marked.** Username in Create user, and Node Name and Host in Add GPU node, carry an asterisk, and the disabled Add Node button now says what is missing.
- **Predictive decisions name the time zone of their hour, and skipped ones show a Reason line.** Activity and Warmup read the same way ("seen together 125x at hour 17 (UTC)"); the engine does not record why a prediction was skipped, so the reason shows `-`. The demo hour now uses UTC to match its clock.
- **Opening the dashboard sends one settings request instead of several.** Concurrent requests for the same settings now share one call; the 15-second refresh and other endpoints are unchanged.
- **Model search for Ollama and llama.cpp no longer lists speech, image and video models, and the Model Advisor no longer claims results are already filtered to what the runtime can load.** The Hugging Face `gguf` tag also covers text-to-speech, speech recognition, image and video models, so they showed up in Browse. `GET /admin/models/search` (and therefore `marbor models search` and the dashboard) now drops results whose Hugging Face pipeline tag is speech, audio, image, video, 3D or detection when the runtime is Ollama or llama.cpp. This is a best-effort filter on Hugging Face's own tags: repos with no tag are still listed, and some of them may not be text models. The Browse help text now says exactly that.
- **The Model Advisor no longer says "What fits <node>" for a node whose VRAM is unknown.** The panel is titled "Curated models for <node>" when the runtime is incompatible, and for a compatible runtime no curated list is shown. One note on the Browse and Favourites tabs explains that fit cannot be computed until the node's VRAM is declared (Edit node > Capacity overrides) or detected by connecting its agent.

## [0.25.1] - 2026-10-20

### Added
- **Eight how-to guides under `docs/guides/`**, published at marbor.in/docs/guides/: load balancing vLLM servers, one endpoint for several Ollama machines, several llama.cpp servers, a Mac fleet with MLX, per-key quotas, what happens when every GPU is busy, Prometheus and Grafana monitoring, and draining a node without downtime. Each one starts from the problem, answers it in the first paragraph and shows the commands. The backend-side commands (`vllm serve`, `llama-server`, `mlx_lm.server`) come from upstream documentation and were not run against real GPUs.
- **`GET /admin/nodes/{name}/sole-replica` and `marbor runtime drain|restart --dry-run` show what a drain or runtime restart would take out of service, without changing anything.** The response lists `models` (models this node is the only possible server of), `cold_load_only` (models that no other node has warm but an Ollama node would have to load on demand) and `unknown` (places where marbor cannot verify the answer, such as a node that has not been polled yet). `--json` on `marbor runtime drain|restart` includes the same `sole_replica` object. The check only knows about drains and restarts made through marbor.
- **`marbor nodes --signal` and `--search` filter the node list, matching the dashboard's filter chips.** `marbor nodes --signal degraded,down --json | jq length` counts the nodes that need attention (any one of the signals matches; the flag can also be repeated). The signals are `healthy`, `degraded`, `draining`, `down`, `agent`, `warm` and `pressure`. `--search` keeps nodes whose name or GPU model contains the text (trimmed, case-insensitive). An empty `--signal` is a usage error (exit 2) and sends no request. The Admin API node list takes the same `signal` and `q` query parameters and every node, from the list or from `GET /admin/nodes/{name}`, now carries a `signals` array. The server decides what each signal means, so the dashboard's chips, chip counts and fleet-health strip read the same array instead of working it out in the browser. An empty match prints an empty table or `[]` and exits 0, so scripts should count rows. Against a server that predates filtering, the flags fail with a clear message instead of printing every node.

### Changed
- **TGI is now a frozen legacy runtime.** Hugging Face archived text-generation-inference in March 2026. Existing TGI nodes keep working (health probe, model discovery, engine metrics and placement) and bugs are still fixed, but new features no longer add TGI-specific work. The README and limitations page mark TGI as legacy.

### Fixed
- **`marbor key list` and `GET /admin/v1/keys` now show each key's daily and monthly request limits.** The limits were enforced but missing from the list, so a key capped at 3 requests a day looked unlimited, and the dashboard's edit form opened blank for those fields. The response gains `dailyLimit` and `monthlyLimit` (live values, so a patch shows up immediately; 0 means unlimited). In the CLI table the DAILY and MONTHLY columns used to show requests used, next to a RATE column that showed a limit; they are now `DAILY USED/LIMIT` and `MONTHLY USED/LIMIT` (for example `3/5`, or `49/-` when unlimited).
- **More demo consistency fixes.** The Models page shows a draining node as amber Draining instead of red Down and labels runtimes the way the GPU nodes page does (llama.cpp, not llamacpp). The Model advisor takes each node's GPU runtime and VRAM figures from the same record as the GPU nodes page. The keep-warm notice names a model that really has a digest mismatch. Users show API keys that exist on the API keys page. Analytics attributes cloud requests to the same provider the Routing page lists, its charts no longer depend on an animation to appear, and the first chart has an empty state. Demo audit entries match the runtime, service unit and VRAM of the node they name. The homepage checklist names the same warm model as the demo, and the Metrics page says the endpoint is unauthenticated plain HTTP.
- **The demo describes one consistent fleet.** Node count, healthy count, warm models, runtimes, agent versions, API key prefixes and the cloud-spend warning now come from the same fixtures on every demo page, and the Models header, Dashboard and GPU nodes page agree. A draining node's status dot now reads Draining instead of its underlying health.
- **Settings no longer shows a raw float for the local token value.** Prices stored as 32-bit floats (for example `0.0020000000949949026`) are rounded for display and when saved.
- **Icon-only buttons and switches in the dashboard now have accessible names,** so screen readers announce what each toggle, delete, view and reorder control does.
- **Website:** the install command wraps on narrow screens instead of truncating, in-page nav links update the URL fragment so sections can be deep-linked, and docs code blocks wrap on phones. The production guide now notes that the metrics endpoint is plain HTTP and should stay on a private network or sit behind a TLS proxy.
- **The thermal watchdog now auto-drains a remote node from its marbor agent's GPU temperature.** Until now only a node on the marbor host itself (read through `nvidia-smi`) could be drained for heat; a remote node showed a temperature and never drained. Upgrade note: if `thermal_watchdog` is already enabled, remote agent nodes will now start draining once their GPU stays at or above `max_temp_celsius` for `consecutive_breaches` polls. The temperature counted is the hottest card in the node's GPU scope (the declared GPU indices, else a verified detected group); with no scope known, or a scope that matches none of the reported cards, the hottest card on the host counts, so on a host shared by several nodes with no scopes set, one hot card drains all of them. A card that reports no temperature, a missing GPU block and Apple Silicon (no temperature) never drain, and a node whose agent is unreachable is not watched at all until the agent answers again. The temperature comes from the agent, so a compromised agent can drain every node on its host. One threshold applies to every vendor, and AMD reports junction temperature, which reads hotter than NVIDIA core temperature, so pick `max_temp_celsius` with your hardware in mind. The dashboard still shows card 0's temperature while the drain looks at the hottest in-scope card; the server log names the card and reading. Recovery stays manual (undrain), and a node that is still hot drains again on the next poll. Off by default.
- **A node URL with an IPv6 zone ID can no longer reach a link-local address.** `http://[fe80::1%25eth0]:11434` was accepted as a node URL even though `http://[fe80::1]:11434` is refused, because the zone suffix hid the address from the link-local check. The check now ignores the zone, so both forms are refused.
- **The standalone demo's built-in database matches the demo fixture again.** `demo/docker-compose.standalone.yml` carries its own copy of the demo database and it had drifted from `demo/marbor.demo.db`. `make demo-db` now regenerates both, and a test fails if they differ.

### Changed
- **Draining a node or restarting its runtime now checks whether the node is the only server of any model it holds - API callers get a 409 until they pass `force=true`.** This is a behavior change for scripts: a drain or runtime restart that used to return 200 on a node that is the only replica of a model now returns 409 with `code: "sole_replica"` and the model list; pass `force=true` (CLI: `--force`) as a query parameter to proceed. Only the exact value `force=true` counts; anything else is treated as not set. To migrate a script, call `GET /admin/nodes/{name}/sole-replica` first, then add `force=true` to the drain or restart request if you accept the result. On the command line, `marbor runtime drain|restart` now asks `Proceed? [y/N]` on a terminal when the node is the only server of a model, and exits 1 without a terminal unless `--force` is passed. The dashboard and CLI show the models and ask you to confirm. Models that only an Ollama node could load on demand are listed as a note and do not block, and places where replica data cannot be verified are listed and do not block. Known limits: the check covers a single marbor process and only drains and restarts made through marbor, so scheduled drains, the thermal watchdog and runtime restarts done outside marbor are not checked; a runtime restart keeps its node counted as unavailable until a poll after it returns confirms it is back (or 15 minutes pass), and that record is lost if marbor restarts. The check reads each node's loaded models, so a single-model node on vLLM, TGI, llama.cpp or MLX always prompts on drain, and a node with no model loaded yet reports nothing. An Ollama node counts as able to cold-load a model only for another Ollama node; for a model on any other runtime, an Ollama node that does not have it loaded does not count, so a false 409 there is cleared with `force=true`. A restart the agent refuses no longer keeps its node counted as unavailable; if the outcome is unclear (a timeout or dropped connection) the node stays counted as unavailable until a later poll confirms it. In the dashboard, if a stop or restart is rejected because the node has just become part of a multi-host replica, the dialog now reopens with the replica notice so you can review it and confirm again. Stop, start, undrain and model unload are unchanged.
- **Status and chart colors now come from the theme, so several colors change.** Activity badges were eight unrelated hues; they now use the dashboard's own colors: Drain is orange, Agent is blue, Runtime is bronze, Node is green, Warmup is amber, Schedule and Predictive share a green-teal, and Config is grey. The Sidebar's pending badge and dot, the Model Advisor's fit and status colors, and the Analytics (amber) and Metrics (green-teal) chart lines use the same colors instead of fixed ones, so they follow light and dark mode together. In dark mode the info blue is also a little brighter so blue text on its tinted background is easier to read.
- **Dialogs and the Model Advisor panel have slightly smaller rounded corners.** The corner radius goes from 16px to 12px, matching the rest of the dashboard. The primary action buttons also now share one definition, with no change to how they look. Disabled primary buttons that had no explicit rule (Re-point alias, Run Benchmark, cloud Save, Add Schedule) now also show the not-allowed cursor.
- **`install.sh` no longer installs man pages from releases older than v0.25.0.** Those publish them as loose files rather than the archive; the installer now says the archive is missing and skips the man pages (the binary install is unaffected).

## [0.25.0] - 2026-10-13

### Added
- **Model aliases can be edited in place.** A pencil button on each alias loads it into the form so you can change its target or rename it (a rename saves the new name and then removes the old one). Changing the target of an existing alias still asks for confirmation first.

### Changed
- **The UI now builds on Node 26.** CI, the release and Pages workflows and the Docker image all build the UI on Node 26 (the latest release). Building the UI yourself needs Node 22.12 or newer.
- **Releases ship the man pages as one archive.** `marbor-man-pages.tar.gz` replaces the 22 separate `marbor*.1` files and `MANIFEST.txt` that each release used to carry, which cuts a release from 34 uploaded files to 12 (the ten binaries, `checksums.txt` and the archive). The pages sit under `man1/`, so `sudo tar -xzf marbor-man-pages.tar.gz -C /usr/local/share/man` installs them. `install.sh` now checks the archive against `checksums.txt` before installing it, and skips the man pages (the rest of the install is unaffected) if it cannot be verified. Installing while the latest release is one of the older ones still works, and `uninstall.sh` removes the same files as before.
- **The marbor binary's source moved to `cmd/marbor`.** It sits next to `cmd/marbor-agent` now instead of in the repo root. `make build`, the Docker image and the release binaries are unchanged. If you build by hand, use `go build ./cmd/marbor` (a bare `go build .` in the repo root no longer builds a binary), and `go install github.com/Anirudhx7/marbor/cmd/marbor@latest` for `go install`.
- **All demo files now live in `demo/`.** The two compose files are `demo/docker-compose.yml` (builds from your checkout, what `make demo` runs) and `demo/docker-compose.standalone.yml` (the self-contained copy that was `docker-compose.demo.standalone.yml`). The mock-node and traffic-generator Dockerfiles, the seeded `marbor.demo.db` and `seed_demo.sql` moved there too, and the demo compose project is now named `marbor-demo`. `make demo`, `make demo-down`, `make demo-db` and `make smoke` work as before; if you ran `docker compose -f docker-compose.demo.yml` by hand, use `-f demo/docker-compose.yml`, and a standalone file you already downloaded keeps working unchanged.
- **Metrics now shows tokens per second per hour, and Analytics owns cost and routing.** Metrics no longer repeats the requests per hour, requests by model, request distribution and savings charts or the cost and request tiles, and it loses its per-chart "Export CSV" buttons (Analytics keeps its server-side Hourly and Models CSV exports). Its one chart plots generation speed per hour for the last 24 hours, with gaps for hours that have no generation timing. Only Ollama-native requests report timing, so other runtimes show no data there, and a marbor restart clears the chart. The Prometheus and Grafana block stays. The page subtitle no longer says "Ollama deployment".
- **Analytics gains a "Saved vs spent per hour" chart.** It plots estimated savings (against cloud list rates) next to actual cloud spend for each of the last 24 hours.

### Fixed
- **Ping warmup no longer cancels its own pings, and a failed ping says why.** The warmup ran under the request that triggered it, which is cancelled as soon as the response is written, so the Ping button aborted its pings with "context canceled". It now runs independently of the caller, and when the runtime rejects a ping the runtime's own error text is shown instead of only a status code.
- **The published Docker image ran the agent, not marbor.** `ghcr.io/anirudhx7/marbor:latest` and the version tags carried the `marbor-agent` binary in place of the control plane, because both release builds name their output `marbor` and the image picked up the wrong one. `docker run ghcr.io/anirudhx7/marbor` did not start the dashboard or the proxy. The next release's image contains the control plane. Until then, use the install script or a release binary, or build the image from the repo's `Dockerfile`. A new pre-release check now starts the built image and fails the release if it does not report `marbor` and answer `/health`.
- **Model metrics keep the real name of installed models.** A model that is installed on a node but not currently loaded, or written with or without `:latest`, now keeps its own label in request metrics instead of being counted as "unknown". A model reads "unknown" until its node's model list has been fetched once.
- **Repeated Content-Encoding headers are rejected.** A compressed request body is now refused with a 415 even when the encoding is on a second header line or in a comma-separated list.
- **A huge number in an unrelated JSON field no longer fails the request.** A value such as `1e999` outside the `model` field is accepted instead of being reported as invalid JSON.
- **A request body that starts with `{` but is malformed now gets a 400.** Truncated, duplicated-key or too deeply nested JSON objects used to be forwarded with an empty or guessed model name; they are now refused up front. Requests with no body are unaffected.
- **Hugging Face GGUF model picker: one row per quantization, so "Pick" lands on a single row.** A multimodal projector file in a subfolder (for example `BF16/mmproj-BF16.gguf`) was listed as if it were a model quantization, and two files sharing one quant label produced two rows with the same pull tag, both marked "Pick". Projector files are now excluded by base name and files sharing a tag collapse to the largest one.
- **"Ping warmup" no longer fails with "warmup not enabled" when only per-node keep-warm is configured.** The button was blocked by the global warmup setting even though per-node keep-warm works without it.
- **Hardware benchmark: the model list falls back to the node catalog when the agent's list call fails,** and shows the agent's actual error. It previously fell back to currently-loaded models only, which is empty when nothing is loaded.
- **A fallback model chain now respects the key's model list.** When a request is moved to a smaller alternate because the requested model does not fit, an alternate the key is not allowed to use is skipped. A request that falls through to a cloud provider after such a swap now sends the model name the client asked for, and no longer carries the local fallback header.
- **A client that cancels or disconnects no longer marks a node unhealthy.** A cancelled request, or a client that walks away mid-stream, is no longer recorded as a node failure, so it cannot push a healthy node into error cooldown or skew its success history and prefix locality.
- **A retried request now gets the retry node's own model profile and limits.** When a request fails over to another node, or a degradation swaps the model, that node's default options and its requests-per-minute and tokens-per-minute caps are applied instead of the first node's.
- **A failover node whose model rate cap is spent is skipped.** The next candidate, degradation, cloud fallback or the original upstream error is used instead of a 429 about a node the client never chose. The 429 for the first node no longer names the node.
- **A degradation chain tries later entries when an earlier one is capped.** If the first fallback model's node cap is spent, the next entry in the chain is tried before cloud fallback or an error.
- **Very long model names are rejected.** A model name over 256 bytes now returns a 400 instead of being copied into metrics, logs and the audit trail. Rejected requests are logged.
- **Ambiguous model keys are rejected.** A JSON body with a repeated `model` key, or a key such as `MODEL` that differs from `model` only in case, now returns a 400, so the key's model list and a case-sensitive backend can no longer read different names from one body.
- **Malformed or absurdly nested JSON bodies are rejected with a 400.** A body that starts as a JSON object but is invalid, has trailing data, or nests more than 1000 levels deep is refused instead of being forwarded unchecked.
- **Compressed request bodies are refused with a 415.** marbor reads and forwards request bodies as plain bytes, so a body with a `Content-Encoding` other than identity is no longer passed on.
- **Unknown model names share one metric label.** A model name that is not on the fleet, an alias, or in a configured chain is counted under `unknown`, so a caller cannot create metric series by inventing names.
- **Query strings are no longer forwarded to cloud providers.**
- **Token counts are no longer lost for large single-line replies.** A final response line larger than 8 KiB used to be cut away, reporting zero tokens; it is now kept whole, and reported as unknown only when it is too large to retain. The usage line of a streamed reply is also kept when it is followed by `data: [DONE]`, and an unreadable last line is reported as unknown instead of falling back to an earlier line. Pretty-printed multi-line replies are still counted.
- **The request log shows the real latency of 5xx responses** instead of zero, and a request cancelled before any response is no longer logged as HTTP 200.
- **Disallowed model names no longer become metric labels.** A request for a model outside the key's list is counted under a fixed label.
- **Cloud providers now receive only an allow-list of client headers.** Only Content-Type and Accept are forwarded from the client, with marbor's own credential, request id and a neutral user agent; cookies, client-address and proxy headers, session ids, other credentials and org or project selectors are no longer sent, and a `Connection` header can no longer strip marbor's own headers from a node or cloud request.
- **Local nodes get connect and TLS handshake timeouts.** Connections to a node give up after 5 seconds (TLS handshake 10), and idle connections are pooled and expire, so a powered-off node no longer holds a request until the operating system gives up.
- **More unsupported OpenAI paths are blocked clearly.** `/v1/fine_tuning`, `/v1/vector_stores`, `/v1/uploads`, `/v1/responses` and `/v1/evals` now return a 501 instead of being forwarded to a node.
- **Path traversal forms of blocked Ollama endpoints are blocked.** Paths such as `/api/../api/pull`, `//api/pull` and `/api/pull/` are cleaned or matched without the trailing slash before the management-endpoint check, and the cleaned path is what is forwarded. Request paths in the proxy's log lines are now quoted and truncated, so an encoded newline cannot forge a log line.
- **Mid-stream upstream failures now reach the client as an aborted stream.** When a node or cloud provider dies after it has started replying, the client's connection is now aborted instead of ending as if the reply were complete. A panic after a response has started also aborts the connection, and a panic before it returns the OpenAI error shape on the proxy port.
- **Panics no longer leak connection counters.** A node's connection and in-flight counts are now released even if a request panics, so a crash can no longer make a node look permanently busy.
- **API key usage and policy no longer get lost.** Daily and monthly USD caps are now kept on keys added or rotated through a reload. Key usage counters are now restored on restart instead of being overwritten with zeros by the next flush. `local_only`, the USD caps and `allow_local_degradation` are now applied at startup instead of only after a reload. The usage flush no longer stops at the first key that fails to save, and reports every failing key by name. `PATCH /admin/keys/{name}` now rejects negative limits and USD caps with a 400, the same as key creation. Restored counters now roll over at local midnight instead of UTC midnight.
- **Router: an unload that gets a non-2xx reply now counts as a failure.** Before, a model could be reported as unloaded when the node answered with an error status.
- **Router: a node URL with no port now matches its agent runtime on port 80 or 443.** An http or https node address without an explicit port is matched to the agent runtime listening on the default port for that scheme.
- **Router: health, telemetry and eviction correctness fixes.** A data race on a node's single loaded-model size is gone, and a panic in an agent poll or a scheduled unload is now recovered, logged and counted as a failure. Two warmups for one node can no longer both evict inside one cooldown window; freed VRAM is credited with the same overhead factor it was charged with, and models of unknown size are no longer picked as eviction victims. Bare model names such as `llama3` now match `llama3:latest` for size estimates, last-used and last-known-size tracking, and the downloaded-model check (which now skips unhealthy nodes). The agent unload call requires a 2xx status, caps what it reads, keeps the decode error and brackets IPv6 hosts; a scheduled unload that fails because the node is down now appears in the node's unload errors. The runtime poll no longer wipes agent-supplied temperature, power or VRAM total, while stale power, CPU and VRAM readings now clear when the agent stops reporting them. A never-reached agent no longer fires a false "agent up", a deployment report with no port is no longer attributed to one of several nodes on a host, and nodes on a default port (80 or 443) match their runtime by it. Probe failures, Docker discovery errors, agent poll failures (including a TLS fingerprint mismatch) and model catalog fetch failures are now logged, rate limited and never including the token. A failed network interface lookup no longer makes local nodes look remote for 30 seconds, the "might be MLX" hint keys off a typed `/health` 404 instead of error text, and the per-node last-used and last-known-size maps are capped.
- **Router: second round of eviction, telemetry and safety fixes.** Eviction no longer credits VRAM it never counted as used, no longer tries to unload from runtimes that cannot unload (it says so in the log once per interval), and a failed headroom check now skips the warm load instead of loading anyway and releases its reservation. A scheduled unload that skips pinned models still reads as ok, with the skip noted as a warning, and a skipped warm load now says whether the cooldown or a lack of room caused it. A panic during a warm load no longer leaves the node load lock held. The last-known size fallback for eviction applies only to runtimes that report no per-model size. A removed node can no longer have its last-used or last-known-size entries re-created by an in-flight request, and the warm-state restore skips nodes that no longer exist. Node and agent connections, the webhook, the cloud-provider key test and the model-pull calls no longer follow redirects, and a redirect reply counts as a failure. Agent reports are bounded before use: sizes, counters, ports and GPU indexes are clamped, percentages held to 0-100, lists and strings capped, and control, line-separator and bidirectional-override characters are removed from stored text. A status reply that is empty (`null`, `{}`), from another service, or without a node_id no longer counts as a healthy agent. VRAM on a multi-GPU agent host is the sum over all devices, the prefix cache hit rate reads as unknown (not 100%) when hits exceed queries, and a cached local nvidia-smi reading expires after three poll intervals. A model size that is unknown only because the node's model list could not be fetched no longer counts as fitting, the unknown-size placeholder reservation is no longer shown as pending pre-warm memory, and a single-model node's whole-GPU VRAM is no longer remembered as that model's size. Log lines built from untrusted text are now cut on a character boundary with control characters removed.
- **Requests to an Anthropic cloud provider through `/v1/chat/completions` without `stream` now get a normal JSON reply.** They were answered as a stream even though OpenAI clients expect no streaming unless asked. Ollama `/api` requests still default to streaming.
- **Streamed Anthropic replies now count tokens.** The final usage chunk carries `total_tokens`, so streamed `/v1` requests to Anthropic no longer record 0 tokens.
- **A truncated Anthropic stream now surfaces as an error.** A stream that ended before it completed used to close as if it had finished; the client now receives an error chunk and the stream ends with an error.
- **An Anthropic stream that ends with an error event (for example overloaded) now surfaces as an error.** The client receives the error message, and Ollama clients see that message instead of a generic one.
- **Anthropic finish reasons are mapped to OpenAI ones.** `end_turn` and `stop_sequence` become `stop`, `max_tokens` becomes `length` and `refusal` becomes `content_filter`, in both streamed and non-streamed replies.
- **Ollama clients now get `done_reason` from cloud replies.** A reply cut off by the token limit shows `done_reason` `length`, and a normal finish shows `stop`.
- **Ollama clients now get string-shaped errors from cloud failures.** An error reply from a cloud provider (such as a bad key) used to reach `/api` clients as the provider's raw JSON object; it is now `{"error":"<message>"}` with the same status. `/v1` clients still get the provider's body.
- **The real cause of a cloud stream failure is now logged on the server.** A dropped connection, an oversized frame or a provider error event is logged with its cause; clients are told when a frame was too large rather than that the stream ended unexpectedly.
- **Cloud replies larger than 64 MiB, or that cannot be read, now return 502.** Marbor used to return a truncated or partial body with a 200 status; the cause is logged on the server.
- **A cloud provider's `Retry-After` header is now kept on rate-limit and overload errors.** A 429 or 503 from the provider reaches Ollama clients with its backoff hint; it is dropped only on the 502 that marbor itself returns for an unreadable or oversized cloud reply.
- **Cloud replies in Ollama format now report `done_reason` `stop` when the provider names no finish reason**, for streamed and non-streamed replies alike; marbor defaults it. For OpenAI-shaped (`/v1`) clients, a reply from Anthropic that names no stop reason has no `finish_reason` field, as before.
- **Cloud error messages are clearer.** A non-streaming provider error with no message now says "upstream error" instead of mentioning a stream, and a provider error chunk is logged once, shortened, on a single line.
- **The Anthropic temperature limit is now logged at most once a minute**, so a client that always sends a high value no longer fills the log.
- **Cloud SSE lines written as `data:` with no space are now read.**
- **Anthropic: `developer` messages are now sent as the system prompt**, and empty system messages are skipped.
- **Anthropic: temperature outside 0 to 1 is now limited to that range** (OpenAI allows up to 2), and the change is logged.
- **A model profile's system prompt is now applied to Ollama chat requests.** It was added as a top-level field that `/api/chat` ignores; it is now added as a leading system message when the client sent none.
- **A request `options` value that is a string, array or number is no longer replaced by the model profile defaults.** The profile defaults are not applied to that request.
- **A request with `"options":null` now gets the model profile defaults.** Null is treated as not set, as Ollama does.
- **Agent: the Docker socket path no longer leaks a socket and goroutines per container on every refresh.** Each collection now closes its connections to the Docker socket when the request finishes, so an agent that runs in a container no longer drifts toward file descriptor exhaustion.
- **Agent: a hung `ps` can no longer stall the agent.** The process listing now has a 4 second limit, reads standard output only (a warning on standard error can no longer be parsed as a process) and is capped in size.
- **Agent: only real runtime server processes are reported as deployments.** A process now counts only when its executable (or the python module it runs) is the runtime, so `tail -f vllm.log`, `docker logs vllm`, `ssh gpu1 vllm serve`, an editor open on a llama.cpp file and `ollama runner` children are no longer reported as runtimes. A runtime process with no `--port` also no longer takes the name or port of a different runtime that happens to be the only one detected. `--top-p 1` or `--min-p 5` are no longer read as a port.
- **Agent: Docker containers no longer report "every GPU visible" from an image default.** NVIDIA images bake `NVIDIA_VISIBLE_DEVICES=all` into their environment, so a container started with `--gpus device=0` was reported as seeing all GPUs. That value is now reported as unknown, the container's GPU device request (`--gpus device=...`) is used when present, and, where `nvidia-smi` is available, the GPUs the runtime actually holds are reported (marked unconfirmed) instead of leaving the scope unknown.
- **Agent: a Docker container's reported port is now stable and is the host port.** It was picked from a randomly ordered map, so it could change between refreshes, and a `--port` inside the container was reported as if it were reachable on the host. Now the host port published for the container's `--port` is used, host networking uses the port as given, and when several ports are published with no `--port` to tell them apart, no port is reported.
- **Marbor no longer treats a running process as stopped when it is owned by another user** (a signal probe that returned a permission error was read as "not running").
- **Turning demo mode off no longer leaves demo data on screen.** The GPU Nodes replica-group suggestions and the Model aliases list are cleared when demo mode changes, so a sample group can no longer appear as live (or be confirmed against the real API), and a slow response from the previous mode can no longer overwrite the current rows.
- **Saving a model alias over an existing one now asks first.** The dialog shows the old and new target, since the change re-points live traffic immediately. The alias form also submits on Enter and its buttons are larger tap targets.
- **Dialogs are keyboard and screen-reader friendly.** They announce as dialogs with their title, take focus on open, close on Escape, keep Tab inside, and return focus afterwards; form errors in the replica and alias dialogs are announced. The replica confirm dialog no longer blanks its title and body while closing.
- **The Nodes / Replica groups switch is exposed as a button group** instead of a tab list that lacked tab panels and arrow-key support.
- **Smaller fixes:** a node named `gpu-node-1` is no longer matched inside `gpu-node-10` when listing nodes to fix before adopting a replica group; dismissed "multi-host group detected" prompts for groups that no longer exist are dropped from browser storage; duplicate "still needed" entries no longer share a React key; retrying a demo pull stays a demo pull instead of calling the real API; the Model Advisor now shows a pull as in progress for models given as bare Hugging Face GGUF repo names.
- **`marbor prefix-locality stats` no longer shows a made-up `hit_rate=0.0%` when the server omits the rate but reports real hit and miss counts.** The rate is now worked out from the counts.
- **The startup banner no longer prints the exposed-dashboard warning when the admin bind address is `localhost`.** Only the loopback classification changed; who can reach the dashboard and the first-login rules are untouched.
- **Changing your password now requires the current password unless the session is a forced-change session (first login or an admin-issued temporary password).** Previously an account that had used "Skip for now" on the forced password change could set a new password from its normal session without entering the current one; now the current password is required there. The forced-change session itself is unchanged.
- **An agent status reply over 8 MiB no longer takes the node out of rotation.** The router used to treat it as agent unreachable; now the node stays reachable and routable, its live VRAM, temperature, power, queue and runtime readings show as unknown until the report shrinks, and replica detection ignores that host meanwhile (logged at most every 10 minutes per host). An oversize reply that is not a JSON object, one that cannot be read, or one that is not valid JSON still counts as agent unreachable. This supersedes the 0.24.0 behavior where an oversize reply counted as a failed poll.
- **A multi-host replica whose worker host agent stops answering is no longer chosen as a routing target.** When the marbor agent on the host of a non-head member goes dark (3 consecutive failed polls, about 6 seconds at the default 2 second poll interval when polls fail fast, up to about 21 seconds when each poll hangs to its 5 second timeout; cleared on the first good poll), the replica's head is left out of routing and the request explain view shows `replica_member_unreachable` for it. A worker with no agent is unknown and gets no protection, and an agent reply over the size cap does not count as down. If excluding the head would leave no candidate at all, the head is used anyway as a last resort and the explain detail says so, so a dead agent alone never turns the model into a total outage. Heads kept as a last resort are always listed in the explain view, even past its 20-entry cap on excluded nodes, so that list can run longer than 20. A pinned session whose head is excluded this way is routed elsewhere for that request only; its pin is kept and resumes when the agent answers again; while the agent is dark, that session's requests are scored independently and may land on different nodes. Routing excludes the head; keep-warm does not. The explain view also now labels `replica_worker` and `replica_unresolved`. Known limits: the signal is the agent, not the worker, so anything that makes an agent poll fail looks like a dead worker (agent token rotation or a 401, a TLS fingerprint mismatch, any non-200 reply, a read or decode error, an over-cap reply that is not a JSON object), and a worker that is alive behind a dead agent is treated as unreachable until the agent is fixed; one dark agent marks every node on its host stale; there is no hysteresis beyond the 3-failure threshold. vLLM and llama.cpp RPC groups are detected automatically; TGI, MLX and Ray need a manual replica declaration. All of this is verified with fakes, not real multi-host hardware.

### Security
- **Node, cloud-provider and webhook URL validation now also rejects link-local/metadata addresses written in legacy IPv4 forms.** This covers dotted hex/octal/mixed spellings (for example `0xA9.0xFE.0xA9.0xFE`, `0251.0376.0251.0376`, `169.254.43518`) and one trailing dot, which glibc-based resolvers accept. Dotless all-digit hosts with a leading zero are now also checked as octal, and a part with a leading zero is checked in both its octal and its decimal reading, so a host is refused if either reading is link-local. A bare `0x` part (a hex prefix with no digits) is now refused as well, since it reads as 0. Linux release binaries use the pure-Go resolver and did not resolve these forms; Windows uses the system resolver (GetAddrInfoW) and was tested not to resolve them; source builds with cgo on glibc could, and macOS was not tested. These URLs are admin-supplied, so this is hardening. If startup fails after an upgrade, the error names the offending node, cloud provider or webhook URL; correct that URL if it is written in one of these forms.

## [0.24.1] - 2026-10-05

### Added
- **`marbor prefix-locality stats` shows prefix-locality routing hit and miss counts and the hit rate.** Matches the Warmup page card; the hit rate shows n/a when no requests have been counted yet. No new Admin API capability.

### Changed
- **The first login of the default `admin` / `admin` account can no longer be skipped.** A fresh install creates the account with the password change pending and the skip allowance already used up, so "Skip for now" is not offered and the server refuses it; an existing install whose admin is still on the default password with the change pending is raised to the same state at startup. A new password also cannot be `admin` or the current password, and the address that completes the first change is logged. Accounts with an admin-issued temporary password can still skip.
- **CI: `go test -race` timeout raised from 600s to 900s** in `scripts/gate.sh`, `scripts/sanity.sh` and the workflow, because the `internal/admin` race suite reached the 600s limit on CI and failed a run. No user-visible change.

### Fixed
- **Dismissing or restoring a replica suggestion no longer erases your other dismissals when the saved list cannot be read.** Previously a failed read was treated as "no dismissals", so the next dismiss or restore overwrote the stored list with just the new entry (or an empty one). Now, if the saved `replica_suggestions_dismissed` setting cannot be read or is not a valid list, dismiss and restore fail with a `500` error that names the setting and says nothing was changed, and the stored value is left untouched. This includes restoring a suggestion that was not dismissed, which used to return success and now errors on a failed read. Listing suggestions (`GET /admin/replica-suggestions`, `marbor nodes suggestions`) also returns the error instead of silently showing no dismissals. A missing, empty or `null` saved value is still treated as an empty list. If the stored value is corrupt, delete the `replica_suggestions_dismissed` row from the `settings` table in the marbor database and retry; the setting is not editable through the settings API.

## [0.24.0] - 2026-10-04

### Added
- **Model Advisor and `marbor fit` now cover vLLM and TGI nodes for a curated set of ungated model families.** Each shows a Hugging Face equivalent with an approximate download size and an estimated VRAM need, and Pull downloads it through the node's agent. Pull only downloads: relaunch the runtime to serve the new model, so the Verify the install works check is hidden for these runtimes on every pull surface (Model Advisor, the What fits list and the Models page), the pull widget says "Downloaded. Relaunch the runtime to serve it.", and `marbor models pull` prints the same reminder on stderr for vLLM and TGI nodes (stdout is unchanged). Full precision (BF16) only, sized against the largest single GPU; unknown sizes show `-`. Llama and Gemma (gated), Mistral and Mixtral (not yet mapped), models without a mapped repo, and llama.cpp and MLX nodes still show curated models as another format; browse Hugging Face for those. The Verify option is hidden in the UI only; the pull API still accepts `verify_load`.
- **Stopping, restarting or draining a node in a multi-host replica now says what it affects.** The confirm dialogs on GPU nodes, `marbor runtime stop|restart|drain`, and the Admin API responses name the replica head and members and state the effect: stopping or restarting the head takes the whole replica offline, doing it to a worker breaks the replica, and marbor does not restart the other members. Draining the head drains the replica; draining a worker has no routing effect, so it says to drain the head instead. Successful responses carry an additive `replica` object (`role`, `head`, `members`, `warning`); standalone nodes are unchanged. The audit log entry for these actions records the replica role and head.
- **Agent reports distributed-runtime evidence.** `marbor-agent` now includes, per runtime, the multi-host launch facts it can see: for vLLM the `--nnodes`, `--node-rank`, `--master-addr`, `--master-port` and `--headless` flags, for llama.cpp the `--rpc` server list (also from `LLAMA_ARG_RPC` when `--read-runtime-env` is on or the runtime runs in Docker). It also reports the host's own non-loopback IP addresses (`host.addrs`). Only those named values are read, never the rest of the command line or environment. The agent advertises `deployment.report` and `deployment.topology` so a marbor can tell an older agent from one that found nothing. This release only collects and reports the evidence: marbor does not act on it yet, so replica membership still comes from the declared `replica_peers`. Ray, TGI and MLX multi-node launches are not detected. Verified with recorded process lines and fake processes; not yet run against real multi-host deployments.
- **Agent opt-in `--read-runtime-env`** (also `MARBOR_AGENT_READ_RUNTIME_ENV=1`, and `marbor-agent service install --read-runtime-env`) reads each runtime process's own environment on Linux to learn which GPUs it was pointed at. Only the five GPU visibility variables (and llama.cpp's `LLAMA_ARG_RPC`, see below) are kept; every other variable (tokens, keys) is discarded in memory and never stored, logged or sent. It needs the same user or root and is off by default. Without it the agent can still report where an NVIDIA runtime sits from the GPU process list, marked unconfirmed.
- **Per-process NVIDIA cross-check.** With the opt-in on and `nvidia-smi` available, the agent compares the GPUs the environment names with the GPUs the runtime's process tree actually holds. A match marks the scope confirmed (and fills in indices when the environment used UUIDs); a mismatch is reported with both sides and trusted by neither. AMD, Intel and Apple scopes are reported from the environment only and are never marked confirmed; Windows is not covered.
- **Pipeline and data parallel degrees for vLLM.** `--pipeline-parallel-size` and `--data-parallel-size` are read next to `--tensor-parallel-size`, so a tensor 4 x pipeline 2 launch no longer shows as plain 4-wide.
- **`marbor nodes` has a DETECTED GPUS column** (`0,1 [verified]`, `2,3 [unverified]`, `unknown`, `-`), and the node API gains additive `detectedGPUScope`, `detectedPipelineWidth`, `detectedDataWidth` and `detectedDrivesPlacement` fields. The agent advertises a new `deployment.gpu_scope` capability; older agents and servers keep working.
- **Replica suggestions.** Marbor now turns the multi-host launch details agents report into confirmable replica suggestions. vLLM multiprocessing and llama.cpp RPC groups are matched across nodes by address. A suggestion is never applied on its own. Confirm declares the same replica membership on every member in one all-or-nothing step, and clearing the declaration on every member undoes it. Incomplete, conflicting and contradicting-declaration cases are shown with the reason instead of being guessed. `GET/POST/DELETE /admin/replica-suggestions...`, `marbor nodes suggestions [confirm|dismiss|restore]`, and a Replica groups tab on the GPU nodes page. Dismissals persist per group and return if the group's members change. Needs a current agent; older agents show "agent update needed".

### Changed
- **Documentation corrections: audit trail, drop accounting and logging.** The audit trail is now described as best-effort and retention-pruned rather than append-only (README, ROADMAP, Settings page), the write-path limitations note reflects that audit drops are counted in `marbor_audit_dropped_total` and warned about at most every 30 seconds, the metrics endpoint is documented as available only when metrics are enabled, the structured access log description now says rejected requests are not written to it, and the logging table now covers source IPs and states that none of the listed fields are hashed or redacted.
- **Breaking: runtime stop and restart on a multi-host replica member now need an explicit acknowledgement.** `POST /admin/nodes/{name}/runtime/stop` and `/restart` on a replica head, worker or unresolved member return `409` with `code: "replica_member"` and a `replica` object unless the request carries `?acknowledge_replica=true` (only that exact value counts; `false`, `1`, `yes` and `TRUE` are still rejected, and the 409 text says so). `marbor runtime stop|restart` gain `--acknowledge-replica`; the dashboard sends the acknowledgement only from its confirm dialog. Start, drain and standalone nodes are unchanged, and the rejection happens before any agent call. **Migration:** scripts that stop or restart a replica member must add `--acknowledge-replica` (CLI) or `?acknowledge_replica=true` (API). The CLI now reports any `409` from the Admin API as a user error (exit 1) instead of a server error (exit 2).
- **Audit log drops are now counted and surfaced.** New Prometheus counter `marbor_audit_dropped_total` increments whenever the async audit queue (still non-blocking, 5000 entries) is full and an entry is discarded. The per-entry drop log line is replaced by a warning at most every 30 seconds that includes the cumulative dropped count. README, SECURITY, PRODUCTION and SAVINGS-MATH docs no longer claim every request is recorded: the audit log is best-effort, covers requests that reach proxy completion handling, and does not persist authentication/policy rejections.
- **Breaking: foreground `marbor-agent` needs TLS or an explicit opt-in.** A hand-written unit or script running `marbor-agent --port=...` with no certificate now exits unless it adds `--bind=<loopback IP>` or `--allow-insecure-plaintext`. Without `--cert`/`--key` the agent now exits with an actionable error unless it is bound to loopback (new `--bind` flag) or `--allow-insecure-plaintext` is passed. `marbor-agent service install` provisions TLS and is unaffected (it cannot set `--bind` or the opt-in flag, so a service is always TLS). **Upgrade note:** an agent service installed before TLS provisioning existed has no `--cert`/`--key` in its unit and will refuse to start after the binary upgrade; re-run `marbor-agent service install` (safe to re-run, it reconfigures the existing service) to regenerate the unit with TLS. A lone `--cert` or `--key` is still refused.
- **A malformed stored API-key `expires_at` now fails closed.** Authentication rejects the key (previously it was treated as non-expiring), logs a rate-limited warning with the key name only, and the key list shows `expired`. The `YYYY-MM-DDTHH:MM` format that create/update already accepted is now also understood at authentication time. Empty still means no expiry. **Upgrade note:** after upgrading, check `marbor keys` for keys showing `expired` that you did not expect, and re-set their expiry. Keys with a `YYYY-MM-DDTHH:MM` expiry, which authentication used to treat as non-expiring, now expire at that time.
- **Default admin credential warning strengthened.** The startup banner and a boot-time log line, repeated on every restart while `admin` / `admin` is still active, now state that a password change is required and that the dashboard is plaintext HTTP, and name TLS/reverse proxy as required for exposed deployments. Behavior is unchanged; a first-boot generated password is a proposed, separate change.
- **CI: `go test -race` timeout raised from 300s to 600s** in `scripts/gate.sh`, `scripts/sanity.sh` and the workflow (job limit 15 to 30 minutes, since the sanity and regression steps now both run `internal/admin`); `internal/admin` alone had grown past 300s on CI and the limit was killing a passing run mid-test.
- **Replica groups moved to their own tab.** Multi-host suggestions no longer sit above the node list. GPU nodes gets a Replica groups tab, shown only when a group or a suggestion exists, with the things that need a decision first, then your confirmed groups, then a collapsed list of dismissed suggestions and of what detection could not see. The Nodes tab keeps a one-line prompt when a group is ready to confirm and small chips on the nodes involved. A suggestion that conflicts with an existing declaration shows a per-node before and after with Adopt detected or Keep declared, and an incomplete group says what was seen and what is still needed. Agent-less and single-host fleets no longer see a permanent coverage line.
- **Adopt detected for a conflicting group.** `POST /admin/replica-suggestions/{id}/confirm` accepts an optional `{"adopt": true, "declaredSnapshot": [...]}` body that overwrites the differing declarations on every member in one all-or-nothing step. It refuses (409) if any declaration changed since you looked, if the group is no longer conflicting, or if a node outside the group declares membership naming a group member. `marbor nodes suggestions confirm <id> --adopt` does the same and shows the before and after first. The snapshot is limited to 256 entries.
- **A dismissed suggestion comes back when its state changes.** Dismissals now remember the state the group was in. A group dismissed while incomplete or conflicting reappears once it becomes complete, instead of staying hidden. The dismiss call takes an optional `{"state": "..."}` body; older dismissals keep hiding.
- **CI now crawls the demo build.** After the demo build step, a headless Chrome loads every dashboard route and fails the job on a failed backend request, a page error, a console error or a blank page. The route list is `ui/src/routes.json`, which `App.tsx` renders, so a new page is covered once it is added there. Run it locally with `npm run demo:crawl` in `ui/` after `VITE_FORCE_DEMO=true npm run build`. The first crawl found the demo calling `/health` on every page and `/admin/requests/live` on the dashboard; the demo no longer makes those requests.
- **Detected GPU scope now steers placement only when it was cross-checked.** Before, a GPU list or parallelism width the agent detected could become a placement requirement on its own. Now it does so only when the agent confirmed it against `nvidia-smi`'s per-process GPU list; otherwise the detection is shown on the node and you adopt it (Adopt in the UI, or `marbor nodes patch <node> --gpu-indices ...`). When the check disagrees or finds no process yet, the agent also withholds the older `gpu_group` field so an older server cannot act on an unconfirmed list.

### Fixed
- **Prefix Locality Hints now apply as soon as you save them.** Turning the toggle or changing the score weight on the Settings page used to be stored but ignored until marbor restarted, with no sign that it hadn't taken effect. The running router now picks the change up immediately, and enabling it on a live process reloads the recent locality history the same way a restart would (without overwriting anything recorded since). Turning it off keeps recorded entries until they expire. The other Advanced routing fields (timeouts, retries, session affinity, queueing) still apply on restart, and the card says so.
- **Removing a node and re-adding one with the same name at the same moment can no longer wipe the new node's fresh state.** Removal clears per-node bookkeeping (session pins, last-used times, VRAM readings, eviction and warm reservations) after releasing the main router lock, so a same-name add landing in that window could lose its first entries. Add and remove are now serialized against each other; the effect was a dropped session pin or an extra prewarm, never cross-node leakage.
- **A key whose stored expiry is malformed now leaves a system-audit trail.** The first time a key carries a given malformed `expires_at` (at startup, on reload, or when a key is created or updated), a system-audit event `api_key_expiry_malformed` is recorded with the key name as the target. It never contains the key value or the malformed string, and it is not repeated per request. The key is still rejected as expired.
- **Audit entries logged while marbor shuts down are no longer silently lost.** An entry that reached the audit logger at the instant it closed could be queued after the writer had already drained and exited, so it was neither saved nor counted. The logger now holds a short lifecycle lock around its closed check and non-blocking enqueue, and shutdown takes that lock before draining: an entry accepted before shutdown is written, and one that arrives after is counted in `marbor_audit_dropped_total`. The request path stays non-blocking and never waits on SQLite. Failed store writes are unchanged: they are logged, not counted as drops.
- **A single-node vLLM launch (`--nnodes 1`) no longer produces a "does not state a master address" replica suggestion.** A one node launch has no peers to match, so it is not a group. A launch that states several nodes, or no node count at all, and no master address still shows the card.
- **An agent status response over 8 MiB now counts as a failed poll.** The router used to read the whole body without a limit. Over the cap the host shows agent unreachable (logged at most every 10 minutes per host); a normal report is far smaller, and unknown fields in a smaller body are still ignored.
- **Retained agent evidence is bounded.** At most 64 addresses and 64 deployments per host are kept for replica detection. If one host reports more, the extras are dropped (logged at most every 10 minutes per host) and replica detection can miss groups on that host.
- **A failed read of a stored JSON setting is now logged.** It was dropped silently and the setting fell back to its default.
- **A legacy admin credential with an unusable (old SHA-256) password hash no longer locks the administrator out for good.** The first-start migration used to copy that hash into the users table, where the bcrypt-only login can never match it. It now keeps bcrypt hashes as before (same username, no forced change) but, for any other hash, recreates the legacy username (whitespace trimmed) as an active admin on the documented default password with a forced password change that cannot be skipped, and logs a startup warning naming the account (never the hash or any password). A failed legacy migration is now logged and stops there instead of falling through to creating a second `admin` / `admin` account. Fresh installs are unchanged. The default-credential boot warning now covers any admin account still on the default password, not only one named `admin`.
- **GPU scope detection no longer invents or misreads GPU lists.** The agent used to read its own environment instead of the inference runtime's, fill in "every GPU on the host" when it found nothing and label that a measurement, and silently drop UUID and MIG entries (so `0,GPU-...` became just `0`). It now reports only what it can show: the runtime's own GPU visibility variable (opt-in, see below), UUID/MIG/sub-device entries kept verbatim, and "unknown" with a reason when it cannot tell. The variable list is now `CUDA_VISIBLE_DEVICES`, `NVIDIA_VISIBLE_DEVICES`, `ROCR_VISIBLE_DEVICES`, `HIP_VISIBLE_DEVICES` and `ZE_AFFINITY_MASK` (the undocumented `XPU_VISIBLE_DEVICES` is gone).
- **llama.cpp `llama-server` is now recognised by the agent's deployment detection.** The agent only recognised llama.cpp when its command line contained both "llama" and "cpp", so a plain `llama-server` process produced no deployment report at all. It is now recognised by the exact name of its executable (`llama-server`, never `llama-cli` or a look-alike name), so llama.cpp hosts that reported nothing before now report a deployment (runtime, port and any parsed flags), and the node shows it as detected information under the usual rules: a declared value always wins and an unconfirmed detection does not steer placement.
- **Detected topology no longer changes routing unless it was confirmed.** Before, any GPU group or parallelism width the agent detected silently became a placement requirement when nothing was declared, so a wrong guess could exclude a healthy node. Now a detection steers placement only when the agent confirmed it against `nvidia-smi`'s per-process GPU list (NVIDIA only); everything else is shown on the node as information to adopt. Declared values still always win. Existing fleets that relied on an unconfirmed detection should click Adopt (or run `marbor nodes patch <node> --gpu-indices ...`) to make it a declaration.
- **Adopt now appears when only a GPU list was detected** (no parallelism flag), and adopts just that list.

## [0.23.1] - 2026-10-02

### Changed
- **Model Advisor "What fits" list can pull the pick.** Each picked row has a Pull button that asks for confirmation (tag, quantization, estimated size, node) and then starts the pull in the existing pull widget; it is disabled while that tag is already pulling on the node. Rows that fit now come first under "Fits (n)", the rest under "Does not fit (n)", with column labels on wider screens and no inner scroll box. A model already on the node shows a "Downloaded" label, and the full tag is shown under the quantization.
- **Disk-short rows show "No Disk Space" instead of "Unknown".** A model that would fit in VRAM but not on the node's free disk now gets the same red badge the detail panel uses.

## [0.23.0] - 2026-10-02

### Added
- **`marbor fit` and a per-node quantization pick.** Model Advisor now lists, for the selected node, which built-in catalog models fit its total VRAM and disk and which quantization to pull, and the new `marbor fit` command shows the same for every node or for a Hugging Face repo (`marbor fit owner/name --node <node> --ctx <tokens>`). The pick is the catalog's recommended quantization when it fits, otherwise the largest one that fits, flagged as a tight fit when it uses most of the VRAM; if nothing offered fits it says so plainly. Figures come from published file sizes or the built-in catalog and are labeled as estimates; when a node's VRAM or a model's size is unknown the answer is `-`, never a guess. The catalog and repo Admin API responses gain an additive `recommendation` field. Read-only: nothing is pulled, changed or routed differently. Alternative-repo suggestions for vLLM, TGI and MLX are not included yet.

## [0.22.2] - 2026-10-01

### Fixed
- uninstall: stopping a pidfile-tracked marbor process now works on Windows

## [0.22.1] - 2026-10-01

### Fixed
- **Pinning or keep-warming a bare model name now covers its `:latest` tag.** Pinning `llama3` did not protect a loaded `llama3:latest` from eviction, scheduled or manual unload, and a bare name in a keep-warm list did not get its keep-warm priority during warmup evictions. Both now treat a bare name and its `:latest` form as the same model, as keep-warm loading already did. A manual unload of `llama3:latest` is refused while `llama3` is pinned (same message as for an exact pin), and benchmark eviction and the pinned-and-warm routing preference follow the same rule. Other tags are unaffected: `llama3` never matches `llama3:8b`.
- **Per-node warmup lock bookkeeping no longer grows for removed nodes.** The lock that serializes a node's make-room-then-load step is now released from memory once no warmup is using it, instead of lingering for the life of the process for every node name ever warmed. No change to warmup behavior.

## [0.22.0] - 2026-10-01

### Added
- **Model aliases: keep existing client model names while serving local models.** Operators can map a requested model name (for example `gpt-4`) to a real model on the fleet (for example `llama3.2:8b`) through the Admin API (`GET/PUT/DELETE /admin/model-aliases`), the CLI (`marbor models alias list|set|remove`), or the Routing page. An aliased request is routed, rate-limited, and recorded under the real model, the request log shows `gpt-4 -> llama3.2:8b`, and the response carries an `X-Marbor-Model-Alias` header. `GET /v1/models` and `GET /v1/models/{id}` list an alias while its target is on the fleet, with the target's status. A key's model allow-list accepts either the alias name or the real model, and a cloud fallback receives the name the client sent (a provider's `default_model` still overrides). An alias that shares its name with a real fleet model always wins and is flagged as shadowing it. Aliases resolve one hop only, apply to both Ollama-native and OpenAI-compatible endpoints (never to model-management endpoints), take effect immediately without a restart, and change nothing for existing requests when none are configured. Local response bodies still report the real model name (a cloud fallback on an `/api/*` path reports the requested name), and `/api/tags` does not list aliases.
- **Keep-warm notices and runtime support flag on every node.** The Admin API node listing adds `warmupSupported` (keep-warm currently acts on Ollama nodes only) and `warmupWarnings` (per-model, non-fatal notices: skipped because the node is a replica worker or unresolved, or the resident copy's content digest differs from the fleet's reference copy of that model - the model is still kept warm in that case). Shown on the Warmup page per model, as a "Keep-warm notice" pill on GPU Nodes, and in `marbor nodes` as a `KEEP-WARM` column plus a warnings block. Runtimes other than Ollama show keep-warm as unsupported rather than silently doing nothing.

### Changed
- **Keep-warm is now replica-, health-, and catalog-aware across the fleet.** The periodic keep-warm cycle, warmup schedules, and predictive prewarming now agree on which nodes they may touch. A multi-host replica worker is never warmed directly: its keep-warm input on the Warmup page is locked with a pointer to the head, a warmup schedule aimed at a worker runs against its head (the schedule node picker labels this up front), and a leftover config stored on a worker can only be cleared or trimmed (Admin API `PUT /admin/nodes/{name}/warmup` rejects anything else with a message naming the head; the Warmup page adds a confirmed "Clear stale config" action). A node with an unresolved replica declaration is skipped and every warmup edit or schedule against it is rejected until the declaration is reconciled. Unhealthy nodes are skipped by both the keep-warm cycle and warmup schedules instead of being pinged anyway. Adding a model to a node's keep-warm list, or to a warmup schedule, is rejected unless the model is already pulled on that node (a bare `llama3` matches `llama3:latest`); disabling or trimming an existing list is always allowed, even when the node is unreachable. Saving a node's keep-warm config now triggers a full warmup cycle immediately.
- **Warmup evictions avoid models in use by live sticky sessions.** When a warmup needs to make room, it now evicts models no live sticky session is using first, and only evicts a session's model as a last resort. Tracking is soft (in-memory, most recent model per session, expires with the session) and adds no stored state.

### Fixed
- **Concurrent warmups on one node no longer double-evict.** The periodic keep-warm cycle, warmup schedules, and predictive prewarming could run their make-room-then-load step on the same node at the same time, each evicting against a stale view of VRAM and unloading more (or the same model twice) than needed. That step is now serialized per node.


## [0.21.0] - 2026-09-24

### Added
- **Multi-host tensor/pipeline-parallel replica scheduling: declare, see, and safely route around node groups.** A tensor-parallel or pipeline-parallel deployment sometimes spans more than one physical machine (one engine process across multiple hosts, not just multiple GPUs on one host) - registering each host as an ordinary independent node used to produce wrong routing (a worker host has no client-facing endpoint) and a misleading dashboard (N unrelated single-GPU rows instead of one deployment). Operators can now declare a multi-host replica by patching every member node with the same member list and head node name (Admin API `PATCH /admin/nodes/{name}` with `replica_peers`, CLI `nodes patch <name> --replica-members a,b --replica-head a`, or the GPU Nodes edit panel's new "Replica membership" section). Once every member's declaration agrees, the fleet resolves each node's role fresh on every routing decision: the head is the only member ever selected as a placement target, workers are automatically excluded from both normal routing and sticky-session revalidation (never orphaning an in-progress session on a now-worker node), and a worker/unresolved node stays fully visible in the node list - never hidden - with a clear "Worker (non-schedulable)" badge and a link to its head, or a distinct "Unresolved replica" badge when two nodes' declarations disagree (asymmetric members, conflicting head, or a reference to a node that doesn't exist) until an operator reconciles them. Declaring one side before the other is legal and expected - it produces the unresolved state on both nodes until the second PATCH lands, not an error on the first. Marbor never launches, configures, or manages the underlying deployment - the operator's own launch process starts it; this only makes routing and the dashboard reflect it correctly once it exists.
- **Routing explainability: scoring phase labels and candidate-exclusion reasons.** The per-request routing explanation (Admin API `GET /admin/v1/requests/{id}/explain`, CLI `requests explain`, and the Requests dashboard's expand panel) now groups each existing score term into one of three named phases - locality, predicted performance, reliability - and lists which nodes were removed before scoring even ran (unhealthy, draining, runtime mismatch, model not loaded, over capacity, insufficient GPU parallelism), capped at 20 with a total count past that cap. Purely additive and behavior-preserving: no change to the weighted-sum arithmetic, no new scoring input, no change to which node wins any request - decision-parity tests pin the pre-existing winner/score/component values unchanged.
- **Per-request prefill time, Ollama only, read-only visibility.** The Requests log (Admin API, CLI `requests list`, and dashboard) now shows how much of each Ollama-backed request's latency was prefill (prompt processing) versus generation, parsed from Ollama's response-embedded `prompt_eval_duration` field - the same response-parsing path that already surfaces generation time. Persisted durably in `request_log.prefill_ms` (additive migration, defaults to 0 for older rows) so the explain-by-id fallback keeps working after a request ages out of the in-memory ring. Deliberately narrower than first scoped: vLLM's queue/prefill histograms are fleet-aggregate Prometheus metrics, not attributable to a single request, so only Ollama gets a real number - every other runtime and cloud requests show `-`, never a fabricated estimate.
- **Engine-native telemetry: KV cache utilization and running/waiting request counts, reporting only.** The marbor agent now scrapes its co-located runtime's own `/metrics` endpoint (not the generic OpenAI-compatible surface) for the three highest-value scheduling signals a runtime can report about itself: how many requests it's currently running, how many are waiting, and what fraction of its KV cache is in use - the real capacity constraint, not GPU utilization percentage. These are additive optional fields on the frozen node-agent protocol (`RuntimeInfo.Engine`), surfaced on the node detail page. Reporting only - nothing in scoring, routing, or placement reads them yet; that is separate future work. Per-runtime coverage: vLLM reports all three (both the legacy and V1 engine, under different KV-cache metric names, both handled); TGI reports running/waiting but has no KV-cache-utilization equivalent to map, so that field stays honestly absent rather than estimated; Ollama, llama.cpp, and MLX are deliberately deferred (no native metrics source verified) and report nothing rather than a guess. A metric a runtime doesn't report is always absent (`-` in the UI), never a fabricated zero - a genuinely reported zero is preserved as a real zero.
- **Prefix-cache hit rate, vLLM only, read-only visibility.** The node detail page now shows a prefix-cache hit-rate percentage next to the engine-telemetry block above, derived from vLLM V1's own raw `prefix_cache_queries`/`prefix_cache_hits` counters (the engine deliberately exposes cumulative counters rather than a rate gauge, so the rate can be computed over any interval). The router computes it from the delta between two consecutive polls, not a cumulative-since-engine-start ratio, and returns unknown (`-`) rather than a fabricated 0% whenever there's no prior poll to diff against or the counters went backwards (an engine restart resets them). No other runtime exposes an equivalent counter pair today, so every other runtime shows `-`. Read-only - never a routing or scoring input, and not exposed in the CLI's node listing, matching the existing curated CLI node shape that already omits the other engine-telemetry fields above.

### Fixed
- **Metrics endpoint now binds loopback-only by default outside Docker, and survives a panic.** `/metrics` (Prometheus) has no authentication of its own, unlike the admin dashboard, but previously bound every interface by default with no crash recovery. A bare-metal/systemd install now defaults to `127.0.0.1:<port>`, widenable via a new `metrics.bind_address` setting (Settings page, `marbor settings set`, or config); inside Docker it still defaults to every interface, since the shipped `docker-compose.monitoring.yml` scrapes it container-to-container by service name and a loopback bind would be unreachable from the Prometheus sidecar. The handler is also wrapped in the same panic-recovery middleware as the proxy and admin servers.
- **Proxy API key lookup is now timing-safe.** The per-request key lookup indexed a map by the raw presented token, which is not a constant-time operation; it now looks up a SHA-256 digest of the token instead, so lookup timing no longer depends on how many leading bytes of a guessed token matched a real key.
- **`POST /admin/keys` no longer silently rotates an existing key on a duplicate name.** A repeated create request (double-click, retried request) used to overwrite the existing key's token and hand back a fresh plaintext secret as if nothing had changed. It now returns `409 Conflict` on a duplicate name; deliberate rotation goes through a new `POST /admin/keys/{name}/rotate` endpoint that preserves the key's existing limits/models/expiry.
- **Cloud-provider test and node TLS probe no longer echo raw connection errors, and a rejected key no longer looks like an admin-session error.** `handleTestCloudProvider`/`handleNodeTLSProbe` returned the raw dial/TLS error text (including host:port) to the client; they now log the real error server-side with a correlation ID and return a generic message, matching the pattern used elsewhere in the admin API. Separately, a cloud provider rejecting the API key under test now returns `502` instead of being remapped to `400` - the `400` mapping existed only to dodge a UI bug that force-logs the operator out on any `401`, but `502` never triggers that path either, so the real status is no longer hidden.
- **Confirm-discipline gaps closed: CLI `models delete` now gated, Users UI cited, routing strategy-set exemption documented.** `marbor models delete <node> <model>` is storage-destructive and irreversible but had no confirm gate, unlike every other CLI delete/remove/revoke/suspend command - it now requires `--yes` or an interactive TTY confirmation via the same `requireConfirm` helper as `nodes remove`/`key revoke`/`users delete`. A prior audit flagged Users.tsx suspend/delete as the only R10 destructive action without cited confirm-modal evidence; both already had action-specific confirm modals (`Users.tsx:697-732`, `734-769`) - the audit's grep had simply missed them, no code change needed. `routing strategy set` remains ungated, now with a one-line documented exemption: it is a reversible config change (set again to revert), not data loss or service disruption, same class as runtime start/stop/restart/drain.
- **Replica-aware GPU Nodes and Models pages, plus a slimmer Edit Node panel.** The head node of a multi-host replica now carries its own "Replica head - N workers" badge with forward links to each worker, mirroring the worker's existing "Part of replica" back-link (both directions reuse the same highlight-and-scroll mechanism, with the highlighted card's pill reading correctly on each end). The Models page no longer counts a model sharded across a replica as wasted duplicates: each resolved replica group counts as one schedulable instance with zero waste, a "Sharded TP=8 - N nodes, M instances" badge replaces the "+N dup" framing for those members, and each shard chip is marked in the "Resident on" list (unresolved nodes still count as separate instances until their declarations are reconciled on the GPU Nodes page). The Edit Node modal collapses capacity overrides, GPU topology (now including declared GPU indices), and multi-host replica into closed-by-default groups that auto-open only when the node already carries the relevant declaration, the parallelism width field stays disabled until a type is picked, and the help copy is cut to one line per field. UI-only, no Admin API or routing change; demo fixtures include a sharded model on the existing head/worker pair. Also fixes a dev-only auth dead end: the password gate's bare `/change-password` and `/skip-password-change` calls 404d under `npm run dev` (the proxy only forwarded `/admin` and `/login`), so neither Set Password nor Skip worked from the dev server while production was fine - both paths are now proxied. Follow-up polish pass: the Models page's sharded/waste badges are now a single shared component consumed by both the desktop table and the card (the table had drifted to a bare "+N dup" with no VRAM figure and an unstyled shard label, while the card kept the full chip - both now render identically), sharded uses a distinct blue/informational color instead of the same amber as actionable waste (this brand's primary color is itself gold, so pairing it with amber barely read as different), and a dead empty JSX block plus duplicated subtitle copy were removed. GPU Nodes' replica role pills (worker/head) get the same blue treatment instead of blending into transient amber status pills (draining, MLX hint), the replica link buttons gained a chevron affordance, and a head with more than 5 workers now collapses the rest into a "+N more" chip. A second follow-up fixes the sharded-model card's own summary framing: a sharded model's headline chip now reads "N instance(s), sharded across M nodes" in the informational blue treatment (instead of leading with "warm/total nodes warm" and relegating sharding to a small secondary badge), and the shard-member chips in "Resident on" are now visually grouped inside one bordered container so they read as one deployment rather than two independent-looking node chips - applied consistently to both the desktop table's Resident-on column and the card. The table's own Warm column had the same contradiction the card headline did - it kept showing raw "2/4" node counts for a sharded model right next to the sharded badge, which read as two different, disagreeing facts about the same model; it now shows the same "N instance(s), sharded across M nodes" framing as the card instead of the raw count for sharded models. Both the table and card keep a separate green warm/total ratio badge alongside the blue sharded chip (matching every other row/card's green ratio badge), scoped to the shard group's own declared membership (e.g. "2/2") rather than the fleet-wide node count or dropping the ratio entirely. A local code review of this whole branch surfaced two more real gaps, now fixed: a model that is both sharded AND separately warm as an independent duplicate outside its shard group (a genuinely mixed shape, e.g. TP=8 warm on its two replica nodes plus one unrelated stray warm copy) previously showed only the shard-scoped ratio and hid the fleet-wide picture entirely - the card and table now also show a "N/M nodes warm fleet-wide" fact in that specific case (only when duplication waste actually exists alongside sharding, never otherwise); and the table's shard status block was missing the `max-w` constraint its sibling badges wrapper already had, so a long sharded label could overflow the Warm column - added to match.

- **Model deletion is now replica-aware across UI, Admin API, and CLI - a node-local delete can no longer strand a multi-host replica.** `DELETE /admin/nodes/{name}/models/{model}` previously deleted from exactly the named node with no awareness of P63 replica topology - deleting from one member of a multi-host TP/PP deployment silently left the rest of the replica holding a now-inconsistent copy. The same endpoint (no new route, no new query parameter) now branches on the target's resolved `SchedulingRole`: a standalone node's delete is unchanged byte-for-byte; a worker or unresolved node is rejected with `409 Conflict`, naming the node's role and (when resolvable) its head, and pointing the operator at GPU Nodes to reconcile or delete via the head instead; a replica head's delete expands into a delete across every member of its resolved component (head included), validated for topology consistency before any agent is dispatched - full success is `200`, any member failure is `502` with a per-member breakdown (`{node, ok, error}`), never a silently-wrong `200 {"ok":false}`. The CLI (`marbor models delete`) and Web UI's Models page both consume this same server-side guard and surface its replica-aware result (member count/names on success, the server's own message verbatim on rejection) rather than duplicating the check client-side. The UI additionally fails closed on missing/stale topology data - a node the last fetch didn't return shows a "topology unavailable" notice instead of falling through to a destructive single-node delete.

- **Demo build: a tab/filter query string permanently broke every route-scoped data fetch on that page.** `currentAppPath()` returns the current route for guarding "only fetch when this page is actually showing" effects across every page. On the public GitHub Pages/forced-demo build (`HashRouter`), it read the query string as part of the route (e.g. `/models?view=catalog` never equalled `/models`), so any page loaded directly onto a tab or filter carrying a query param (Models' Catalog tab was the one found, but the same bug applied to any `?`-bearing URL on any page) silently skipped every effect gated on that comparison - node topology, in Models' case, never loaded, breaking the sharded/duplicate badges and blocking the delete flow with a false "topology unavailable" notice until the operator navigated away and back (which dropped the query string). Fixed at the source: `currentAppPath()` now strips the query string before comparing, matching what every one of its ~15 callers already assumed. Production (non-demo) builds were never affected - they read `window.location.pathname`, which never includes the query string.

### Docs
- README Configuration section now notes that nodes, keys, quotas, and routing config live in `marbor.db` (SQLite) and survive a reboot with no re-registration, pointing at the systemd unit and Compose restart policy in the Production Deployment Guide. The LiteLLM interop guide (`docs/integrations/litellm.md`, both directions) is now linked from the README integration guides list. The cost-savings card and the LiteLLM guide itself needed no changes - both were already shipped and prominent (dashboard hero card on Dashboard and Routing; complete two-direction doc matching the dedicated LiteLLM integration).
- README hero subheading now names the placement signals (model residency, GPU capacity, replica topology, workload pressure, runtime capabilities) instead of the endpoint-compatibility/cost-attribution line, which is already covered later in the README (Multi-Backend table, OpenAI-compatible endpoint section, cost tracking in the bold intro paragraph) - the hero's job is to state what makes placement decisions different, not restate coverage that already has its own section.

### Fixed
- **Config reload now writes a system-audit row and can't leave auth/cloud/node state half-applied.** `POST /admin/config/reload` (and the SIGHUP path) re-syncs live auth keys, cloud providers, and nodes from `marbor.db` in three stages, but a store read failure partway through a stage previously left earlier stages already mutated live while later ones kept their old values - a silent partial apply. The three store reads now all happen before any live state is touched, so a failure on any one of them leaves auth/clouds/nodes completely untouched, matching the "keep previous state" behavior the SIGHUP caller already logs. The reload endpoint also writes a `config_reload` row to the system audit log, closing the last mutating admin operation that had none.
- **Fresh-install backup: cryptic 403 replaced with plain-language copy, path proven end-to-end.** A zero-setup install (default `admin/admin`, `must_change_password:true`) got a raw `password_change_required` string as its backup error, and the underlying 401-redirect report had never actually been reproduced or tested against the real HTTP stack (all existing backup tests called the handler directly, bypassing `adminAuth`). Settings.tsx now shows "Change or skip your password first" for that error code instead of the raw string. A new integration test (`TestFreshInstallBackup_BlockedUntilPasswordStep`) drives `Server.Handler()` end-to-end: confirms the 403 never clears the session or resembles a 401, then proves backup succeeds with a real SQLite file after skipping the password step, even on a completely empty fleet. No allowlist change - backup stays gated behind the password step, only the error copy and test coverage changed.
- **Restore validation hardening: reject foreign/newer-schema backups pre-swap.** `store.ValidateBackupFile` previously only ran `PRAGMA quick_check`, so a well-formed but non-marbor SQLite file (or a genuine `marbor.db` backup from a newer, incompatible release) passed validation and only failed after the live database had already been swapped and the process restarted. It now also requires a `schema_version` row no greater than the running binary's `CurrentSchemaVersion`, rejecting both cases with a 422 before any swap - the same one function every entry point (backup upload, one-click restore, and the staged-copy re-validation right before the final rename) already funnels through, so the fix applies everywhere at once. No real historical backup fixture is affected.
- **R1 honesty audit (A.3): three estimate-presented-as-measurement gaps.** An R1 honesty audit of every dashboard page and CLI output found 3 confirmed gaps (out of an otherwise clean 15-page/55-CLI-file surface): (1) `Analytics.tsx`'s hero "Total Saved vs Cloud" showed the reference-rate savings figure with no `estimated` qualifier, unlike its sibling `Metrics.tsx` ("Estimated Savings") and `SavingsCard.tsx`'s explicit caption - retitled to "Estimated Saved vs Cloud"; (2) Model Advisor's per-variant `vram_est_mb` was rendered as bare "VRAM" instead of "est. VRAM" like the sibling GPUNodes fit table - now says "est. VRAM" with a tooltip disclosing the estimate basis; (3) `marbor savings`'s human-readable output printed `saved: $0.00`/`cloud spent: $0.00` when the server correctly returns JSON `null` for a deployment with local traffic but no parseable token data (a not-yet-recognized runtime) - `CloudSavings.SavedUSD`/`CloudSpentUSD` in `internal/cli/client.go` are now nullable `*float64` (fixing `--json` too, which shared the same non-pointer decode) and `runSavings` prints `-` instead of a fabricated `$0.00`.
- **CI: `marbor agent` collided with main.go's reserved "agent" removed-subcommand word, and `cloud providers update` had the same whole-object-PUT data-loss bug as L43.** `resolveCommand` special-cases the literal top-level word `agent` to `"agent-removed"` (the v0.19.2 binary split) before ever consulting the CLI registry, so the new P-A2-09b `marbor agent get/enable/disable/regenerate` group was unreachable and broke `TestResolveCommand_MatchesRegistry` in CI. Moved to `marbor node agent ...`, nested as a sibling of the existing `node control ...` group - consistent naming, no reserved-word collision. Also fixed while investigating: `cloud providers update` sent every optional field's Go zero-value when the operator omitted a flag (e.g. omitting `--enabled` silently disabled a previously-enabled provider) - now reads the current provider first and fills in any unset flag, same fix as L43; and `agent regenerate --json` reported a fabricated `"enabled": false` (the server's regenerate response has no `enabled` key, unlike enable's) - now set explicitly to `true` since the endpoint 404s on a not-yet-enabled agent. Orphaned `docs/man/marbor-agent.1` removed (agent no longer has its own top-level man page, matching `activity`/`spill`'s precedent).
- **Dashboard readability pass: wired the already-fetched Plus Jakarta Sans, tabular numerals on metric values, fixed garbled trend arrows, tighter page titles; website headlines rewritten for varied cadence.** `ui/index.html` already loaded Plus Jakarta Sans 400-700 but `--font-sans` still pointed at Inter, so the app silently rendered system UI - now it uses the intended face with zero new dependencies. `MetricCard` values render `tabular-nums` so live polls no longer jitter; the trend arrows were mojibake and now render as ▲/▼; every page `h1` shares `tracking-tight`. Website section headlines no longer all share one `X. *Y.*` template. No layout, API, or behavior change.
- **Design-language consistency: one pill shape, one warning amber, website focus/press/hover states.** Status pills were split between `rounded-full` (Sidebar badges, Requests Demo/Live chips) and the `Badge` component's `rounded-md` - all pills are now `rounded-md`; warning text was split between `amber-800` and `amber-700` in light mode (Warmup, Settings) - now uniformly `amber-700`. Deliberately untouched: the sprawling green/blue/emerald accents are per-runtime/per-status color coding (GPUNodes runtime map, activity kinds), not drift, so unifying them would destroy meaning. Website: headlines get `text-wrap: balance`, keyboard users get a visible gold `:focus-visible` ring, buttons get press feedback (`translateY(1px)`), and the three pain-point cards share the feature rows' hover language via a new `.pain-card` class instead of inline styles.
- **Dashboard first-poll skeletons for the fleet strip, capacity card, and metric row.** Live mode starts with empty nodes/summary, which flashed "No nodes registered yet" and all-zero metrics as fact for a beat before `/admin/v1` answered. Those three regions now render same-shell pulse placeholders until the first poll resolves (error path included); demo data is synchronous so demo mode never sees them. ModelAdvisor needed nothing - its search flow already carries pulse/loading states.
- **Dynamic first-poll skeletons: placeholder count follows the fleet's real size.** Fixed skeleton counts would lie about fleet shape, so both node grids (Dashboard panel, GPU Nodes page) now render as many placeholder cards as the last successful poll saw, persisted per browser in `marbor-last-node-count` (clamped 1..12, 3 on first-ever visit) via a shared `lib/nodeCount` helper. Same change gates GPU Nodes' false "No inference nodes found matching your search" empty state behind its own new first-poll flag. Counts only learn from non-empty polls (a transient empty payload must not shrink the next visit's grid), all placeholder roots are `aria-hidden`, and the GPU Nodes placeholder mirrors the real card's full section rhythm (actions, stat tiles, telemetry rows, VRAM) so resolving causes no height step. Demo mode never enters either branch.
- **Website small-screen overflow: kicker pill and console header now wrap.** The version kicker was an unwrappable `inline-flex` pill (~470px) and the scheduler-console header a nowrap flex row - both painted past narrow viewports. Below 640px the kicker becomes a wrapping block; the console header wraps. (Investigation note: this repo's headless captures floor the layout viewport at ~500px, so sub-500px proof needs a real device - anything narrower asserted from screenshots alone is suspect.)
- **GPU node status pills: severity-ordered, sentence case, icon well dropped.** A troubled node could stack seven all-caps flags at equal weight, burying e.g. a TLS mismatch under a draining notice. Pills are now collected with static severities and rendered worst-first (destructive, warning, muted) via one map, recased to the app-wide sentence voice with the draining pill's in-flight suffix preserved. The decorative server icon well left the card header (name + status dot carry identity). No behavior or threshold change.
- **Model Advisor initial-load skeletons (dynamic counts, spinner retired).** The slowest-loading page showed a generic spinner plus a false "No favourites yet" empty state and popping node pills while the catalog resolved. The browse grid now shimmers as many card placeholders as the last successful search returned (persisted per browser, clamped 1..9), the favourites empty state waits for load, and the node pills shimmer per last fleet size - reusing the same `lib/nodeCount` helper as the fleet grids. The spinner is gone from the initial path (search-in-flight keeps its honest pulse + stale grid).
- **Sentence-case UI copy, mono-stack call, Warmup toggle geometry, banner number-pair wrapping.** Every page/modal/panel header now uses sentence case (`GPU nodes`, `Request log`, `Add GPU node`, `Restore from backup` - acronyms VRAM/API/GPU/TLS/LiteLLM untouched); all-caps micro-labels, form labels, and table headers stay as-is as a deliberate second voice, matching Linear-style conventions. `--font-mono` drops the never-loaded JetBrains Mono for the system stack (tabular, offline-safe - loading it would buy nothing). The Warmup predictions toggle adopts the shared switch geometry (h-4 knob, muted off-state) instead of its bespoke chunkier variant. The budget banner's `($spent / $cap)` pair is `nowrap` so amounts never split mid-pair on narrow screens.
- **Universal gold focus ring on every toggle switch.** Only the Warmup predictions toggle showed a ring on focus (per-instance `focus:ring` classes); every `h-6 w-11` switch - Settings' shared `Toggle`, API Keys, Warmup - now gets the same ring from one unlayered rule in `index.css`, on click-focus and keyboard focus alike. The Warmup toggle's per-instance classes were removed as redundant.

### Added
- **CLI parity: `marbor settings get/set` (P85).** `GET/PUT /admin/settings` had full UI coverage in `Settings.tsx` but no CLI - the last whole-surface Law 6/DoD-9 gap the P0 audit found. Whole-object, not per-field flags: `config.Config` is large (Admin/Routing/Docker/Audit/Webhook/Savings/Warmup/ContextWindows/CloudProviders/...) and the server already merges a partial JSON payload onto the current config, so `marbor settings get > file.json`, edit, `marbor settings set --file file.json` mirrors how the Settings page itself round-trips the object - same mask-preserve behavior applies (echoing back a `"***"` secret leaves the real stored value untouched). New `Client.Settings`/`Client.UpdateSettings`, both raw-JSON passthrough (no mirrored client-side struct, same rationale as `SystemInfo`/`ModelCatalog`).

- **CLI parity: full `marbor nodes patch` field coverage + `marbor nodes tls-probe` (P429).** `nodes patch` previously exposed only 3 of `PATCH /admin/nodes/{name}`'s fields (parallelism, VRAM overrides); it now covers the rest - `--url`, `--runtime`, `--gpu-model`, `--vram-total-mb`, `--gpu-indices`, `--max-in-flight`, and `--tls-clear` (clearing a pin only - setting one stays exclusive to `nodes confirm-tls`, so a fingerprint is never pinned without the operator independently confirming it out of band). New `marbor nodes tls-probe <node>` reads the node's Marbor Agent certificate fingerprint (`POST /admin/nodes/{name}/tls-probe`) without pinning it, closing the gap where that read was UI-only. `Client.PatchNodeFieldsWithPtr` became the struct-based `Client.PatchNodeFields`/`NodePatchFields`; no new Admin API capability, no change to pin UX.

- **Optional bounded-drain grace period, plus persisted thermal/scheduled drains (P428).** `POST /admin/nodes/{name}/drain` now accepts an optional `grace_period_seconds` (absent means infinite, today's frozen behavior unchanged - no enforcement engine reads it yet, that's future work), round-tripped through `GET /admin/v1/nodes` (`drainedGraceSeconds`), the CLI (`marbor runtime drain --grace-period=N`), and a new field in the UI's Drain confirm modal. Separately: thermal auto-drain and scheduled drain/undrain now persist to `marbor.db` like a manual drain already did - previously only a manual drain survived a restart, so a restart could silently undrain a still-hot GPU or a maintenance window. A deleted node's drain row is left behind on delete (matches the existing `node_overrides` pattern, now covered by a regression test rather than being an undocumented gap).
- **UI parity: inline user edit on the Users page (P-A2-09d, the A.2 audit's one UI-side gap).** `PATCH /admin/v1/users/{id}` (`handlePatchUser`) has had a CLI command (`marbor users patch`) since P84 but no UI surface - `Users.tsx` only wired `listUsers/createUser/approveUser/suspendUser/deleteUser/resetUserPassword`. Adds a new `patchUser()` export in `ui/src/lib/api.ts` and an "Edit email/role" pencil action (desktop table + mobile card) opening a small modal (username read-only, email + role editable), matching `CreateUserModal`'s existing field styling. Demo mode already round-trips through the same in-memory `demoUserStore`, so no `mockData.ts` change was needed. Verified via the demo build at 1280px and 375px (no horizontal scroll, edit persists and reloads correctly) and `make build`/`tsc -b` clean (R6).
- **CLI parity: `marbor benchmark`, `marbor agent`, `marbor node control clear`, `marbor users change-password/skip-password-change` (P-A2-09).** The A.2 parity audit's final remaining group: all 4 benchmark endpoints, all 4 agent-lifecycle endpoints, `handleClearNodeControl`, and the two password-plumbing endpoints had full UI coverage but no CLI. New `marbor benchmark run|progress|cancel|runs` (`progress` reads a single snapshot off the SSE stream rather than following it live, same one-request-per-command rationale as `models pull-progress`), `marbor agent get|enable|disable|regenerate` (R8: `get` never returns a token; `enable`/`regenerate` print the one-line install command carrying a short-lived single-use enrollment code, matching the UI), `marbor node control clear` (completes the control trio alongside the existing `probe`/`accept`), and `marbor users change-password`/`skip-password-change` - both interactive-only with masked prompts (`termpw.go`'s `readPassword`, same discipline as `login`), and both rotate the saved session token exactly like the server rotates the session cookie, while preserving the session's existing username/role (a code-review self-check: `authenticatedClient`'s saved-session path never populates `client.Username`/`client.Role`, so a naive overwrite would have silently blanked them). `agent disable` and `node control clear` are R10-destructive (`--yes`/interactive confirm). 13 new `Client` methods, no new Admin API capability. This closes every CLI-side gap from the A2 audit (56 of 56 fully-absent handlers + both partials now have CLI commands); the one remaining item, `handlePatchUser`'s UI gap, is a separate `ui/src/lib/api.ts`/`Users.tsx` fix, not CLI work.
- **CLI parity: pull triage, predictive/warmup/system-info helpers, and pending-user count (P-A2-08b/c/d).** The A.2 parity audit found 10 remaining diagnostic endpoints with full UI coverage but no CLI: `GET /admin/pulls` (`marbor pulls`), `DELETE /admin/nodes/{name}/pull` (`marbor models cancel-pull`), plus a point-in-time `marbor models pull-progress` reading the same active-pulls list rather than following the separate SSE stream (operational-interfaces.md's one-request-per-command contract); `GET /admin/warmup` + `PUT /admin/warmup/predictive` + `POST /admin/warmup/ping` (`marbor warmup status/predictive set/ping`); `GET /admin/predictive/decisions` (`marbor predictive decisions` - the `Client.PredictiveDecisions` method already existed with no command wired to it); `GET /admin/system-info` (`marbor system-info`, raw JSON - large host/GPU summary); `POST /admin/config/reload` (`marbor config reload`); and `GET /admin/v1/users/pending-count` (`marbor users pending-count`). Nine new `Client` methods, no new Admin API capability.
- **CLI parity: `marbor backup` + `marbor analytics` + `marbor savings` + `marbor metrics summary` (P-A2-07).** The A.2 parity audit found all 8 endpoints (`GET/POST/PUT /admin/backup[/list|/restore|/upload]`, `GET /admin/analytics[/export]`, `GET /admin/metrics/savings`, `GET /admin/metrics/summary`) had full UI coverage (`Settings.tsx`/`Analytics.tsx`/`Dashboard.tsx`) but no CLI. New top-level `marbor backup now|list|restore|upload`, `marbor analytics show|export`, `marbor savings`, and `marbor metrics summary`, via eight new `Client` methods. `backup now`/`analytics export` stream the binary/CSV/JSON response straight to a local file; `backup upload` reads a local file and posts it as `multipart/form-data`. `backup restore` is R10-destructive (overwrites `mesh.db` and restarts marbor) and requires `--yes`/interactive confirmation, same pattern as `key revoke`. No new Admin API capability.
- **CLI parity: `marbor catalog` + `marbor models search/repo` (P-A2-06c).** The A.2 parity audit found `GET /admin/models/catalog`, `/admin/models/search`, and `/admin/models/repo` had full UI coverage (`ModelAdvisor.tsx`) but no CLI. All three are read-only Hugging Face/fleet-discovery endpoints with large, purely-informational nested response shapes, so (same rationale as P-A2-06b's `model-config`) the CLI prints the raw server JSON rather than a hand-mirrored type. Three new `Client` methods, no new Admin API capability.
- **CLI parity: `marbor model-config` + `marbor nodes fit` (P-A2-06b).** The A.2 parity audit found all 5 model-config endpoints (`GET/PUT/DELETE /admin/model-config`, `GET /admin/model-configs`, `GET /admin/model-config/capabilities`) plus `GET /admin/nodes/model-fit` had full UI coverage (`GPUNodes.tsx`'s `ModelConfigModal`/fit drawer) but no CLI. `store.ModelConfig` has ~40 optional per-runtime sampling/load-time fields, so `model-config set` deliberately takes a JSON body via `--from-json` (a literal string or `@path/to/file.json`) instead of ~40 individual flags - a documented scope choice (smaller surface, forward-compatible with new fields with no CLI change), not a silent omission. `model-config get/list` likewise print the raw JSON response rather than a hand-mirrored type. `marbor nodes fit` shows per-node VRAM fit analysis. Six new `Client` methods, no new Admin API capability.
- **CLI parity: `marbor favorites list/add/remove` (P-A2-06a).** The A.2 parity audit found `GET/POST/DELETE /admin/favorites` (per-user starred model list) had full UI coverage (`ModelAdvisor.tsx`) but no CLI. New top-level `marbor favorites` group via three new `Client` methods hitting the existing endpoints - no new Admin API capability.
- **CLI parity: `marbor cloud providers` + `marbor cloud budget-status` (P-A2-05).** The A.2 parity audit found all 7 cloud-provider/budget endpoints (`GET/POST/PUT/DELETE /admin/cloud/providers[/{name}[/reorder|test]]`, `GET /admin/cloud-budget-status`) had full UI coverage (`Settings.tsx`) but no CLI. New top-level `marbor cloud` group: `cloud providers list|add|update|delete|reorder|test` and `cloud budget-status`, via eight new `Client` methods hitting the existing endpoints. R8 handled explicitly: `list`'s `CloudProvider` type has no API-key field (the server never returns it either), `update` omitting `--api-key` sends `""` so the server preserves the currently stored key (matching `handleUpdateCloudProvider`'s own `"" `/`"***"` preservation logic), and `test`'s key is sent once for the one-shot probe and never persisted or echoed back. `providers delete` is R10-destructive (`--yes`/interactive-confirm, same pattern as `key revoke`).

### Fixed
- **Code review hardening of the P-A2-01..04/08a CLI parity commits.** `orch-review`-style multi-angle review of the five parity commits above found: (1) `schedules delete` and `routing rules remove` had no `--yes`/confirm gate, unlike every other destructive CLI command in the package (`nodes remove`, `key revoke`, `users suspend/delete`) - both now use the same `requireConfirm` pattern; (2) `nodes warmup set` was a whole-object PUT with no partial-update semantics, so toggling `--enabled` alone without repeating `--models` silently wiped a node's previously configured warmup model list - it now fetches the current config first and fills in any flag not explicitly passed, giving it patch-like behavior from the operator's side without adding a new Admin API capability (Law 6); (3) three near-duplicate inline comma-list parsers in `keys.go`/`users.go` were consolidated onto the new shared `parseCommaList` helper; (4) `GetPinned`/`SetPinned` no longer decode into two separately-declared identical anonymous structs; (5) `AuditQuery` and `SystemAuditFiltered` now share a `queryParams` helper instead of each hand-building `url.Values`; (6) `schedules patch`'s six hand-copied visited-flag-to-pointer blocks now use a new generic `ptrIf[T any]` helper. No behavior change to any already-shipped command besides the two confirm-gate additions and the warmup-set fix.

### Added
- **CLI parity: `marbor routing rules` + `marbor routing strategy` (P-A2-04).** The A.2 parity audit found all 6 routing endpoints (`GET/POST/DELETE/PUT /admin/routing/rules[/{id}[/toggle]]`, `GET/PUT /admin/routing/strategy`) had full UI coverage (`Routing.tsx`) but no CLI. New top-level `marbor routing` group: `routing rules list|add|remove|toggle` and `routing strategy get|set`, via six new `Client` methods hitting the existing endpoints - no new Admin API capability, strategy enum validation stays server-side.
- **CLI parity: `marbor schedules list/create/patch/delete` (P-A2-03).** The A.2 parity audit found `GET/POST/PATCH/DELETE /admin/schedules` (time-of-day warmup/unload/drain/undrain automation) had full UI coverage (`Warmup.tsx`) but no CLI. New top-level `marbor schedules` group with `list`, `create --action --node --at [--models --days --enabled]`, `patch <id>` (only visited flags change, same `RunCtx.IsSet` pattern as `key patch`), and `delete <id>`, via four new `Client` methods hitting the existing endpoints - no new Admin API capability, no new schedule validation (server remains authoritative).
- **CLI parity: `marbor nodes warmup/pinned/prewarm` (P-A2-02).** The A.2 parity audit found `GET/PUT /admin/nodes/{name}/warmup`, `GET/PUT /admin/nodes/{name}/pinned`, and `POST /admin/nodes/{name}/prewarm` had full UI coverage (`Warmup.tsx`/`GPUNodes.tsx`) but no CLI, despite being small, already-validated-server-side surfaces. `marbor nodes warmup get|set <node>`, `marbor nodes pinned get|set <node>`, and `marbor nodes prewarm set <node> --disabled true|false` call the five existing endpoints via five new `Client` methods - no new Admin API capability invented.
- **CLI parity: `marbor requests list/live` + `marbor audit` (P-A2-08a).** The A.2 parity audit found `GET /admin/requests`, `/admin/requests/live`, and `/admin/audit` had full UI coverage (`Dashboard.tsx`/`Requests.tsx`) but no CLI - the CLI's `requests` group only had `explain`, so headless request triage meant either the dashboard or a raw curl. `marbor requests list`/`marbor requests live` expose the two existing request-log endpoints (dashboard-formatted vs raw live-widget shape); a new top-level `marbor audit [--limit --model --key --node --status --cloud --since --until]` exposes the persisted, filterable request audit log - deliberately kept distinct from `marbor activity` (`GET /admin/system-audit`, which covers operator actions like drain/agent/runtime, not individual proxied requests). Three new `Client` methods (`Requests`, `LiveRequests`, `AuditQuery`), no new Admin API capability invented.
- **CLI parity: `marbor nodes add` / `marbor nodes remove` (P-A2-01).** The A.2 three-surface-parity audit (`.local/ai-audits/A2-three-surface-parity-audit-2026-09-04.md`) found `POST /admin/nodes` and `DELETE /admin/nodes/{name}` had full UI coverage (`GPUNodes.tsx` Add Node / Remove) but no CLI equivalent - the only way to change fleet membership headlessly was a direct SQLite edit or a raw API call. `marbor nodes add <name> <url> [--runtime x] [--gpu-model x] [--vram-total-mb n]` (upserts by name, reporting added vs updated) and `marbor nodes remove <node> [--yes]` (destructive per R10, same `--yes`/interactive-confirm pattern as `key revoke`) now call the existing endpoints via two new `Client` methods (`AddNode`, `DeleteNode`) - no new Admin API capability invented. Docs regenerated (`make man`): `docs/cli.md`, `docs/man/marbor-nodes.1`, README CLI table.
- **Per-model VRAM override wiring: Admin API, CLI, and UI (P411).** `config.NodeConfig.VRAMOverrides` (an operator-declared per-model VRAM size in MB, consumed as tier-3 fallback by `estimateModelSizeBytes` when a model has no real measurement and no disk size - mainly needed for vLLM/TGI/llama.cpp/MLX, which don't expose per-model size via their APIs) has existed on the consumption side since P410, but had zero way to actually set it - no endpoint, no flag, no UI field - so it was permanently empty in every real deployment. Wired end-to-end following the exact existing pattern for other per-node overrides (P397's `parallelism_type/width`, P75's `gpu_indices`): `PATCH /admin/v1/nodes/{name}` now accepts `vram_overrides` (whole-map replace, rejecting non-positive MB values with 400), persisted in a new additive `node_overrides.vram_overrides` JSON column (same JSON-blob convention as `gpu_indices`, malformed blob drops only that field per R1/R8), reapplied on boot via the existing `NodeOverrides()`/`PatchNode` path. CLI: `marbor nodes patch <node> --vram-override model=mb[,model2=mb2]` (empty clears). UI: a new "Per-Model VRAM Overrides" field in the GPU Nodes edit modal plus a small badge on the node card, demo parity in `mockData.ts`. **Key format:** plain model name, not quant-variant-qualified - P406 (DONE) added a content-identity `Digest` signal, but it's an opaque runtime-probed value for live warm/fungibility comparison, not a stable string an operator would type ahead of time, and P412 (real HF-repo-based resolution) hasn't landed; a correction for one quant variant applies to every variant sharing that model name until a real signal exists, documented explicitly in `config.go` as a deliberate, tracked deferral rather than an oversight.
- **Benchmark latency decomposition: real TPOT alongside TTFT, plus p95/p99 tail percentiles (P408).** The in-dashboard hardware benchmark and `marbor bench` CLI previously measured only Time-To-First-Token, aggregated to p50/min/max. Both now also derive genuine Time-Per-Output-Token from real decode-phase SSE chunk timestamps (`bench.MeasureChatLatency`, `LatencySample.TPOTMs`) - nil when a response was too short (fewer than 2 output tokens) to compute a rate, never a fabricated value (R1). `aggregateSamples` extends p50/min/max with p95/p99 (nearest-rank); `store.BenchmarkRun` gains `ColdP95Ms/ColdP99Ms/WarmP95Ms/WarmP99Ms` and nullable `ColdTPOTP50Ms/WarmTPOTP50Ms` (additive schema migration - both the `CREATE TABLE` and `ALTER TABLE` paths). New `marbor_request_tpot_seconds` Prometheus histogram plus two new Grafana panels (TTFT/TPOT percentiles); live-traffic TPOT observation is not yet wired into the streaming proxy hot path (parsing every chunk's JSON content in that path is a bigger, separately-risked change - R2) so that panel reads empty for live traffic until a follow-up lands. CLI (`marbor bench --json`), the admin API, and the dashboard's Benchmark page/demo data all show the new fields, rendering `-` for a not-computable TPOT. **Deliberately out of scope:** real per-request queue-time/prefill-time decomposition - no runtime's generic OpenAI-compatible response exposes that split honestly across Ollama/vLLM/TGI/llama.cpp/MLX, and estimating it from TTFT alone would be a synthetic number presented as a measurement; future work would need to probe a runtime's native endpoint or its own metrics for a real per-runtime signal.
- **Context-length-aware VRAM headroom for warm-loading and quantization-fallback decisions (P405).** The scheduler's eviction/headroom checks (`ensureHeadroom`, `ModelFitsAnyHealthyNode`) previously sized every model purely from its on-disk weight size, so a 1K-token request and a 32K-token request against the same model were treated as needing identical VRAM even though real KV-cache footprint scales with context length - and a GQA model (few KV heads) tied with an MHA model (KV heads == attention heads) of the same disk size, even though GQA's real per-token cache is genuinely smaller. New `internal/router/kvcache.go` (`EstimateContextAwareBytes`, `KVCacheBytesPerToken`) applies the same formula and runtime-family overhead constants Model Advisor's `computeContextFeasibility` already used for its own advisory display, now shared as the single source of truth (`internal/admin/catalog.go` calls the exported router version instead of keeping a private copy - router cannot import admin, since admin already imports router, so the reuse direction is admin-calls-router). `ensureHeadroom` (the proactive warm path, which has no live per-request context) sizes against the operator-declared `config.context_windows` entry for the model, pushed to the router via a new `SetContextWindows` (mirroring the existing `SetTimezone`/`SetLiteLLM` admin-push pattern, called at startup and on every settings update); `ModelFitsAnyHealthyNode`'s two call sites in the quantization-fallback path (`internal/proxy/proxy.go`) instead get a live per-request estimate from the same char-count/4 heuristic already used for the context-window admission check. Real GQA-vs-MHA architecture facts (KV heads vs attention heads) are cached per model (`Router.SetModelArchFacts`) opportunistically whenever an operator looks the model up in Model Advisor (GGUF `/api/show` metadata or an HF repo's `config.json`) - a model never looked up simply falls back to the non-GQA-aware linear estimate, never a fabricated guess (R1). **Deliberately out of scope for this item:** the live per-request routing/placement *score* (`computeNodeScore`/`scoreComponents` in `internal/router/placement.go`) is unchanged - this item only reaches the headroom/fit/eviction checks, not the `free_vram_headroom` weighted-scoring term itself, to keep blast radius contained to one atomic fix rather than reopening the weighted-placement formula alongside it. No API/schema/protocol change. Verify-before-build found the source audit's line-number references and "reuse via wrapping catalog's private function" plan stale/inverted against current code; the shape above is the corrected, smallest-change equivalent.
- **Deployment-aware GPU placement for tensor/pipeline/expert-parallel workloads (P397/P397b).** Previously the scheduler only checked VRAM, never GPU count - a model requiring e.g. tensor-parallel width 8 (`vllm --tensor-parallel-size 8`) could be routed to a 4-GPU node and crash the runtime, since `gpu_indices` was only used for Model Advisor sizing, never scheduling eligibility. Nodes can now declare `parallelism_type` (`tp`/`pp`/`ep`/`dp`) and `parallelism_width`; the router derives `required_gpus = max(len(gpu_indices), width)` and gates `routeInternal`/`RouteExcluding`/sticky session routing on `isGPUGroupSufficient` - fails open (never rejects fleet-wide) when GPU telemetry is unknown, and hard-422s an explicit `gpu_indices` too small for the declared width. The marbor agent now also auto-discovers a node's actual parallelism (`CUDA_VISIBLE_DEVICES`, `ps`, `systemctl show`, Docker container inspect) and surfaces it as a one-click "Adopt" in the GPU Nodes edit modal instead of requiring the operator to type GPU indices by hand; a mismatch between declared and detected values shows as a non-blocking warning. Also extends model-metadata parsing (architecture, `num_experts`, `quantization` including NVFP4) generically from HF `config.json`, with no model-specific branching - missing fields fall back to the existing estimated-fit path, never fabricated (R1). Admin API, CLI (`marbor nodes patch <node> --parallelism-type --parallelism-width`), and UI parity in the same change. SQLite migration is additive/nullable (`node_overrides.parallelism_type/parallelism_width`), no schema version bump, upgrade- and downgrade-safe.
- **CLI keys and users parity (P84): `marbor key list/create/revoke/patch` and `marbor users list/create/approve/suspend/reset-password/patch/delete`.** Closes the highest-value parity gap where `internal/cli` could toggle routing flags for a key it could not create/list/revoke and had zero users coverage. Every command is exactly one Admin API request via `internal/cli/client.go` (Law 6). Keys: `list` shows masked previews, `create --name` prints plaintext once, `revoke <name> --yes` and `users suspend/delete --yes` require a `--yes` or interactive TTY prompt (non-TTY aborts with a clear `--yes` hint, porting R10 to the CLI), `patch <name>` sends only visited flags via `RunCtx.IsSet`. Users: `create --user`/`list`/`approve <id> [--api-key-name --create-key --key-*]`/`suspend`/`reset-password`/`patch`/`delete`, with `--json` on every leaf and human tables showing `-` for null (R1). `marbor key set-local-only` / `set-allow-local-degradation` remain as convenience aliases. Docs regenerated: `docs/man/marbor-key.1`, new `docs/man/marbor-users.1`, `docs/cli.md`, README CLI table.
- **Requests empty state now offers an inline "Clear filters" action, matching Activity.** When a filtered search on `/requests` returns zero rows, the empty-state message (desktop table row and mobile card) now includes the same inline "Clear filters" link Activity already had, instead of leaving the operator to manually reset every field. The existing top-of-filter-bar "Clear all filters" button and the new inline link now share one `clearAllFilters` function.
- **Activity date-range filter UX rebuilt, and a shared clear-filter pattern extracted for both Activity and Requests.** The Range row's 6-pill toggle (`All time/1h/24h/7d/30d/Custom`) hid the From/To pickers behind a "Custom" click, jumping the layout and offering no way to see or clear a custom range without deselecting it first. Replaced with a `Quick range` select plus always-visible `From`/`To` `CustomDateTimePicker` fields, mirroring `ui/src/pages/Requests.tsx`'s existing Quick range/From/Until layout - editing either date field switches the select into a disabled "custom" state (matching Requests' `sincePreset`/`sinceInput` exclusivity), and clearing both fields via their own inline "x" falls back to "All time" automatically. Also fixed the same latent issue in Requests.tsx itself while at it: its Quick range select could show a stale preset label (e.g. "Any time") while a custom range was actually active and the select disabled, because `sincePreset` was force-reset to `'all'` instead of a distinct marker - `SincePreset` gained a `'custom'` value so the select correctly falls back to its placeholder text instead. `ClearableInput` and `FilterField` (previously private to `Requests.tsx`) are now shared components at `ui/src/components/FilterField.tsx`, with `ClearableInput` gaining an optional leading `icon` prop - Activity's Target/Source IP/Search inputs now get the same inline "x" clear affordance Requests' filters already had, instead of requiring the user to manually backspace the text.
- **Activity enterprise filters: pagination-undercount fix, unified single view, contains-match username.** `QuerySystemAuditLogFiltered` (`internal/store/sqlite.go`) previously applied the SQL `LIMIT` before the Go-side kind reconciliation check, so a kind-filtered page could silently return fewer rows than requested even though more matching rows existed further back - fixed with a bounded overfetch-and-loop (`maxSystemAuditScan = 5000`, `queryLimit = fetchLimit * 4` when kind reconciliation is active), covered by a new regression test (`TestQuerySystemAuditLogFilteredKindPagingReturnsFullPage`). The `Username` filter now matches anywhere in the username (contains), consistent with `Target`/`SourceIP`, instead of prefix-only - covered by `TestQuerySystemAuditLogFilteredUsernameContains`. `ui/src/pages/Activity.tsx` retires the last `?view=audit` remnant (`useSearchParams`/`isAuditView`) that still gated the Source IP filter input and column behind a query param - Source IP is now always visible, removing the last split-view duplication the operator had to reconcile across two URLs of the same table. `/system-audit` now redirects to plain `/activity` (was `/activity?view=audit`, a stale target once the param stopped being read).
- **Activity and System Audit now share one enterprise-filtered timeline (dedup).** `GET /admin/system-audit` is the single canonical table, now with server-side filters `?from=RFC3339&to=RFC3339&kind=drain|agent|runtime|node|warmup|schedule|predictive|config|all&action=<exact>&user=<prefix>&target=<contains>&source_ip=<contains>&limit=1..200&before=RFC3339` combining with AND, paginated via `before` cursor on `idx_system_audit_log_ts`, kind mapped server-side via the same `toActivityKind` bucket as the UI (`ui/src/lib/activityKind.ts:7`, `internal/store/sqlite.go:systemAuditKind`), predictive returns 0 rows from this table. No new table, no secret field, times are real UTC RFC3339 or `-`. Store adds `QuerySystemAuditLogFiltered` and fixes `AppendSystemAuditLog` to store `UTC().Format(RFC3339)` (P142). Frontend merges to one canonical component: `ui/src/pages/Activity.tsx` is the fleet timeline, `/system-audit` redirects to `/activity?view=audit` for bookmarks, single nav entry Activity at top (Audit Trail alias hidden). Default fleet view hides config, shows predictive strip distinct above the timeline; audit view shows Kind=all plus Action and Operator dropdowns and Source IP column. Enterprise filter bar uses `CustomDateTimePicker` with presets Last 1h/24h/7d/30d/Custom plus Kind/Action/Operator/Target/Source IP inputs, server-side via query params with 250ms debounce on text fields, pagination via Load more / before cursor showing "showing N of M". Keeps 30s auto-refresh and 375px table+cards+modal pattern. CLI `marbor activity` now supports `--from`, `--to`, `--before`, `--kind`, `--action`, `--user`, `--target`, `--source_ip`, `--limit`, `--json` via `SystemAuditFiltered` and the same kind map. Demo `fetchSystemAuditFiltered` returns 25 entries spanning minutes/hours/days with schedule entries so date presets and kind=schedule demo meaningfully, `mockData.ts` keeps 14 adequate entries covering each kind with varied timestamps. No new route, no new datastore.
- **Scheduled warmup/drain/undrain firings now appear on the Activity timeline with actor system.** `internal/router/scheduler.go:fireSchedule` now writes `scheduled_warmup`, `scheduled_unload`, `scheduled_drain`, `scheduled_undrain` rows via `store.AppendSystemAuditLog` directly (router has no request), with `status=ok|error` and error aggregate, so a schedule firing is compliance-auditable without a separate table. `GET /admin/schedules` CRUD already audited as `create_schedule`/`patch_schedule`/`delete_schedule` via `logSystemChange`. `ui/src/lib/activityKind.ts` and `internal/store/sqlite.go:kindActionMap` now include kind `schedule` (teal badge) so server and client agree. Predictive warmup decisions remain via `GET /admin/predictive/decisions` distinct section, not interleaved, to avoid clock skew. R6 `make build` clean, 375px Activity check pass.
- **Operations timeline page: unified fleet activity feed (P389).** Reuses the persisted `GET /admin/system-audit` timeline as the single source of truth and merges `GET /admin/predictive/decisions` client-side, no new aggregation endpoint and no new SQLite table. Every fleet change shows what, when, and who with real parsed audit values or `-`, never estimates. Kind taxonomy locked and tested: drain (`drain_node`, `undrain_node`, `set_node_prewarm`), agent (`enable|disable|regenerate|enroll_marbor_agent`), runtime (`runtime_start|stop|restart`, `accept|clear_node_control`), node (`add|update|remove|patch_node`), warmup (`unload_model`, `set_node_warmup`, `set_pinned_models`, `pull_model` variants, `delete_model`), predictive (system-generated, distinct section not interleaved), and config fallback. UI: new page at `/activity` placed second in the sidebar as `Dashboard -> Activity -> GPU Nodes` to tell the operations-first story, with kind filter and text search both filtering client-side for v1 (limit 100, kind `all` default, predictive shown as a distinct badge/section above the timeline to avoid clock-skew interleaving), four stat cards, desktop table plus mobile cards, and a detail modal sharing the SystemAudit visual language. Sidebar ordering and the `Activity` icon are the only `Sidebar.tsx` change. CLI: new `marbor activity [--limit 100] [--kind drain|agent|runtime|node|warmup|predictive|config|all] [--json]` fetching the same two admin endpoints and applying the same kind map client-side before printing a predictive section followed by a tab-aligned activity table. Demo parity: `ui/src/lib/api.ts` `fetchSystemAudit` DEMO now returns 16 plausible static entries spanning drain, unload, runtime restart, node join/leave, agent enable/disable, warmup pin, pull, token regeneration, plus the existing three predictive decisions, so `/demo/` exercises the feed honestly labeled as demo data. No new secret-bearing field, no new route, no new datastore.
- **Sidebar collapse/expand toggle (desktop only).** The left nav now has a desktop-only collapse button (hidden below 768px where the existing slide-in drawer applies) that shrinks the sidebar to an icon-only 64px strip (`w-16`) and expands it back to the full 256px (`w-64`). Main content reflows between `md:ml-16` and `md:ml-64` with a 200ms ease transition and no layout overlap; the mobile drawer overlay and its `h-dvh` fix are unchanged. Collapse state is persisted in `localStorage` (`marbor-sidebar-collapsed`) and read synchronously on mount with no flash, and works in `/demo/` (forcedDemo) without touching demo data. The toggle is a keyboard-accessible button with `aria-label` and `aria-expanded` and a visible focus ring. UI-only: `ui/src/components/Sidebar.tsx`, `ui/src/App.tsx`, new `ui/src/hooks/useSidebarCollapsed.ts`; no Admin API, CLI, or store change. Verified at 375px (no horizontal scroll) and 1280px (no reflow) on Dashboard, GPU Nodes, and Settings.
- **The dashboard now opens with a Fleet Health strip answering "is my fleet OK" before anything else.** One line summarizes the fleet as `X/Y healthy · N draining · M down` (plus a degraded count whenever one exists, so the numbers always reconcile), followed by alert badges for the things that need attention right now: which nodes are down, how many marbor agents have gone stale, and how many nodes are in VRAM pressure (>=90% of their real measured usage). Every figure is aggregated live from the same `/admin/v1/nodes` data the page already polls - counts of actual states, never estimates (R1) - with an honest empty state when no nodes are registered. Backing the stale-agent badge, `GET /admin/v1/nodes` gains an additive `agentStale` field (absent unless true): it is set only when an enabled marbor agent IS configured for a node's host but its consecutive poll failures crossed the health failure threshold - i.e. an enrolled agent went dark - and deliberately never for hosts that simply run without agents by choice. The distinction is made in the router where that hysteresis already lives (`NodeState.AgentStale`, cleared on recovery, on success, and when an agent is disabled), so "your agent stopped answering" and "this node never had an agent" are finally distinguishable from the dashboard.
- **A Fleet Capacity card sits under the health strip answering pain #5: do I need another GPU, or am I placing badly?** It shows cluster-wide VRAM used/free summed only from real per-node readings (used from each node's own live measurement, capacity from nodes that report a total, with a visible `N/M nodes reporting capacity` caption whenever some nodes' totals are unknown - never padded to a fake cluster number), plus warm-model counts (unique models vs total resident instances across the fleet) and a duplicated-models figure counting models warm on more than one node at once - the placement waste another model's VRAM could be using. Unknown figures render as `-`, never guesses (R1).
- **Fleet-first Models page (P390): every per-model question answerable at a glance.** `GET /admin/models` now enriches the live fleet aggregation with `warm`/`vram_bytes`/`runtime` per node (from `router.NodeState` `LoadedModels[].SizeVRAM` + `Runtime`, no new DB or RPC) plus `total_vram_bytes` (sum of warm-copy VRAM, `0` omitted) and `drift_details` (short `a1b2c3 vs d4e5f6` hex of distinct digests, sorted, empty when consistent). UI `ui/src/pages/Models.tsx` flips from catalog browsing to fleet intelligence: four summary cards at the top (distinct warm models, total warm copies + total warm VRAM, drifted models, duplication waste copies + duplicated VRAM), filters by name search plus drifted-only and warm-only toggles (client-side, instant), and per-model rows showing `warm_count/total_nodes`, per-copy VRAM + total warm VRAM, drift badge (`consistent` vs `drift a1b2c3 vs d4e5f6` with short digest chips + `StatusDot` per node linking to `/gpu-nodes` for mutations - read-first, no drain/prewarm controls cloned). Desktop table plus mobile stacked cards (`hidden md:block` / `md:hidden`, no horizontal scroll at 375px), Fleet tab default with Catalog as secondary tab reusing the same fleet data (link to Model Advisor for curated fit, pull modal stays). CLI parity: `marbor models` table now shows `TOTAL VRAM` and `DRIFT` (`-` when unknown) plus new `marbor models fleet [--drifted-only] [--json]` on the same `GET /admin/models` endpoint (Law 6 - no new route), verified via `internal/cli/testdata/help/models.golden`, `docs/cli.md`, and `docs/man/marbor-models.1` regenerated. Demo parity: `ui/src/lib/mockData.ts` `mockModelCatalog` now has 8 models with diversified fleet state - two drifted (`llama3.3:8b` `3b7e0a vs 9f8a1c` and `qwen2.5-coder:14b` `a1b2c3 vs d4e5f6`), three duplicated/waste cases (`mistral:7b` on `ollama`+`vllm`, `gemma2:9b` on 3 nodes including `tgi`, `llama3.3:8b` already), mixed runtimes (`ollama`, `vllm`, `tgi`, `llamacpp`), per-node `warm`/`vram_bytes`/`runtime` and `total_vram_bytes`/`drift_details` populated. `make build` (R6) embeds fresh `web/dist`, `tsc -b` + `go vet` + `gofmt` clean.

### Fixed
- **B.1 full-repository code review: six hardening fixes across store/proxy/auth/admin/router/marboragent.** The first-ever whole-repo review (`.local/ai-audits/B1-full-repo-code-review-2026-09-04.md`) found 47 findings across every package; the 6 NOW-priority ones (a 7th, a claimed missing TLS host-group check, was verified INVALID against current code) are fixed here, each as its own atomic commit: (1) `migrateMarborAgentRekeyByHost`'s legacy-name-to-shared-host-key migration omitted `scope`/`scheme` from its column list, silently escalating a `readonly`-scoped or downgrading an `https`-scheme marbor agent to the defaults (`admin`/`http`) on upgrade from v0.19.x - irreversible once the old-keyed row was deleted by the same migration; (2) `statusRecorder`'s no-newline response-tail buffer (used for single-JSON-document responses like `/v1/embeddings`) grew to 32 MiB per request, letting a burst of concurrent embeddings requests OOM the control plane - now capped at 1 MiB, with the tail marked truncated so token count correctly reports unknown (`-1`) rather than a fake `0`; (3) `auth.AddKey` left a rotated key's old token valid in memory until the next full reload or restart when the new key was added under the same name (the path the admin "add key" API takes for rotation) - now revokes the old token immediately; (4) 30 of 32 Admin API JSON-decoding handlers (including the unauthenticated login and marbor-agent-enrollment endpoints) had no request body size limit, allowing a multi-gigabyte body to be allocated before any validation - all now capped at 1 MiB, matching the one handler that already had this guard; (5) the marbor agent's foreground `--cert`/`--key` flags silently fell back to plaintext HTTP (log-only warning) when exactly one was set instead of both, letting the bearer token traverse the network unencrypted on a typo - now fails closed; (6) `Router.RemoveNode` left `warmupSuppressed` and `showCache` entries for the removed node in place indefinitely (a slow leak that compounds under fleet churn) - now purged alongside the state it already swept.

- **An MLX node with `runtime: auto` was permanently misdetected as llamacpp and stayed silently unhealthy forever, with no explanation (P409).** MLX's (`mlx_lm.server`) `/v1/models` response is byte-for-byte identical in shape to llama.cpp's - no `owned_by` or other field to distinguish them - so `runtime: auto` always resolves an MLX node to `"llamacpp"`, and its health check then fails forever because `mlx_lm.server` has no `/health` route llama.cpp's probe requires. There is genuinely no signature to detect MLX by from `/v1/models` alone (confirmed against `internal/runtime/detect.go`'s own existing comment), so this is a diagnostic fix, not a detection fix: `pollNode` (`internal/router/health.go`) now sets a new `NodeState.RuntimeMismatchHint` whenever a node currently probed as `llamacpp` fails specifically because `/health` returned 404 (the real, observed signature - never a guess that the node IS MLX, per R1), cleared on the next successful probe or whenever the operator explicitly changes the node's runtime (`PatchNode`). Surfaced through the Admin API (`nodeResp.runtimeMismatchHint`, `omitempty`) and a new amber "POSSIBLE MLX NODE" badge on the GPU Nodes page (mirrors the existing TLS MISMATCH badge pattern) suggesting the operator set `runtime: mlx` manually - manual MLX already worked correctly before this fix and is unaffected. CLI parity is automatic: `marbor nodes --json` already serializes the full node response, so the new field needs no CLI code change (consistent with existing diagnostic fields like `drainedReason`, which also aren't columns in the human-readable table). Demo parity: `gpu-node-06` in `mockData.ts` now demonstrates the hint. Regression tests: `TestDetectRuntime_MLXMisdetectedAsLlamaCpp` (`internal/runtime/detect_test.go`, documents the known limitation), `TestPollNode_SetsRuntimeMismatchHint_LlamaCppHealth404` / `TestPollNode_NoRuntimeMismatchHint_GenericFailure` / `TestPollNode_ClearsRuntimeMismatchHint_OnSuccess` (`internal/router/health_test.go`). Blast radius: `internal/router/router.go`, `internal/router/health.go`, `internal/admin/admin.go`, `ui/src/types/index.ts`, `ui/src/pages/GPUNodes.tsx`, `ui/src/lib/mockData.ts` - no protocol version bump, no SQLite schema change.
- **Placement scoring's connection-based load signal treated every connection as equal work, letting a few prefill-heavy requests undercount as "less busy" than many decode-light ones (P404).** `scoreComponents`' `inverse_queue_depth` term (and the equivalent branch for statically-VRAM-reserving runtimes like vLLM) was `1/(1+ActiveConns)` - a flat count with no notion of how much actual compute each connection represents, so a node serving 2 heavy prefill requests could score better than a node serving 8 light decode continuations even though the first is the more loaded node in practice (Orca-style continuous batching: connections are not fungible slots). Both terms now weight `ActiveConns` by the node's own recent real observed time-to-first-byte (`NodeState.RecentTTFT`, capped at the last 10 requests, fed by a new `Router.RecordTTFT` call from the proxy's existing TTFT measurement) via a new `effectiveLoad` helper - a node whose connections have recently been slow to first byte (compute-bound, prefill-heavy) is now scored as more loaded per-connection than one whose connections return bytes quickly (decode-light), while a node with no TTFT history yet falls back to the unweighted raw count unchanged (no regression for cold/idle nodes). No fabricated data (R1): the weighting signal is only ever a real measured duration, never estimated. Covered by a new regression test (`TestPlacementScoring_TTFTWeightedLoad`); `internal/router/router.go`, `internal/router/placement.go`, `internal/proxy/proxy.go` - no API/schema/protocol change.
- **Cold-start VRAM reservation failed open when a model's size was genuinely unknown, letting a burst of concurrent cold starts double-book the same node (P402).** `reserveColdStartBytes` only ever consults zero-I/O size data on the request-routing hot path (R2), and previously reserved 0 bytes when a first-seen model had no `lastKnownVRAM` and no `vram_overrides` - so two concurrent requests for never-seen models could both see the same "fully free" headroom snapshot and tiebreak onto the same node, which could then OOM once both loads actually landed. `reserveColdStartBytes` now falls back to a conservative `unknownModelReserveBytes` (2 GiB) placeholder reservation - a scheduling guard, never surfaced as a VRAM measurement (R1) - so the node's `free_vram_headroom` score term is discounted for a concurrent request the same way an already-known model's reservation always was. Covered by a new regression test (`TestRoute_ColdStartReservationPreventsDoubleBooking_UnknownSize`); one file (`internal/router/eviction.go`), no API/schema/protocol change.
- **`ModelFitsAnyHealthyNode` failed open on a fleet with no real VRAM capacity signal, letting a model that could OOM a node be treated as "fits" (P403).** A remote node with no marbor-agent and no manually declared `vram_total_mb` never gets a real `VRAMTotalMB` (`health.go`'s `api` source only ever populates real used-VRAM, never a total) - but the fit check folded "capacity unknown" together with "model size unknown" into one fail-open boolean, so a fleet with zero capacity signal anywhere still reported a 70B model as fitting. Now tracks whether ANY healthy node has a known capacity separately from whether the model's size is known: no capacity signal fleet-wide returns `false` ("not confirmed to fit"), while the existing "capacity known, size unknown" case is unchanged and still fails open (R1 - no data to disprove fit). Covered by a new regression test (`TestModelFitsAnyHealthyNode_NoCapacityKnownFailsClosed`); one file (`internal/router/eviction.go`), no API/schema/protocol change.
- **`QuerySystemAuditLogFiltered` could silently skip a row sharing its timestamp second with the last row of an internal overfetch page.** `ts` is stored via `time.RFC3339` (1-second resolution), and the kind-filter overfetch loop (see the pagination-undercount fix below) advanced its cursor with `ts < lastSeenTs` alone - a same-second row not returned on one internal iteration was then excluded from every later iteration too, since it is `== cursor`, not `< cursor`. The cursor now also carries the row's autoincrement `id` as a tiebreaker (`ts < ? OR (ts = ? AND id < ?)`, ordered `ts DESC, id DESC`), covered by a new regression test (`TestQuerySystemAuditLogFilteredSameSecondTieWithinKindPaging`). Found via code review of the enterprise filter work above.
- **The GitHub Pages demo rendered a white screen when deployed under any path other than `/marbor/demo/`.** The demo build hardcoded an absolute asset base (`/marbor/demo/`) in `ui/vite.config.ts`, and `App.tsx` hardcoded the same path as the router basename and login-reset target - so deploying the identical artifact under a different first path segment (e.g. `/marbor1/demo/`) loaded the HTML but 404'd every JS/CSS request against the other deployment's URLs, leaving an empty page. The demo build now uses a relative asset base (`./`), the router basename is derived at runtime from where the app actually loaded, the sidebar's "Back to website" link goes to `../`, and the landing page's demo links are relative (`demo/`) - one artifact now works under any mount point, and each deployment's landing links to its own demo.
- **Session tokens (admin and user) were stored in plaintext in SQLite (P93).** `admin_sessions` and `user_sessions` now store a SHA-256 hash of the token, hashed at every insert and lookup inside `sqlite.go`; the lookup comparison itself was already correct. No schema change. Existing sessions in a running deployment will need one re-login after upgrading, since their stored plaintext token no longer matches a hashed lookup.
- **Settings page toggle switches could render with the knob detached from the track on desktop.** The shared `Toggle` (and every inline switch on the Settings page) lacked `shrink-0`; in rows with long labels (Trust Proxy Headers was the visible case), flexbox squeezed the 44px track below its set width while the knob kept its fixed translate offset, leaving it outside the track. All switches now carry `shrink-0`. Pre-existing since the component's introduction (4181a) - not a regression from the mobile sweep, but surfaced by it.
- **The mobile navigation drawer used `h-screen` (100vh), which on phone browsers includes the area behind the URL bar - so the drawer footer (theme toggle, logout) rendered below the visible screen with body scroll locked while the drawer was open, making light mode effectively unreachable on mobile.** The drawer now uses `100dvh` (dynamic viewport height, with a `h-screen` fallback for browsers without dvh support), so it always exactly fits the visible viewport. Found during the mobile sweep retest below; desktop and headless-browser testing never surfaced it because headless viewports have no dynamic browser chrome.
- **Full-app mobile (375px) and light-mode UI sweep: fixed every issue found across all 13 admin pages.** Horizontal scroll on the Dashboard at 375px (the GPU Nodes Status panel header legend did not wrap; the page was 19px wider than a 375px viewport in both themes - the legend now wraps). Contrast failures, all verified against WCAG AA for small text: light-mode `--success`/`--warning` tokens darkened (4.2:1 -> ~5-6:1 for 10-12px status text such as "Demo Mode", "active", "Pending"); dark-mode `text-destructive` badges/brightened via `dark:text-red-400` where red text sat on dark surfaces at 3.7:1 (Badge destructive variant, GPU Nodes TLS MISMATCH / WARMUP FAILED / UNLOAD FAILED badges, Analytics "blocked" cells); raw Tailwind palette colors used as text without a light-mode-safe step replaced app-wide (`text-amber-500` -> `text-amber-700 dark:text-amber-400`, plus isolated `text-red-500`/`text-orange-400`/`text-green-500`/`text-blue-500` outliers); the Analytics chart Legend now renders labels in theme-aware muted foreground instead of inheriting the series stroke color (`#f59e0b` amber was ~2:1 on light paper). Tap targets: sidebar drawer links and footer actions raised to a 44px minimum height, hamburger/close buttons to 44x44; on viewports under 768px icon-only `p-0.5`-to-`p-2` buttons get a 40px minimum box, toggle switches and inline chip icon buttons get expanded hit areas via CSS without changing their visual size. Embedded `web/dist` rebuilt.

### Changed
- **Dashboard headline restructured around fleet state instead of traffic.** The seven-card traffic row (Active Requests, Queued, Avg Latency, Tokens/min, Warm Hit Ratio, Cold Starts, Saved vs Cloud) no longer owns the top of the page: the six traffic metrics are now a compact secondary row below the new fleet surfaces, while Saved-vs-Cloud got a promotion rather than a demotion - it sits in its own green block beside the Fleet Capacity card, above the fold, as the fleet-ROI answer ("is my fleet healthy, is capacity fine, and what is it saving me" in one screen). The same shared card also appears on the Routing page next to the strategy picker that drives the local/cloud split, so the two surfaces can never drift. Since every locally served token counts as avoided spend at the configured reference rate, even a pure local fleet shows real value; the block is explicitly captioned "estimated" (parsed token counts priced at the reference rate, not a billed amount) so the more prominent figure stays R1-honest, and it still shows `-` until real token counts exist. The static architecture-flow diagram was removed from the dashboard route rather than relocated: Settings is organized as configuration sections with no natural home for it. Demo fixtures were extended so `/demo/` exercises every new surface honestly labeled as demo data: a down node whose enrolled agent went dark (stale badge), a node at ~95% VRAM (pressure badge), and a model kept warm on two nodes simultaneously (duplicated count).
- **Project renamed from Ollama Mesh / ollama-mesh to Marbor / marbor (hard cutover).** No behavior change - name change only. Binary entrypoints are now `marbor` (control plane + CLI) and `marbor-agent` (marbor agent); the default database is `marbor.db`; the config/data directory is `~/.marbor` (and `%APPDATA%\marbor` on Windows); all environment variables use the `MARBOR_*` prefix (`MARBOR_DB_PATH`, `MARBOR_BACKUP_DIR`, `MARBOR_AGENT_SECRET`, `MARBOR_ENROLL`, `MARBOR_SERVER`); Prometheus metrics are `marbor_*`; HTTP headers are `X-Marbor-Version` / `X-Marbor-Model-Fallback` / `X-Marbor-Signature`; the auth realm is `realm="marbor"`; OS service identities are `com.marbor.agent` (launchd), `marbor-agent.service` (systemd), and `marbor agent` (Windows). The Go module path is now `github.com/Anirudhx7/marbor`, the GitHub repository is `Anirudhx7/marbor`, and container images are `ghcr.io/anirudhx7/marbor`.

## [0.19.3] - 2026-08-20

### Added
- **Model Advisor now gives context-length feasibility advice, not just weight-only VRAM fit (P71).** Previously the only context-aware calculation anywhere in marbor was a flat linear guess (`sizeMB*1.10/1.20 + ctxLen*0.15/0.20`) with no idea of a model's actual maximum context or how much of the estimate was KV cache versus weights. The Hugging Face repo-detail flow (`GET /admin/models/repo`, the panel behind Model Advisor's "Target Context Length" slider) now sources real transformer architecture facts where they exist - Ollama's own `/api/show` endpoint for a model already downloaded to an Ollama node (never called by marbor before this), or a Hugging Face repo's `config.json` for vLLM/TGI/MLX safetensors repos (fetchable before any node downloads the model) - and runs a real per-token KV-cache formula (`2 x layers x kv_heads x head_dim x 2 bytes`) instead of the linear guess whenever those facts are available. Each variant's response now carries a `context_feasibility` object with an explicit `confidence: "derived" | "estimated"` label, the model's declared maximum context when known, which resource (model weights or KV cache) is the limiting factor, and - only ever on the "derived" path - a suggested lower context length that would fit. The linear formula is kept unchanged as the fallback for every case with no real architecture facts (llama.cpp GGUF repos, any repo with no `config.json`, not-yet-downloaded models), and is always labeled "estimated," never presented as more certain than it is. Advisory only: no runtime configuration is changed automatically, and a runtime's own launched context ceiling (e.g. vLLM's `--max-model-len`) remains an explicitly-stated unknown, since marbor has no way to discover it. Scoped entirely to the actual Model Advisor UI/API surface (`handleModelRepo`, `ModelAdvisor.tsx`) - the curated static catalog and the unrelated GPU Nodes fit table (`handleModelFit`, which has no context-length input) are unchanged.

### Fixed
- **Enabling "Use HTTPS" for a node's marbor agent on the GPU Nodes page silently switched that node's inference runtime endpoint (Ollama, vLLM, etc.) to `https://` too, taking the node down** - most runtimes don't serve TLS on their own port. Root cause: the marbor agent's transport scheme was never an independently configured property; every marbor agent URL builder in both `internal/router` and `internal/admin` derived its scheme from the node's runtime `URL` field instead of having one of its own, so the two settings were silently the same value under the hood. The marbor agent now has its own `scheme` field, persisted separately (`marbor_agent.scheme` column, defaults to `http` for existing rows) and configured independently via `POST /admin/nodes/{name}/agent {"port": ..., "scheme": "http"|"https"}` - changing it never touches the node's runtime URL, and vice versa. The GPU Nodes UI now has two distinct HTTPS checkboxes (Edit Node's, for the runtime endpoint; the Agent panel's own, for the Agent connection) with copy clarifying they're unrelated - the previous single mislabeled checkbox was the direct source of the confusion. The TLS certificate probe/pin flow (`POST /admin/nodes/{name}/tls-probe`, TOFU pinning, sibling-host consistency, no-downgrade) now also operates against the Agent's own host:port and scheme rather than the runtime URL, since the certificate being pinned was always the Agent's, not the runtime's.
- **GPU Nodes UI: Runtime and marbor agent configuration were still visually mixed in one "Edit Node" modal** even after the fix above gave them independent backend state - the TLS Certificate Fingerprint (probe/pin/reset) section lived in Edit Node (Runtime fields) rather than the Manage marbor agent modal, forcing operators through a "Manage Agent -> enable HTTPS -> go back to Edit Node -> Probe & Pin" round trip and inviting the exact Runtime/Agent confusion the backend fix above was meant to end. TLS Certificate Fingerprint (with probe/pin/reset) now lives entirely inside the Manage marbor agent modal, colocated with the Agent's own port/scheme controls; Edit Node is Runtime-only. Changing an already-enabled Agent's port or scheme is a new capability added by this same change - the previous UI had no way to edit those fields once an agent was enabled, only Regenerate Token and Disable Agent - and it ships gated behind a "Reconfigure marbor agent connection?" confirm from day one, consistent with every Runtime address change requiring confirmation. The dialog is scoped entirely to "Agent connection" and states plainly that it reissues the Agent's token, since `POST /admin/nodes/{name}/agent` always mints a fresh one. The default Agent port shown in the UI is now `9200` (the marbor agent binary's actual default) instead of the stale `11435`. "Regenerate Token" also now shows a confirm dialog first - it invalidates the currently-running agent's token immediately (`POST /admin/nodes/{name}/agent/regenerate` swaps the live token in the router's state right away), and previously fired with no warning.

## [0.19.2] - 2026-08-19

### Added
- **The marbor agent is now a separate binary, `marbor-agent`, instead of a subcommand of `marbor` (`marbor agent ...`).** Capability isolation: a GPU/node-agent host no longer needs an executable capable of starting the control plane, and vice versa. `marbor` has no remaining call path into the agent's runtime-startup logic (`nodeagent.Run`/`service.New`), proven by a source-level test (`TestServerBinary_DoesNotDependOnAgentRuntime`) - `internal/admin` still imports `internal/nodeagent`'s shared wire types and auth-scope constants, which is the frozen marbor agent Protocol v1 boundary, not a capability leak. `marbor-agent` has no path to control-plane code at all (`TestAgentBinary_HasNoControlPlaneCapability`, via `go list -deps`). Same repo, same version, same release, same protocol - only the entry point and shipped artifacts changed. **This is a clean break, not an upgrade path:** `marbor agent ...` now exits 1 with a redirect to `marbor-agent`; there is no existing Ollama Mesh deployment to migrate, so no compatibility/detection logic for a prior installation was built or is needed. `install.sh ROLE=agent`/`install.ps1 -Role agent` install the dedicated binary and still require `TOKEN` or `ENROLL`+`MESH` exactly as the combined binary's agent role always did. `.goreleaser.yaml` ships both binaries as separate release artifacts; `make build`/`make backend` build both.
- **GPU node registration and marbor agent enrollment deployment guides** (`docs/deploy/gpu-node-registration.md`, `docs/deploy/node-agent-enrollment.md`) - document the existing scriptable Admin API workflow (`POST /admin/nodes`, `POST /admin/nodes/{name}/agent`) as two separate guides matching two separate operations, each with its own curl walkthrough and pointer to the matching first-party Ansible playbook below. No new API surface - this documents automation the Admin API already supported.
- **First-party Ansible playbooks for fleet registration and marbor agent enrollment** (`ansible/`) - two independently-idempotent playbooks rather than one combined script, since registering a node's runtime endpoint and enrolling its agent are separate operations an operator may want to re-run independently: `ansible/playbooks/register-gpus.yml` (registers nodes via `POST /admin/nodes`, confirmed idempotent, polls until each reports healthy) with `ansible/inventory.example.yml`, and `ansible/playbooks/install-node-agent.yml` (enrolls/installs agents on already-registered nodes, skipping any node that's already `agentPresent`+`healthy` since `POST /admin/nodes/{name}/agent` always mints a fresh token and would otherwise disrupt a live agent on every re-run) with `ansible/inventory-agents.example.yml`. `ansible.builtin` modules only, no external collections, no new API surface. Source only, checked into this repo - not published to Ansible Galaxy or any external registry.

### Fixed
- **The Admin UI's TLS-mismatch panel (GPU Nodes) showed and copy-buttoned `marbor agent service status` as the command to run on a node** - stale from before the marbor agent binary split; it now correctly shows `marbor-agent service status`, which is the command that actually exists post-split. A doc comment in `internal/admin/admin.go` had the same stale invocation, also fixed.
- **`POST /admin/nodes` (Add Node) silently doubled a node's live capacity when re-registering an existing node name.** SQLite's `UpsertNode` was always idempotent by name (`INSERT OR REPLACE`), but `Router.AddNode`'s duplicate-URL guard only ever compared a new node's URL against *other* names' URLs - it never checked whether the name itself already existed - so re-adding the same `{name, url}` appended a second, independently-polled `*NodeState` to the router's live pool while the database still (correctly) held one row. Reachable via a plain double-click of "Add Node" in the UI, and would become a routine operation once fleet registration is automated (e.g. a re-run Ansible playbook). The DB-vs-router mismatch self-healed on the next marbor restart, but live routing weight/capacity was silently wrong until then. `Router.AddNode` now upserts by name: re-registering an existing name updates that node's config in place (URL included, matching the DB layer's existing unconditional-replace behavior) instead of appending a duplicate. `POST /admin/nodes` now returns `200 OK` for an update and `201 Created` only for a genuine new node. Fixing this also surfaced and closed a latent data race: a node's URL was previously assumed immutable after creation and read unguarded during health polling; polling now snapshots the URL under the node's own lock, since an upsert can now change it from a concurrent request.

## [0.19.1] - 2026-08-18

### Added
- **`internal/cli` is now driven by a single command registry instead of a hand-rolled `switch` statement (P83).** One data-driven tree (`internal/cli/registry.go`/`registry_tree.go`) is now the single source of truth for every command's name, flags, positional arity, and help text - the dispatcher, `--help` output, man pages, `docs/cli.md`, the README table, and shell completions all read the same declaration, so they cannot drift out of sync with each other the way the old scattered `switch`/const-string/hardcoded-map/hardcoded-whitelist combination already had. Adds root/group/leaf `--help` (previously 6 of 12 command groups dumped raw, context-free Go flag defaults) and "did you mean" suggestions for typo'd commands and subcommands.
- **`marbor completion bash|zsh|fish`** - shell completion scripts generated from the same command registry, covering every command and flag. No network calls, no auth required.
- **`man marbor`** - man pages for the CLI, generated from the registry (see the docs-generation entry below).
- **Man pages, a generated `docs/cli.md` reference, and a self-updating README CLI table, all derived from the CLI command registry.** `cmd/gen-docs` (`make man`/`make docs`) walks `internal/cli`'s registry and writes `docs/man/marbor.1` (root) plus one page per command group (`docs/man/marbor-{models,runtime,nodes,node,key,requests}.1`), `docs/cli.md`, and README.md's CLI table (rewritten in place between `<!-- BEGIN/END CLI TABLE -->` markers - the generator fails loudly if those markers are missing rather than guessing where to edit). A new CI step re-runs the generator and fails the build on any drift, so these three surfaces can never go stale again the way the old hand-written README table had (missing 11 of 20 shipped commands before this). `install.sh` now best-effort fetches the man pages from the same release tag as the binary (uploaded as extra GitHub release assets via `.goreleaser.yaml`) into `/usr/local/share/man/man1`, running `mandb` if present; a failed or skipped fetch never fails the install. `uninstall.sh` removes the same files. Generation is deterministic (byte-identical across repeated runs) and stdlib-only - `cmd/gen-docs` is a standalone package never imported by the shipped binary.
- **`marbor uninstall` - a real subcommand instead of a bare word that silently started the marbor server.** Removes the marbor's own systemd service (Linux, if `install.sh SERVICE=1` was used) or nohup-tracked background process if present, and uninstalls the marbor agent service on this host if one is installed - one command covers both roles that can be running on a machine. `--purge` additionally removes the installed binary. `marbor.db` is never touched; remove it yourself if you also want the database gone.

### Fixed
- **`whoami <anything>` (and every other zero-positional-argument CLI command) silently discarded trailing garbage arguments instead of erroring.** Reported: `marbor whoami status` and `marbor whoami dead` both printed the same successful `whoami` output as if the extra word wasn't there. Root cause: the shared flag-parsing helper never checked for leftover positional arguments after `flag.Parse` stopped at the first non-flag token. Every command in the registry now validates positional-argument count; a new test walks the live registry and asserts every command rejects one extra trailing argument, so a future command added without this check fails automatically instead of silently.
- **`marbor key`, `marbor spill`, and `marbor requests` were completely unreachable from the compiled binary**, despite being implemented, tested (against the CLI package's own entrypoint, which bypassed the bug), and documented in the CLI's own `--help` text. `main.go`'s command router held an independent hardcoded whitelist that simply omitted these three names, so typing them exited 1 with "unknown command". The router now derives its whitelist from the same command registry the CLI itself uses, so a command can never again exist in the CLI but not the binary.
- **`marbor logout` never ended the server-side session** - it only deleted the local session file, leaving the session valid and usable (e.g. via a captured bearer token) until it expired on its own. `logout` now also calls the session-invalidation endpoint; a network failure during that call is logged as a warning and never blocks deleting the local file, so `logout` still can't fail or leave a stale local session behind.
- **A CLI bug where a boolean flag other than `--json` could swallow the next argument as its own value** (e.g. a hypothetical `--verbose foo` would consume `foo`) - the internal flag/positional splitter only knew about `--json`; it now derives the full set of boolean flags from each command's own registry declaration.
- **`key set-local-only`/`set-allow-local-degradation`, `nodes confirm-tls`, and `whoami` ignored `--json`** and always printed human-readable text; all four now honor `--json` like every other command. `whoami --json` also stopped silently discarding a JSON-encoding failure's real exit code.
- **Any unrecognized command (including `uninstall`, which was never a binary subcommand) used to silently fall through to starting the full marbor server instead of erroring.** A typo or a missing subcommand could accidentally boot a production marbor process. Unrecognized bare words now print an "unknown command" error with usage and exit 1; dash-prefixed server flags (`-db`, `-seed-node`, ...) are unaffected.
- **`marbor agent service status` no longer prints a stale TLS certificate fingerprint after the service has been uninstalled.** `Uninstall` never deletes the cert/key files it created (so a future re-install keeps the same pinned fingerprint), so the fingerprint was still being printed for a service `status` had just reported as "not installed."

**Compatibility note:** this item deliberately keeps every existing command name, flag name, and successful-invocation output unchanged - the only intentional behavior change is that CLI commands which previously accepted and silently discarded extra/malformed arguments now reject them with a usage error and exit 1. A script or pipeline that was relying on that swallowing behavior (rather than passing correct arguments) will need updating.

## [0.19.0] - 2026-08-15

### Added
- **Per-node in-flight cap that sheds to the next candidate instead of queueing (P64).** A field study found an operator running a reverse proxy in front of marbor specifically to get an instant 503 when a node was overloaded - the existing `queue_max_depth`/`queue_timeout_ms` request queue only engages when the *entire cluster* has no eligible node, not when one specific node (e.g. a slow local model) is individually saturated; `ActiveConns` fed placement scoring but never made a node ineligible, so it could accumulate unbounded concurrent requests and still be selected. A new Tier-1 hard-constraint check (`isUnderCapacity`, sibling to P79's `isEligibleForModel`) now excludes a node once `ActiveConns >= max_in_flight`, wired into all three routing call sites including the sticky-session shortcut - a pinned session cannot bypass the cap. Configurable via a global `routing.max_in_flight_per_node` default (Settings page, `PUT /admin/settings`) and a per-node override (GPU Nodes edit modal, `PATCH /admin/nodes/{name}`, mirrors the existing `vram_total_mb` override pattern - 0 means "use the global default"; both reject negative values and values above `math.MaxInt32`, which would otherwise invert the cap's intent). Failover is handled entirely by the existing `RouteExcluding`/retry/cloud-fallback chain - a capped node is simply excluded from candidacy, so the next-best node or cloud/503 fires exactly as it already does for an unhealthy node. Default is 0 (uncapped) on both the global setting and every node - this is a new hard constraint, so upgrading never starts rejecting traffic until an operator explicitly sets a cap. **This is a best-effort cap, not an atomic concurrency guarantee:** the eligibility check and the connection-count increment are not a single atomic operation (no reserve-then-commit step, unlike the VRAM cold-start reservation path), so a burst of concurrent requests arriving at the same instant can briefly overshoot the configured cap before it takes effect on the next routing decision; closing that gap would require per-node reservation accounting, deliberately out of scope here. The global default also only takes effect on process restart, same as `queue_max_depth`/`overflow_sla_ms` today - none of these routing knobs are live-reloadable yet. Not yet exposed via the CLI - `queue_max_depth`/`overflow_sla_ms` (global) and `vram_total_mb` (per-node override), the two existing settings this feature directly mirrors, have no CLI exposure today either, so this follows that precedent rather than introducing new CLI surface unilaterally.
- **`bench/loadtest` - SQLite write-path capacity measurement (P53).** An earlier architecture-risk audit claimed the per-request `audit_log`/`request_log`/`hourly_buckets`/`model_stats` writes were synchronous on the request goroutine; re-verification found that was already fixed (all three now go through bounded 5000-slot async queues with drop-on-full), but nobody had ever measured what request rate that async design can actually absorb before it starts dropping entries. `bench/loadtest` sweeps a target request rate against the real marbor + real SQLite store, reporting target/sent/completed/failed RPS and p50/p95/p99 latency per step, flagging any step where the load generator itself can't keep up (`GENERATOR-SATURATED`) so a slow client is never mistaken for a saturated marbor, and passively sampling `.db`/`-wal` file sizes (it never issues `PRAGMA wal_checkpoint`, which would actively force a checkpoint and perturb the exact WAL-growth behavior under test). It deliberately does not compute a "ceiling" itself - the operator cross-references the marbor's own queue-full drop log lines against the swept rate. Measured against a single marbor process + single mock backend node: no drops up to 300 req/s, a latency knee around 400 req/s, first queue-full drop around 500 req/s (reproduced across two independent runs) - published in `docs/PRODUCTION.md`'s new "Write-Path Capacity" section, with the caveat that this single-node setup can't cleanly separate queue-capacity limits from the one backend's own service time.
- **Settings page's "Model Context Windows" and "Local Model Fallback Chain" cards now use a searchable model-name dropdown instead of a plain free-text field, including the fallback chain's comma-separated alternates list.** Both cards previously took a hand-typed model name with zero validation against the live fleet - a typo or wrong tag was silently accepted and would never match anything at admission/routing time. The single-value model-name input on each card is now the existing `CustomCombobox` component (already used on the Requests page filters). The Local Model Fallback Chain's alternates field is a list, not a single value, so it gets a new sibling component, `CustomTagCombobox` (`ui/src/components/Select.tsx`) - it autocompletes against whatever's being typed after the last comma and appends a selection as a completed entry rather than replacing the field, so picking several alternates from the dropdown composes instead of overwriting. Both components are populated from `GET /admin/models` (demo mode uses the mock catalog), filter as you type, and still accept any typed value - an operator-declared name for a not-yet-pulled or temporarily unavailable model remains a valid config entry; nothing is force-validated against the live catalog. No Admin API, SQLite schema, or routing-hierarchy change - purely a UI input-affordance fix. The Local Model Fallback Chain card's Add button also now rejects a self-referencing chain (a model listing itself as its own alternate) or a duplicate alternate within one chain client-side, mirroring `config.go Validate()`'s existing server-side rule - previously the client let either into the pending list and the operator only found out on Save, via a generic "validation failed" error; the server-side rule itself was already correct and needed no change.

- **Local-first degradation: an operator-declared chain of local model alternates, tried before cloud egress (P67).** Previously, "the requested local model has no available node" went straight to cloud fallback (or a 503) with no intermediate local option - for a privacy-motivated buyer, a degraded-but-local answer is strictly better than a cloud answer. A new `routing.local_degradation_chains` setting (Settings page's new "Local Model Fallback Chain" card, or `PUT /admin/settings`) maps a model name to an ordered list of local alternates, with no assumption about size/capability ordering and no requirement that an alternate already be downloaded (a cold pull is still better than cloud egress). Substitution requires a new **per-key** `allow_local_degradation` policy (default false) - the operator, not the calling client, decides which keys may receive a locally-degraded model; a client cannot self-authorize it. The chain is walked (single non-blocking `RouteExcluding` call per candidate, never queuing, never touching session affinity) at both existing cloud-fallback decision points - immediately after `WaitForNode` fails, and after the per-node retry budget is exhausted - reusing the existing `X-Ollama-Marbor-Model-Fallback` response header and `requestedModelName` audit/logging fields for transparency. Single-hop only, enforced in code (a `degradedOnce` guard), so even a cyclic operator config can substitute at most once per request. A duplicate alternate within one chain (`{"a":["b","b"]}`) is now a config validation error, same as an alternate listing itself. New `marbor_local_degradation_total{from,to}` Prometheus counter, `PATCH /admin/keys/{name}` `allow_local_degradation` field, API Keys page "Allow local degradation" toggle, and `marbor key set-allow-local-degradation <name> <true|false>` CLI command. No Routing Hierarchy or placement-scoring change - this is a new pre-cloud fallback sub-step, not a scoring factor.
- **Per-key `local_only` policy and fleet-wide cloud-spill accounting (P66) - a data-sovereignty control for buyers who cannot let a prompt leave the premises.** Cloud fallback existed as a global, all-or-nothing resilience feature with no way to say "this key must never leave local nodes" and no visibility into how often fallback actually fired. A new `local_only` boolean on an API key now fails a request closed (503 `local_only_blocked`) instead of spilling to a cloud provider whenever no local node is available - checked at both existing cloud-fallback decision points in `internal/proxy`, before the cloud provider chain is even computed. A new `spill_counters` table (bounded by keys x providers, never grows with request volume) durably counts every completed routing decision per key as `local`, a named cloud provider, or `blocked`; deleting a key never deletes its spill history. Surfaced through `PATCH /admin/keys/{name}` (`local_only`), the new `GET /admin/spill` read endpoint, the API Keys page's edit modal (a toggle next to the USD-cap fields) and a new "Cloud Spill" table on the Analytics page, and two new CLI commands (`marbor key set-local-only <name> <true|false>`, `marbor spill`). Spill accounting is best-effort - a SQLite write failure here is logged and ignored, never allowed to affect an inference request. No schema explosion, no new datastore, no distributed state: one policy column plus one small increment-in-place counter table.
- **CLI persistent auth: `marbor login`/`logout`/`whoami`, so every other command can stop passing `--token`/`--username`/`--password` on every single invocation.** `marbor login` authenticates once (interactively - password input is not echoed - or via `--username`/`--password`, or `--token` to save an existing token) and persists the resulting session to a local file (`0600`, under `os.UserConfigDir()`); every other credentialed command (`nodes`, `models`, `runtime`, `node control`) now falls back to that saved session as the lowest-priority credential source, after `--token`/`MARBOR_TOKEN` and `--username`/`--password`/`MARBOR_USERNAME`+`MARBOR_PASSWORD` - a saved session is never used against a different `--server` than the one it was authenticated against, so juggling multiple marbor instances from one machine can't cross-wire tokens. The saved file deliberately never stores an expiry timestamp - the server remains the sole source of truth for session validity, discovered via a real 401 (which now also prints a "run marbor login again" hint when the expired credential came from the saved file specifically). `marbor whoami` reports the saved identity, live-verified with one request against the Admin API rather than trusted from the local file alone; `marbor logout` deletes the saved session and is idempotent. No Admin API or SQLite schema change - this reuses the existing `POST /admin/v1/login` session-cookie mechanism as-is.
- **CLI Group 2 parity: `marbor models pull|delete|unload|list` and `marbor runtime drain|undrain|health` - closing the last gap between the Admin API/UI surface and the official CLI.** An audit found the Admin API and UI already fully supported model pull/delete/unload/list (per-node) and runtime drain/undrain/health-check, but the CLI (`internal/cli`) had zero commands for any of them, despite exposing the identical `/admin/...` routes the UI already calls. `marbor models pull <node> <model>` (async, mirrors the UI's fire-and-forget pull flow), `marbor models delete <node> <model>`, `marbor models unload <node> <model>`, `marbor models list <node>` (the per-node local inventory, distinct from the existing fleet-wide `marbor models`), `marbor runtime drain <node> [--reason=X]`, `marbor runtime undrain <node>`, and `marbor runtime health <node>` (an on-demand active liveness probe - a populated `ok:false` result is a successful probe reporting a down runtime, not a CLI failure) now all exist, each exactly one Admin API request per the CLI's existing contract. No new marbor agent Protocol capability, Admin API route, or UI surface was added - a `diagnostics.health` capability originally proposed for this item was deliberately not built as a new capability; it would have duplicated the existing `runtime.health_check` (now exposed via `marbor runtime health`), so the CLI gap was closed against the existing capability instead. A permanent "CLI parity" rule has been added to the workflow's Definition of Done: no new Web UI operation may ship without a corresponding CLI command, to prevent this gap from re-accumulating.


- **A manually or scheduled-unloaded keep-warm model is now visible on the dashboard instead of just sitting cold with no explanation.** The suppression that makes an unload "stick" until an explicit rewarm previously had no admin API or UI surface at all - an operator saw a keep-warm-configured model stay cold with no way to tell "deliberately suppressed" from "warmup broken" from "never scheduled." `GET /admin/nodes` now includes a `warmupState` array per node (model, state, reason - `manual_unload` or `scheduled_unload` - and since timestamp); the Warmup & Scheduling page's per-model rows show "Suppressed - unloaded by schedule, resumes on next warmup" in the same slot as a warmup-failure message, and GPU Nodes gets a `SUPPRESSED (n)` badge alongside the existing WARMUP FAILED/UNLOAD FAILED ones. The raw internal suppression map/bool is never exposed directly.
- **`runtime.logs` (P58) - fetch recent log lines from a node's inference runtime process, closing the last Phase-3 gap left by P43.** Once an operator has accepted a control driver for a node, a new "View Logs" button on the GPU Nodes page's Runtime Control panel - and the equivalent `marbor runtime logs <node> [--lines=N]` CLI command - fetch a point-in-time snapshot of the runtime's recent output, wherever the accepted driver knows to look: `journalctl -u <unit>` for systemd, `docker logs --tail` for Docker, `log show` for launchd, and Windows Event Log (`wevtutil`) for a Windows service. This is a snapshot fetch, not a live tail - no streaming/follow flag is used on any driver. The Process driver (a bare PID file with no supervisor) has nothing to read logs from and returns a clear "not supported without a supervisor" error rather than guessing, matching how it already handles Start/Stop with no supervisor configured. New agent route `POST /v1/runtime/logs`, new capability `runtime.logs`, new Admin API route `POST /admin/nodes/{name}/runtime/logs`. Unlike start/stop/restart, this is a pure read and is not audit-logged.
- **`runtime.stop`/`start`/`restart` are now live end-to-end (P43 Step 3), on top of the ControlDriver registration flow shipped earlier.** Once an operator has accepted a control driver for a node (GPU Nodes page's "Runtime Control" panel, or `marbor node control accept`), Start/Stop/Restart buttons - and the equivalent `marbor runtime start|stop|restart <node>` CLI command - actually start/stop/restart the inference runtime process via the agent's systemd/Docker/process/launchd/Windows-service driver. Marbor never asks the agent to guess: it builds `{driver, identifier}` from its own store-backed config on every request and sends it fresh, so the agent itself holds no persisted control state. A node with nothing accepted returns a clear "Runtime control unavailable: no control driver configured" (422) rather than a guess. The Process driver's `Start` also needed a launch command Step 2 never persisted - a new optional `start_command` field (added to the existing `node_control` table, not a new one) is set once at Accept time and only ever sent back for that driver's Start action. New agent routes `POST /v1/runtime/{start,stop,restart}`, new capabilities `runtime.start`/`runtime.stop`/`runtime.restart`, new Admin API routes `POST /admin/nodes/{name}/runtime/{start,stop,restart}`.
- **marbor agent gained a ControlDriver registration flow - the foundation for `runtime.stop`/`start`/`restart` (P43).** A new `ControlDriver` abstraction, orthogonal to runtime detection, answers "how does the agent start/stop/restart the inference runtime process" (systemd unit, Docker container, PID file, launchd label, or Windows service - v1 covers all five). The agent probes its host (confidence-ordered: service manager, then container, then PID file, then a reachable port as a last resort that never selects a driver) and reports what it found; an operator reviews the evidence and explicitly accepts a driver + identifier from the GPU Nodes page's Manage marbor agent modal (new "Runtime Control" section) - a later re-scan can surface a different suggestion, but never silently switches what's accepted. New `GET`/`POST .../accept`/`DELETE /admin/nodes/{name}/control` endpoints; new additive `control` object in the agent's `/v1/status` payload. The actual `runtime.stop`/`start`/`restart` capability wiring (agent action handlers, admin API dispatch, `marbor runtime` CLI commands) lands in a follow-up - this ships the discovery/configuration foundation it depends on.
- **New `marbor` CLI - a thin, read-only client of the Admin API.** `marbor version`, `marbor status`, `marbor nodes`, and `marbor models` are the first four commands, each exactly one Admin API request (never talking to a marbor agent directly). `--json` ships on every command from day one - that's the real compatibility contract, human table output is not - and every command follows a fixed exit-code convention (0 success, 1 user/input error, 2 server error, 4 auth failure). `nodes`/`models` need a session, obtained via `--token` or `--username`/`--password` (which the CLI exchanges for a token through the existing `POST /admin/v1/login`); `status`/`version` need no auth. Build with `make marbor-cli` or `go build -o marbor-cli ./cmd/marbor`; goreleaser now publishes `marbor-<os>-<arch>` binaries alongside the existing `marbor-<os>-<arch>` ones. No `restart`/`stop`/`start`/`delete`/`drain` commands yet - those need a real `ControlDriver` first. `scripts/smoke.sh` now builds the CLI and runs `status`/`nodes`/`models` against the live demo stack's Admin API as its final step, so the CLI is gated the same way as the proxy/admin/metrics surfaces on every smoke run.
- **Admin API v1 surface audited for CLI/SDK consumption; one gap closed.** `POST /admin/login` was the only admin route missing its `/admin/v1/` twin (every other route already had one via the existing route-mirroring helper) - a v1-only client would have had to special-case login. `POST /admin/v1/login` now works identically to the existing `/admin/login`.
- **The Models page now warns when the same model name has different content on different nodes** (e.g. the same tag re-pulled with a different quant/build mid-rollout, without every node picking up the change yet). Ollama already reports a per-model content digest that marbor previously discarded; it's now tracked per node and, when two or more nodes disagree on it for the same model name, a warning badge appears on that model's card, with each node chip's tooltip showing its own digest so you can see which node is the outlier. Other runtimes (vLLM/TGI/llama.cpp/MLX) don't expose a comparable digest today, so this check is silently skipped for them rather than guessing.

### Changed
- **The admin CLI's `--token`/`MARBOR_TOKEN` credential flag has been removed outright (no deprecation period).** A bearer token passed as a CLI argument or read from an env var is still visible in shell history, `ps`/Task Manager, and process-creation logging for the life of the process - the same exposure class already fixed for the marbor agent's own `--token` flag. Unlike that case, the admin CLI has no external users yet and `marbor login`/`logout`/`whoami` (persisted session file) plus `--username`/`--password`/`MARBOR_USERNAME`+`MARBOR_PASSWORD` already cover every legitimate credential path, so there was no compatibility reason to keep a warn-then-remove window open - it's gone now. `--token` is an unrecognized flag on every command as of this change.
- **BREAKING: the standalone `marbor`/`marbor-cli` binary (`cmd/marbor`) has been merged into the main `marbor` binary.** marbor is a single static Go binary by permanent design - shipping a second binary for the Admin API CLI violated that. `marbor version`/`status`/`nodes`/`models`/`runtime`/`node control` now dispatch from the same binary as the server and the existing `agent`/`bench` subcommands, reusing the exact same `internal/cli` logic unchanged; `marbor --help` now shows one unified command table covering the server, marbor agent, benchmark tool, and Admin API CLI instead of separate help systems. Every existing server flag (`--db`, `--seed-node`, `--log-format`, `--version`) and `install.sh`'s invocation pattern keep working byte-for-byte identically. There was no external usage of the standalone `marbor`/`marbor-cli` binary (beta, no released users), so it was removed outright with no alias or migration shim - `go build -o marbor-cli ./cmd/marbor` and goreleaser's `marbor-<os>-<arch>` release asset no longer exist as of this change.

### Fixed
- **The marbor agent's bearer token authorized every route on the agent with no per-action distinction - possessing it meant full read+pull+delete+unload+runtime-control on that node, with no scoping mechanism in place before Group 3 ("Maintain": runtime upgrade/install, config rotation) capabilities can safely ship (P54).** `requireToken` (`internal/nodeagent/auth.go`) did a single exact-match bearer check; every route in `Server.Handler()` was gated identically. A token replayed directly against the agent's HTTP port (the realistic leak channel - a log aggregator, a leaked backup, a departing contractor's copy, not an attacker who already has a shell on the box) could reach every action regardless of what it was actually issued for. Three ordinal tiers now exist (`readonly < operator < admin`), embedded as a `"<tier>."` prefix on the token itself rather than looked up from a database - the agent process has no DB access, only the bare token string it's configured with, so the scope has to travel with it. `GET /v1/status`/`GET /metrics` now require `readonly`; every models/runtime route (pull/list/delete/unload, start/stop/restart/logs/disk, health check) requires `operator`; `admin` is reserved, unused by any route today, exclusively for a future Group 3 action. A request with a valid token but insufficient scope now gets `403` (previously such a concept didn't exist - either the token matched, `200`, or it didn't, `401`). Every token minted before this change - a bare random string with no recognized prefix - parses as `admin` (full scope) via a backward-compat fallback, so an already-enrolled node keeps working unchanged after an agent binary upgrade with no forced token regeneration; new tokens are also minted at `admin` scope by default, since no Group 3 action exists yet to justify restricting one (mechanism-only ship - no UI/CLI scope-selection surface added, nothing operator-facing changes today). `node_agent` gains a `scope` column (plaintext, non-secret, observability only - the agent never trusts this column for enforcement, only its own configured token). No P24/TLS, RBAC/tenant concept, Group 3 capability, or routing/proxy change.
- **The persisted request log and the live `/admin/requests` API both recorded `status:200` for real non-200 backend responses (P82), discovered as a byproduct of P81's real-runtime verification.** `internal/proxy/proxy.go` built a semantic label ("warm"/"loading"/"error"/"aborted" locally, "cloud"/"aborted" for cloud fallback) for `admin.LogRequest`'s status parameter - correct for the cold/warm counters and the `/admin/requests/live` dashboard badge it also drives, but `internal/admin/admin.go` then tried `strconv.Atoi()` on that same label twice: once to populate the persisted SQLite `request_log.status_code` column, once independently for `/admin/requests`' numeric `Status` field. Both `Atoi()` calls always failed on the non-numeric label and silently defaulted to `200`, discarding the real HTTP status `statusRecorder` had captured correctly all along (already used correctly for Prometheus metrics and `audit_log`). `LogRequest` now takes the real numeric status as its own parameter (`statusRecorder.StatusCode()`, threaded from both the local and cloud proxy call sites), independent of the semantic label; a new `RequestLog.HTTPStatus` field carries it into both consumers, and the broken `Atoi()` calls are removed. The semantic label's existing behavior (cold/warm tracking, the live-requests badge) is unchanged. New regression tests (`internal/proxy/status_logging_test.go`): a local 5xx response, a local 4xx response, a cloud-path 429, and a real on-disk SQLite round-trip all now assert the actual status code, not a fake 200.
- **A model pull whose real download size couldn't be resolved from the curated catalog skipped the disk-fit hard block entirely, letting an arbitrary-size download proceed against a node with no free-disk check at all (P73).** `handleNodePull`'s disk-fit gate (`internal/admin/admin.go`) lived entirely inside the branch where `findCatalogVariantSizeMB` resolved a known size; any tag outside the ~26-model curated catalog (an uncurated Ollama registry name, a Hugging Face repo id, a custom quantization) returned `ok=false` and skipped the check - including the Docker-container-volume override - as if the gate didn't exist for that model, a self-documented v1 scope limit rather than a silent gap. A new `classifyUnknownSizeDiskFit` (`internal/admin/catalog.go`) closes it: since there's no size to compare against free space, it applies a conservative policy floor instead - refuse an unsized pull once free disk drops below 10% of the node's total or 5GB absolute, whichever floor the current headroom fails first - rather than fabricating a size estimate (R1). Mirrors `classifyDiskFit`'s telemetry-unknown handling exactly (shared via a new `diskTelemetryUnknown` helper), so a node without disk telemetry behaves identically to how it always has for both known- and unknown-size pulls. The gate sits before the direct/agent pull-path branch, so it covers both paths and all 5 runtimes uniformly, same as the pre-existing known-size check. A previously-succeeding pull of a non-catalog model tag on a node with thin free-disk headroom now returns `507 Insufficient Storage` instead of dispatching unchecked.
- **Model Advisor's multi-GPU sizing (P69) double-counted one physical VRAM pool across two node entries sharing a host, and never disclosed when a fit verdict wasn't backed by any per-device reading at all (P75 Gaps B/C/D).** Gap B: host-scoped agent telemetry (`pollAgentHost`) reports every physical GPU identically to every node sharing a `Host`, so two independent runtime processes each pinned to a different GPU (e.g. two Ollama instances on one 2-GPU box, each launched with a different `CUDA_VISIBLE_DEVICES`) both saw the full `AgentGPUs` array and were each sized against the combined total - a confirmed false-green on an already-shipped, deliberately-supported topology (multi-runtime-per-host). Gap C: vLLM/TGI/MLX with tensor-parallel actually configured were always sized as single-device ("largest"), since `ggufOnlyRuntime`'s runtime-name check has no way to see a real multi-GPU launch. Both share one fix: a new per-node "declared GPU indices" field (`PATCH /admin/nodes/{name}` `gpu_indices`, persisted in `node_overrides.gpu_indices`) lets an operator state which physical GPU indices a specific node/runtime instance actually uses - detected, not declared, was considered and rejected (Law 5: no clean ROCm/Intel/Apple equivalent to `nvidia-smi --query-compute-apps`, and PID-to-GPU correlation degrades under container isolation). `nodeVRAMCapacity` (`internal/admin/catalog.go`) now intersects `AgentGPUs` against a declared scope before sizing: a scope narrowed to one device sizes against that device alone (never the host aggregate); a declared multi-GPU scope switches basis to "combined" regardless of runtime name, since the declaration itself is the operator's confirmation of a real multi-GPU deployment for that instance. An undeclared or stale (matching no reported device) scope falls back to the pre-existing behavior unchanged. Gap D: a node with no agent, or a "declared" whole-node VRAM total (no per-device breakdown), previously looked identical - at the same apparent confidence - to a confirmed single-GPU reading; the catalog/repo-fit responses now carry a `gpu_count_unknown` flag, and the Model Advisor page shows an explicit "GPU count unknown" caveat whenever it's set. New Edit Node modal field ("Declared GPU Indices", comma-separated) lets an operator set the declaration; no CLI command was added for it - `PATCH /admin/nodes/{name}` as a whole has never had CLI coverage for its existing `vram_total_mb`/`gpu_model` fields either, so this is a pre-existing gap not introduced or widened by this item. Confined to the Advisor's own display path - `internal/router/placement.go` reads only `n.VRAMTotalMB` for real routing/eviction and never calls `nodeVRAMCapacity` or reads the new field, so live traffic routing is unaffected. Gap A (the "combined capacity" label not disclosing it means "uses all N GPUs exclusively") remains BACKLOG, unchanged by this fix.
- **Two token-accounting bugs defeated P78 Phase 1's unavailable-vs-zero distinction for real embeddings traffic, both confirmed by inspecting the marbor's own `request_log` (P80).** `statusRecorder.tail` (`internal/proxy/proxy.go`) unconditionally capped the retained response body at 8KB (`tailMax`); a real embeddings response (e.g. nomic-embed-text) is a single JSON document with no newlines, and its `usage`/`eval_count` field can sit anywhere relative to a large embedding array well past that boundary, so `tokenCount()` never saw a fragment starting at `{` and silently returned `0` for a real, known, non-free request - a live example of exactly the failure R1 exists to prevent. `Write` now only truncates to `tailMax` once it has observed at least one `\n` in the body (line-oriented NDJSON/SSE responses, where the count reliably lives in the final short line); a single-JSON-document response is instead buffered in full, up to the same `maxRequestBodyBytes` ceiling the proxy already applies to request bodies - bounded, not an unconditionally larger fixed buffer. Separately and independently, even when `tokenCount()` correctly returned `-1` (unavailable), the one call site logging it (`proxy.go:657-661`) clamped negative values to `0` before `LogRequest`, so the request log/dashboard could never actually show "unavailable" - that clamp is now removed; `-1` passes through unchanged (both downstream consumers, `AddKeyTokens` and the `tokens >= 0` cost-accumulation guard three lines below, already treated it correctly as "skip," so no other code needed to change). Confined entirely to `internal/proxy/proxy.go` - no schema, config, or routing-hierarchy change.
- **A healthy non-Ollama node with the wrong model loaded could silently serve a request for a different model (P79)** - the router had no hard model-identity eligibility check: model presence (`LoadedModels`) was only ever a scoring bonus (`isModelWarm`/`computeNodeScore`), never a prerequisite for candidacy. Live incident (`req-af404f8a`, surfaced during P78 Phase 2 verification): an Ollama node serving `nomic-embed-text` drained, a healthy vLLM node with a different embedding model was selected instead, and `POST /v1/embeddings` for `nomic-embed-text` returned a real HTTP 200 with an embedding from the wrong model - a silent wrong-model success, not an error a caller could detect. A new `isEligibleForModel` check in `internal/router/placement.go` now hard-filters candidates: Ollama nodes remain exempt (they can load/pull a requested model on demand, preserving existing cold-start behavior), but every other runtime is only eligible if `LoadedModels` already contains the exact requested model, using the same match semantics as `isModelWarm`. Wired into all three routing paths that could otherwise bypass it - `routeInternal`, `RouteExcluding`, and `Route`'s sticky-session shortcut (a pinned session can no longer reuse a node ineligible for a newly-requested model). When no node is eligible, routing falls through to the existing no-node/cloud-fallback/503 path unchanged - no new error path was added. No new state, no config field, no marbor agent protocol change, no TaskMode - confined entirely to `internal/router/placement.go` reading data already reported by every runtime's existing poller.
- **Cloud fallback for Ollama's embeddings endpoints (`/api/embeddings`, `/api/embed`) was broken or silently mistranslated in several ways** (P78 Phase 1 - embedding proxy hardening, not full embeddings support; end-to-end local-runtime verification is a separate future phase). `translateCloudPath` had no case for the newer plural `/api/embed`, so a cloud-fallback request for it was sent to `<cloud-base>/api/embed` - an Ollama-only path most OpenAI-compatible providers don't expose; it now maps to `/v1/embeddings` like the legacy singular endpoint (this also automatically extends the existing Anthropic-embeddings-501 rejection to `/api/embed`). The legacy singular `/api/embeddings` request body uses `"prompt"`, but nothing renamed it to OpenAI's expected `"input"` field before forwarding - a new `rewritePromptToInput` step fixes this (`/api/embed` already uses `"input"`, so it needed no change). On the response side, `cloudtranslate.go` had no branch for an embeddings-shaped cloud response at all (only chat/completions `choices[]`), so it fell through to raw OpenAI-JSON passthrough instead of Ollama's native shape - it now translates to the correct shape per endpoint (`{"embedding":[...]}` for the singular legacy path, `{"embeddings":[[...]],...}` for the plural path, embeddings placed by the provider's reported `index` rather than assumed receive order), and carries the cloud's real `usage.total_tokens` through as `prompt_eval_count` when reported (never fabricated). Separately, `statusRecorder.tokenCount` returned `0` for Ollama's legacy `/api/embeddings` local response shape (`{"embedding":[...]}`, no usage fields at all) - a real "no data available" case was being logged as a real zero-token request; it now returns the existing "unavailable" sentinel (`-1`) instead. `/v1/embeddings` (OpenAI-native clients calling the marbor directly) was already a correct pass-through and needed no change. No new endpoints, no schema/config change, no Model Advisor/UI change, no routing-hierarchy change - a real, confirmed gap in the router's model-availability-based node selection (it has no concept of workload/task mode, so nothing today prevents a chat-only vLLM node from receiving embeddings traffic or vice versa) was investigated and documented but deliberately not fixed in this item; it needs its own future design.
- **Model Advisor's Hugging Face search returned unfiltered, mostly-irrelevant results for vLLM/TGI/MLX nodes, and the search UI itself read as a secondary section beneath the VRAM card rather than the primary discovery surface (P77).** Following P76's decision to make Hugging Face Browse the one universal model-discovery path for all five runtimes (the curated Ollama catalog is never extended to other runtimes - HF already serves them natively), `handleModelSearch` (`internal/admin/catalog.go`) only ever added `&filter=gguf` for Ollama/llama.cpp; vLLM/TGI/MLX searches went out with no format filter at all. A live check against Hugging Face's `/api/models` found the obvious fix insufficient - `filter=safetensors` alone returned mostly embedding/vision/time-series repos (only ~1 in 10 of the top-downloaded results was an actual LLM), and `filter=mlx` alone mixed in ASR/vision repos alongside real LLMs. Both now also carry `&pipeline_tag=text-generation`, verified live to cleanly narrow results to genuine text-generation repos for both filters. `ModelAdvisor.tsx` also reorders so the search bar and its filters appear immediately below the node selector, with the VRAM card following as supporting context instead of gating the search UI; the "How to add models" copy no longer hardcodes "GGUF" and now names the active runtime. No CLI search/browse command was added - HF search/browse has never had one, for any runtime, and this item fixes an existing operation's filtering rather than introducing a new one. **Follow-up (same day):** the P76 banner ("This node runs {runtime} - the curated model catalog is Ollama-format only...") has been removed - it was written for a state where the page gave no other explanation, but the "How to add models" copy above now states the same runtime-filtering fact directly, making the banner redundant rather than additive. `curatedCatalogIncompatible` (the client-side helper that gated it) is removed as dead code along with it. **Second follow-up (same day):** the single `pipeline_tag=text-generation` filter added above was itself too narrow - verified live that canonical vision-language repos vLLM and MLX genuinely serve (`llava-hf/llava-1.5-7b-hf`, and the entire actively-maintained `mlx-community/llava-*` catalog) carry `pipeline_tag=image-text-to-text`, not `text-generation`, and would have been silently excluded. Hugging Face's `/api/models` has no OR syntax for `pipeline_tag` (repeated params and comma-separated values both verified live to return zero results) - `handleModelSearch` now issues one upstream request per relevant pipeline_tag (`text-generation` and `image-text-to-text`) for vLLM/TGI/MLX and merges/dedupes/re-sorts the results, capped back to 25. The older `conversational` pipeline_tag was checked and found dead (zero live results) - deliberately excluded rather than silently missing. **Third follow-up (same day):** the two-request merge above was auditing itself into two new issues, both fixed before shipping - the two upstream requests were sequential (compounding `hfHTTPClient`'s 10s timeout to ~20s worst case) and all-or-nothing (one tag's transient failure discarded results the other tag already fetched successfully). Both requests are now issued concurrently, and the request only fails if every tag's fetch failed. Also fixed in the same pass: `ModelCard`'s pipeline_tag badge used `.replace('-', ' ')` (replaces only the first hyphen), rendering `image-text-to-text` as "image text-to-text" instead of "image text to text" - now `.replaceAll(...)`. Verified TGI actually supports vision-language inference (Idefics2/3, LlavaNext, Qwen2/2.5-VL, Mllama - HF ships a dedicated "Vision Language Model Inference in TGI" doc page), confirming `image-text-to-text` was correctly in scope for TGI's filter set, not over-broadened. Re-verified `handleModelRepo`'s safetensors aggregation and `pullFormatIncompatible`'s format gate are both already modality-agnostic (they never distinguished text-only from vision-language repos, so neither needed a change) and that MLX repos genuinely ship `.safetensors` files (not some other extension the aggregation logic would have silently missed).
- **Model Advisor gave a non-Ollama operator zero acknowledgment that the curated model catalog doesn't apply to their node (P76).** P70 fixed the underlying fit-verdict and pull-rejection correctness for non-Ollama runtimes, but the curated catalog (`internal/admin/catalog.go`'s `catalogModels`) has no UI renderer anywhere - so a vLLM/TGI/llama.cpp/MLX operator saw no catalog and no explanation why, only the "Browse Hugging Face" search below (which already defaults to their runtime and already returns only compatible tags). `ModelAdvisor.tsx` now shows one banner line above the GPU Node VRAM card - "This node runs {runtime} - the curated model catalog is Ollama-format only. Search Hugging Face below for {runtime}-compatible models instead." - whenever the active node's runtime can't consume the catalog's Ollama-library-format tags (mirroring `pullFormatIncompatible("ollama-library", runtime)` client-side; no new API call). Still no catalog grid/UI for the underlying data - that remains explicitly out of scope, same exclusion P70 stated.
- **Model Advisor's curated catalog showed a capacity-based green/yellow/red fit for Ollama-only tags on vLLM/TGI/llama.cpp/MLX nodes, and pulling one of those tags (or a GGUF Hugging Face reference) onto an incompatible runtime was never blocked before the download started (P70).** The compiled catalog (`internal/admin/catalog.go`) is entirely Ollama-library-format tags (e.g. `llama3.2:3b`); the node's declared runtime was read but never reached `classifyFit`, so a node running any other runtime saw the same fit verdicts as an Ollama node even though it can never load that format at all - ample VRAM does not make an unloadable format loadable. `handleNodePull` had no runtime/format check either: it gated only on node health, disk space, and agent capability, so a mismatched pull either failed deep inside a cryptic `huggingface-cli` subprocess error, or - for a GGUF Hugging Face repo id stripped down to `org/repo` - downloaded successfully and only turned out unloadable the first time something tried to use it. Both paths now run the same confidence-scoped compatibility check (`classifyPullTagFormat`/`pullFormatIncompatible`): an Ollama-library tag is only compatible with `ollama` (and an undeclared runtime, matching existing defaults); a `hf.co/...`/`.gguf` GGUF reference is only compatible with Ollama and llama.cpp. The catalog's per-node fit gains a new `incompatible` verdict, distinct from green/yellow/red - a capacity word must never also carry a compatibility fact. `handleNodePull` now rejects an incompatible combination with a clear `422` before any bytes move. A bare `org/repo` Hugging Face id (the format vLLM/TGI/MLX's own browse-and-pull flow already produces) is deliberately never blocked - it's ambiguous with an Ollama community-published model of the same shape, and confirming its real format needs repo metadata the pull endpoint doesn't have; this is a stated, deliberate scope limit, not a silent narrowing to Ollama-only (Law 5). CLI parity was already satisfied before this fix - `marbor models pull` (`internal/cli`) dispatches to the same `handleNodePull` handler and already surfaces a 422's error body legibly via `doRequestBody`'s existing `StatusUnprocessableEntity` case, so it inherits this rejection with no CLI code change needed. This also corrects the P22 queue entry's claim that "Pull stays gated on the node's real declared runtime," which was not true of the code before this fix.
- **Model Advisor sized every model against GPU 0 only, producing wrong fit verdicts on multi-GPU nodes (P69).** The node aggregate VRAM fields (`VRAMTotalMB` and friends) have always been sourced from the agent's first reported device only - a deliberate, documented tradeoff for the single-value routing/UI fields that predate the per-device `AgentGPUs` array. The catalog and Hugging Face fit endpoints (`GET /admin/models/catalog`, `GET /admin/models/repo`) inherited that single-device figure as if it were the node's entire capacity, so a 2x24GB node showed every model sized against just 24GB - a false "won't fit" for anything that would actually fit once spread across both cards. Both endpoints now size against `AgentGPUs` when it's present: summed across all devices for a runtime that shards a model across them (Ollama, llama.cpp), or the single largest device for a runtime that pins a model to one GPU (vLLM, TGI, MLX) - summing for the latter would silently turn a real "won't fit on one card" into a false green, which is worse than the false red it replaces. Falls back to the existing single-aggregate behavior unchanged when a node has 0 or 1 reported devices. The catalog response gains two new fields (`gpu_count`, `vram_fit_basis`); the Model Advisor page labels the VRAM line "combined across N GPUs" or "largest of N GPUs (runtime does not shard across GPUs)" whenever they apply, so the fit basis is never left implicit.
- **`GET /admin/settings` emitted `Proxy`/`Auth`/`Nodes`/`Routing`/`Metrics`/`LiteLLM` as capitalized top-level JSON keys instead of lowercase, silently breaking the Settings page's load path for every field under those sections** (proxy port/log level/access log, auth mode, every `routing.*` knob, Prometheus enable/port, LiteLLM). `config.Config`'s struct fields for these six sections had a `yaml` tag but no `json` tag, so Go's `encoding/json` fell back to the exact (capitalized) Go field name on encode - while `PUT`'s decode worked fine regardless (Go's `Unmarshal` matches field names case-insensitively), so saves always appeared to succeed while every reload silently fell back to the UI's hardcoded defaults instead of the real saved value. Found while verifying P67's new Settings card round-trips correctly; fixed by adding the missing `json:"..."` tags to match what the UI (and the existing `PUT` payload shape) already expected.
- **`marbor bench --help`, `marbor agent --help`, and `marbor agent service install|uninstall --help` wrote their help text to stderr instead of stdout**, inconsistent with the GNU convention (`internal/cli`'s commands, `agent service start|stop|status --help`) already established elsewhere in the same binary. Each now intercepts `-h`/`--help` before `flag.FlagSet.Parse` runs and prints to stdout, exiting 0 - a genuine bad-flag error still prints to stderr with a non-zero exit, unchanged.
- **Model Advisor's Pull button only ever appeared for Ollama nodes, even when a node's marbor agent already supported pulling models for vLLM/TGI/llama.cpp/MLX** (`internal/nodeagent/actions.go`'s `pullCommands` map has real entries for all five runtimes). The UI hardcoded `runtime === 'ollama'` instead of asking whether the node's agent actually advertises the `models.pull` capability - the same check `handleNodePull` already uses server-side to choose direct-vs-agent dispatch. The model catalog API (`GET /admin/models/catalog`) now includes each node's `runtime` and effective `capabilities` (mirroring the admin API's own dispatch decision - a node whose agent is present but disabled in settings correctly reports none), and the Pull button shows whenever the node is Ollama or its agent capability list includes `models.pull`.
- **`free_vram_headroom` placement scoring misread vLLM's static KV/weight pre-allocation as VRAM pressure (P62).** vLLM reserves its configured `gpu-memory-utilization` (commonly 0.8-0.9) at startup, so an idle, fully-able-to-serve vLLM node reported ~90%+ VRAM used by design and scored ~1.6 of a possible 20 on that factor, versus ~18 for an idle Ollama node - in a mixed fleet, vLLM nodes were systematically deprioritized for cold-start placement even when correct. Placement now detects static-reservation runtimes (vLLM today; TGI's static mode is deliberately deferred pending confirmation against real telemetry) and scores that factor from in-flight request count instead of free bytes, neither crediting nor penalizing the reserved block. Warm-resident routing (weight 50) was never affected - this only distorted cold-start placement. No config/schema change, no new scoring factor, existing weights unchanged.
- **The Predictive Warmup Engine could silently reload a model an operator had manually or scheduled-unloaded**, if that same model also happened to be a likely-next model in the transition history or a time-of-day pattern - overriding the operator's explicit "keep this cold" decision the same way the periodic keep-warm pinger used to before it respected unload suppression. Both of the engine's warmup trigger points (transition-based top-3 prediction and time-of-day prewarm) now check the same suppression state the manual/scheduled unload path sets, and skip a suppressed model instead of reloading it - visible in the Predictions tab as a normal "skipped" decision, same as any other unmet prediction.
- **A drain didn't actually keep a node empty if any keep-warm config or warmup schedule was still active for it.** Both the periodic keep-warm pinger and scheduled/manual warmup (`WarmModels`) loaded models with no check of the node's draining state - routing and eviction already correctly skipped draining nodes, but warmup itself was never wired to respect it, so a drained node's keep-warm models got silently reloaded on the very next warmup tick. Both paths now skip a draining node entirely; a "warmup" schedule fired against a draining node now records `last_status: error` instead of a false "ok".
- **The marbor agent is now host-scoped: one physical machine running multiple runtimes (e.g. Ollama on :11434 and vLLM on :8000 on the same box) needs exactly one agent process/enrollment, and every node row on that host now correctly shows "Agent" instead of only whichever node happened to be enabled first.** Previously each node row got its own independently-generated bearer token, but only one running agent binary could ever hold the matching token - every other node on the same box would poll with the wrong token, get a 401, and permanently show "No Agent." The agent itself now detects every runtime listening on its host each cycle (not just the first one found) and reports them all; each discovered runtime gets a stable, opaque identity (independent of its type name or port, so reconfiguring a runtime's port never changes its identity) persisted locally by the agent. `NodeState.Host` (new, defaults to the node's own URL hostname - no action needed for existing single-runtime installs) groups node rows that share a physical machine; marbor now polls each host's agent exactly once per interval (not once per node) and fans the result out to every node on it. New optional `host` field on the add-node API/config (`internal/store` `runtime_nodes.host` column, additive), new additive marbor agent Protocol fields (`Telemetry.Runtimes[]`, `RuntimeInfo.id`/`.port` - `ProtocolVersion` unchanged, old agents/marbor binaries keep working via the existing singular `runtime` field). Known limitation, not yet handled: two separate instances of the *same* runtime type on one host (e.g. two vLLM processes) can't be told apart by type alone and fall back to port-based matching, same as before this fix.
- **Runtime Control's discovery could match the marbor agent's own service instead of the runtime it was looking for** (e.g. systemd unit `marbor-agent.service` contains the substring `ollama`, the runtime name being searched for), and separately, **Start/Stop/Restart/Logs were gated on runtime health instead of agent reachability** - stopping the runtime intentionally (as designed) made the router mark it unhealthy on its next poll, which then blocked Start from ever dispatching again through the same gate, a functional dead end. All four discovery probers (systemd/launchd/Windows service/Docker) now exclude the agent's own registered service name before matching, and the runtime start/stop/restart/logs handlers now gate on the marbor agent's own reachability (already tracked independently for telemetry purposes) rather than the runtime's. The "Set Manually" control-driver form no longer silently pre-fills its Identifier field from a (possibly wrong) discovery result while leaving the Driver dropdown on an unrelated default, and now requires an explicit confirmation step before persisting, matching Start/Stop/Restart's existing confirm dialog - both real contributors to nodes ending up misconfigured with the wrong driver.
- **Windows marbor agent service reinstall/upgrade could fail with a port-already-in-use error, and `install.ps1` could fail to even download a new version over a running install.** `sc stop` only requests a stop and returns immediately, without waiting for the old process to exit - `windowsManager.Install` now polls for the service to actually stop (10s timeout) before starting it again, closing a race where the new instance's bind could hit the old process still holding the port. Separately, Windows locks a running `.exe`'s file, so `install.ps1` re-downloading `marbor.exe` over an already-installed, running agent service failed outright; the script now stops that service first (if present) and waits for it to exit before downloading. Neither race exists on Linux/macOS, where `systemctl restart` is atomic and a running binary's file can be replaced freely.
- `install.ps1` now pauses with "Press any key to exit..." before the window closes (skipped when non-interactive, e.g. `$env:CI` set) - previously the console vanished the instant the script finished, whether launched via a double-clicked shortcut or the enroll one-liner's `powershell -Command "...; irm ... | iex"`, taking the install summary or any error message with it.
- **Running `marbor.exe` (or `marbor agent ...`) directly and hitting a fatal error (bad `--db` path, port already bound, invalid config, etc.) still closed the console instantly on Windows** if launched without an existing shell keeping the window open (double-click, a desktop shortcut, Explorer) - the error was on screen for one frame and gone. Every `log.Fatalf`/`os.Exit` call site in `main.go` and `internal/nodeagent` now routes through a shared exit path (new `internal/winexit` package) that, on Windows, pauses for a keypress only when the process owns its own console (`GetConsoleProcessList` reports just this process attached) - already-open terminal usage is unaffected. No behavior change on Linux/macOS.
- **Re-uploading a backup file you'd already downloaded (or a scheduled backup firing when marbor.db hadn't changed since the last run) silently added a byte-for-byte duplicate to the restore list**, under a new timestamp with no indication it was identical to one already there. Both entry points into the backup pool - `POST /admin/backup/upload` and the scheduled backup job - now hash the file (SHA-256) and check it against every existing backup in the target directory before saving; an upload that matches an existing file reuses that file instead of adding a copy, and Settings shows a clear notice ("This is the same backup as `marbor-backup-<timestamp>.db`, already in the list") instead of silently selecting it. A scheduled run that would produce an identical file is skipped (still recorded as a successful run, so the scheduler doesn't retry every tick) rather than cluttering the list.
- **Two concurrent requests for different cold (not-yet-loaded) models could both get routed to the same node**, each seeing the same stale last-poll VRAM snapshot and neither aware the other had just claimed headroom there, before the next poll cycle could catch up - a real double-booking risk on the VRAM-aware placement marbor is built around. Cold-start routing now writes a lightweight, in-memory VRAM reservation the instant a node is picked, and placement scoring discounts a node's free-VRAM headroom by whatever's already reserved-but-unconfirmed there - closing the race without adding any blocking work to the request path (a live poll confirming the model resident clears the reservation immediately, rather than waiting out its 5-minute safety TTL). This reservation is now written as soon as a node becomes the leading scoring candidate rather than only after every candidate has been scored, narrowing the same race further.
- **Every inference request paid for up to three synchronous SQLite writes on the request path** (an audit-log insert, an hourly-bucket upsert, a model-stat upsert) before the response could even be attributed. Audit-log entries and local-request stats now go through the same bounded async queue the request-log already used, off the request path entirely. Cloud-request stats stay synchronous on purpose - the daily/monthly cloud spend cap reads that exact write in real time, and making it async would let a burst of requests blow past a configured budget cap before the writer caught up.
- **A node that just crashed kept scoring as if its last-known VRAM/queue/loaded-models snapshot were still fresh** for the full grace window before health checks flip it unhealthy (several consecutive failed polls). Placement scoring now applies the same penalty an upstream-error cooldown already gets once a node's telemetry is older than that grace window, instead of trusting a snapshot that's actually gone stale.
- **A model name re-pulled with different underlying weights on some nodes but not others (a mid-rollout update, a mismatched quantization) was invisible to routing** - warm-residency scoring and session pinning treated every node's copy under that name as fully interchangeable. The router now remembers the first-observed content digest per model name and stops crediting a digest-mismatched copy as a warm hit; a runtime that doesn't report a digest (anything but Ollama today) is never flagged.
- **Sticky-session pinning was in-memory only - a marbor restart dropped every in-flight session**, forcing a cold KV-cache round-trip on each one's next request. Session affinity is now persisted and restored on the same schedule as warm state; an entry already past its TTL at restore time is skipped rather than restored and immediately swept.
- **`install.sh`/`install.ps1`'s `TOKEN=` (legacy/manual) marbor agent install path re-serialized the token it already had as an environment variable back into a `--token=` CLI argument** when handing off to `marbor agent service install`, exposing it via `ps`/Task Manager for the life of that one-shot install process. Both installers now rely on the binary's existing `TOKEN` env var fallback instead of passing `--token` at all (`install.sh` additionally uses `sudo -E` so `TOKEN` still reaches the child process when installing without root). Directly passing `--token=<token>` to `marbor agent` or `marbor agent service install` still works unchanged, but now prints a one-line deprecation warning to stderr pointing at the `TOKEN` env var / `ENROLL` flow instead; the flag itself is deprecated, not yet removed.

## [0.18.0] - 2026-07-30

### Changed
- **Open WebUI integration guide overhauled based on real-world testing against current Open WebUI releases.** OpenAI-compatible connection is now the primary recommendation (Ollama-native's API key field is inconsistent across Open WebUI versions), Docker Compose is now the lead example, and new sections cover Desktop-vs-server, localhost-vs-LAN-IP, Docker networking, and a connectivity check to run before touching the Open WebUI UI at all.

### Added
- **marbor.db now has a real backup story: manual download plus scheduled auto-backup, both from a new Settings > Backup & Restore card.** Until now there was zero backup mechanism anywhere - losing `marbor.db` meant losing every node, API key, routing rule, and setting with no way back. "Download Backup Now" streams a consistent point-in-time copy (SQLite `VACUUM INTO`, safe alongside live traffic) straight to your browser; Scheduled Backup lets you enable automatic backups with a configurable interval (hours) and retention (how many backup files to keep, oldest pruned first), writing to a configurable target directory (`GET`/`PUT /admin/settings`'s new `backup` object, `POST /admin/backup` for the manual download). `docker-compose.yml` gained a second named volume (`marbor-backups:/backups`, `MARBOR_BACKUP_DIR` env), kept deliberately separate from the existing `marbor-data:/data` volume - so deleting the container, or even wiping the data volume, doesn't take the backups down with it, and vice versa. The same card can also restore a listed backup with one click (see next entry); a fully manual fallback (`docs/backup.md`) remains for anyone not running under a process supervisor, and restore is never restricted to any particular directory - any backup file works from any path, on Docker or bare metal alike.
- **The restore picker's file list no longer grows unbounded on screen** - it's now a single dropdown (select the backup, one fixed Restore button next to it) instead of one row per file, which would have kept expanding forever as scheduled backups accumulate. A new **+** button next to it lets you attach any `.db` file from your own machine (a plain `<input type="file">`, so it works the same on Chrome/Firefox/Safari/Edge across Linux, Windows, and macOS) - marbor validates it's a genuine SQLite database (the same `PRAGMA quick_check` every other backup path uses) and saves it into the same target directory under the standard `marbor-backup-<timestamp>.db` name, so it shows up in the dropdown and restores exactly like a scheduled backup. A spinner replaces the **+** icon while the upload is in flight. New `POST /admin/backup/upload` endpoint.
- **One-click restore from a scheduled backup, from the same Settings > Backup & Restore card.** Picking a listed backup and clicking Restore validates the file (`PRAGMA quick_check`, rejecting anything corrupt or unrelated before touching the live database), gracefully drains in-flight requests, stages the full copy alongside the live `marbor.db` and only swaps it in via an atomic rename once the copy is proven complete (a failure mid-copy leaves the live database untouched), then exits the process. Because marbor doesn't restart itself, this relies on the deployment's own process supervisor to bring it back up - `docker-compose.yml` now sets `restart: unless-stopped` for exactly this reason, systemd's example unit already has `Restart=on-failure`, and Kubernetes' default `restartPolicy: Always` needs nothing extra. Running the bare binary with no supervisor at all means a restore will stop marbor and leave it stopped until started manually - `docs/backup.md` calls this out explicitly rather than leaving it a silent surprise, and documents the fully-manual alternative for that case. New `GET /admin/backup/list` and `POST /admin/backup/restore` endpoints back the picker.
- **A model pull is now hard-blocked when the target node doesn't have enough free disk space for it, with no override.** Unlike a VRAM-fit warning, a pull that exceeds free disk WILL fail (partial download, or worst case fills the node's disk and disrupts the OS/other running models) - so unlike the VRAM confirm-anyway dialog, there is no "pull anyway" option here. The check runs both client-side (Model Advisor's Pull button now disables with a "No Disk Space" badge) and server-side (`POST /admin/nodes/{name}/pull` itself now returns 507 before ever starting the download), so it can't be bypassed by calling the API directly. Uses the marbor agent's existing disk telemetry (`Node.DiskFreeGB`) - no new wire field. Only covers models with a known size today (the curated catalog, and Hugging Face repo browsing where the size comes from HF's own reported file sizes); an arbitrary/uncurated Ollama registry tag has no size to check against and skips the check entirely rather than guessing one. Skipped (never fabricated as "OK") whenever disk telemetry itself is unavailable - no agent, or a non-Linux agent, which doesn't report disk stats yet.
- **marbor agent gained an on-demand "Check now" health probe.** Passive node health already rides in telemetry, but it's only as fresh as the last ~15-30s poll cycle. A new `runtime.health_check` agent capability (`GET /v1/runtime/health`) triggers a real, live liveness probe against the node's own runtime right now - Ollama via `GET /api/tags`, vLLM/TGI/llama.cpp via their real `GET /health`, MLX via `GET /v1/models` (mlx_lm.server has no dedicated health route) - and reports a real ok/error/latency result, never a cached or fabricated one. Surfaced as a "Check now" button in the GPU Nodes page's Manage marbor agent modal, shown only for nodes whose agent reports the capability.
- **Model Advisor now has a per-user Favourites tab.** Star any Hugging Face model while browsing to save it, then switch to the Favourites tab to see just your starred models as a list - no more re-searching every visit. Favourites are scoped to the logged-in admin account (backed by a new `favorites` table keyed on user id), so each operator has their own independent list.
- **Model pulls can now opt into a "verify it loads" check.** A model whose blob downloads fine can still fail to actually load - most often a community Hugging Face GGUF whose declared architecture isn't supported by the node's installed runtime version - and previously the only way to find out was the first real chat request (or the Hardware Benchmark) hitting a cryptic runtime error, long after the pull reported success. Pulling with `"verify_load": true` (a checkbox on the Pull Model modal and Model Advisor, checked by default for community/Hugging Face pulls) now runs one real chat-completion probe (the same one the Hardware Benchmark uses) after the download succeeds and before reporting `success`. A model that fails this probe gets a distinct `load_failed` status with Ollama's real error surfaced verbatim, plus a one-click Delete button right there - nothing is auto-deleted. A model that passes is evicted right back afterward, so a compatibility check never has the side effect of silently keeping a model warm (and possibly displacing whatever was already resident).
- **A hidden hardware benchmark page** for validating the warm-routing speedup on your own hardware without leaving the dashboard. Reachable only via a "Hardware Benchmark" card on Settings (no Sidebar entry - it's a self-service diagnostic, not part of normal operation): pick an already-known node + model, and it runs a live cold-vs-warm Time-To-First-Token comparison through the marbor's own proxy, streaming progress and showing a shareable result card. No manual API key entry needed - the marbor auto-provisions a scoped, ephemeral key for the run and deletes it unconditionally when the job ends (success, failure, or cancel). Results persist to a new `benchmark_runs` history table (`GET /admin/benchmark/runs`).

### Fixed
- **Model Advisor's VRAM fit-check (green/yellow/red) classified against a node's currently-free VRAM instead of its total capacity, producing false "Too Large" verdicts.** Free VRAM shifts with whatever else happens to be warm on the node right now - a model that needed more than the transient free figure but well within the node's total capacity (e.g. a model roughly the size of what's already loaded) was wrongly marked red, even though it would fit fine once other models evict or idle-time-out. The advisor now answers "will this model ever fit on this hardware" (against total VRAM capacity) rather than "does it fit this instant" - that instantaneous check is already routing's job (`free_vram_headroom` placement factor). Separately, the Pull button no longer hard-blocks on a red verdict with only a disabled-tooltip explanation - clicking Pull on a red-flagged variant now shows a confirm dialog ("needs ~X GB, node has ~Y GB total - pull anyway?") so an admin can still proceed when they know better.
- **The "Check now" on-demand health check returned a hard 501 for any node without a marbor agent (or one running a pre-`runtime.health_check` build)**, even though marbor already runs a real, agent-free liveness probe against every node's runtime on its own periodic poll cycle. `GET /admin/nodes/{name}/health-check` now falls back to that same probe (read-only - it never touches the periodic poller's own health/failure-count state) when no agent capability is available, so the button now works for every node. The "Check now" button (GPU Nodes' Manage marbor agent modal) is no longer gated on agent capability either.
- **The Request Log page could stay empty forever even with real traffic flowing, then double-count every request once audit logging was on, with every row showing a broken "0" status badge.** Five compounding issues: (1) per-request audit logging (the Request Log page's data source) defaulted to off, while the Dashboard's "Live Requests" widget reads a separate always-on in-memory buffer - so a fresh install showed traffic on the dashboard but nothing in the log; (2) that dashboard widget itself displayed the *oldest* entries in its 50-row buffer instead of the newest, so it barely updated under sustained load; (3) toggling audit logging on in Settings updated the database but never reached the already-running logger, so enabling it mid-session silently kept doing nothing until a restart; (4) every request was actually written to `audit_log` twice - once by the proxy's own logging call, and again by a redundant internal write inside `LogRequest` under a separately-generated ID - invisible while audit logging defaulted off, but producing exact duplicate rows the moment it was turned on; (5) the status column stored a semantic label (`warm`/`loading`/`error`) instead of the real HTTP status code the frontend's badge and the server's own success/client_error/server_error filter both expected, so every row showed as a red "0" regardless of actual outcome. Audit logging now defaults to on and applies immediately when toggled, the dashboard widget shows the true latest 10 requests, the Request Log page requests up to 1000 rows instead of silently capping at 50, each request is logged exactly once, and the status column reflects the real response code.
- **The dashboard rendered a blank white page on the `:main` image** - a dependabot PR bumped `react` from 19.2.0 to 19.2.8 without bumping `react-dom` alongside it, and React 19's `react-dom` enforces an exact version match against `react` at mount time, throwing before anything rendered. `react-dom` is now pinned to the same 19.2.8. `scripts/smoke.sh` only exercises API endpoints and never caught this class of failure since it doesn't load the SPA in a browser.
- **The Hardware Benchmark page's model dropdown showed no models at all for a node without a marbor agent (or one still detecting its runtime), even though that node had models on disk.** It only ever consulted the marbor agent's `models.list` capability, falling back to the router's live warm-models view (empty whenever nothing happens to be loaded in VRAM) rather than the same `/admin/models` aggregation the Models page already uses, which queries the node's `/api/tags` directly. It now falls back to that aggregation instead, so all on-disk models show up regardless of marbor agent presence.
- **The Hardware Benchmark page let you pick an embedding-only model (e.g. `mxbai-embed`, `nomic-embed`) and always failed it with a cryptic "does not support chat" error** - the benchmark measures chat-completion Time-To-First-Token, which embedding models have no endpoint for. Ollama's own architecture classification (`details.family` from `/api/tags`) is now threaded through the marbor agent's `models.list` response and the `/admin/models` catalog, and known embedding/encoder families are excluded from the model picker.
- **Cancelling a hardware benchmark run mid-sample didn't actually abort the in-flight request**, so the ephemeral API key it provisioned stayed live for up to 5 minutes (the per-sample timeout) instead of being deleted immediately - `bench.MeasureChatTTFT` now takes a `context.Context` threaded from the job's cancel, so a cancelled cold/warm sample tears down its outbound request right away. Separately, **two benchmark runs started for the same node at nearly the same time could race** - both would pass the "not already running" check before either was recorded, then evict/reload the model out from under each other and corrupt both runs' numbers; the check-and-reserve now happens atomically under one lock before any I/O, so a second run on a busy node gets a `409` instead. Also: the Cancel button was a no-op when a run was started via the in-app Demo Mode toggle rather than the build-time public demo, and an unrecognized progress-stream phase (e.g. from a newer marbor version) could leave the page showing raw text with no way back - both now resolve correctly.
- **An idle (downloaded but not loaded) model on a vLLM/TGI/llama.cpp/MLX node never showed up on the Models page at all.** `GET /admin/models` only ever sourced rows from live warm state and Ollama's `/api/tags` - the marbor agent's own `models.list` capability (already used by the per-node "manage agent" view) was never consulted here, so a downloaded-but-idle model on any non-Ollama runtime was invisible to the Models page's Pull/Delete/Advanced Settings actions. It now also merges in each agent-capable node's idle catalog, skipping any (model, node) pair already reported by the existing two sources so warm counts for Ollama-only fleets are unaffected.
- **A model pull's progress bar vanished on browser refresh, even though the download was still running.** The progress widget tracked pulls in a module-level in-memory map with no persistence - a page reload wiped it, with nothing to re-populate it even though the pull kept running server-side as its own goroutine. A new `GET /admin/pulls` endpoint lists currently-downloading jobs from the marbor's own bookkeeping; the widget now calls it once on mount and resubscribes to each one's progress stream, so a refresh mid-download picks the bar right back up instead of losing it.
- **A newly-pulled model didn't appear on the Models page until the next 5-second poll, or a manual refresh.** The pull-progress tracker and the Models page catalog were entirely decoupled - nothing told the page a pull had just finished. The Models page now listens for pull completions and refetches its catalog immediately when one succeeds, instead of waiting on the next poll tick.
- **A "Ping warmup now" click could silently fail with no visible result, and the Warmup & Scheduling page never refreshed on its own.** The trigger is fire-and-forget on the backend (it returns immediately while the actual per-model pings run in the background), but the page had no polling interval at all - a resulting warmup failure only ever showed up after a manual reload or navigating away and back. The page now polls every 10s (matching GPU Nodes' cadence) and additionally re-fetches a few seconds after a manual ping, so failures surface without a reload.
- **A keep-warm model failing its warmup ping, or a scheduled unload failing, was invisible anywhere except the Warmup page or the marbor's own log output.** The GPU Nodes page already polls every 10s but never surfaced `warmupErrors`, so an operator watching node health during an incident had no clue a keep-warm model was stuck without separately navigating to Warmup & Scheduling. Scheduled unloads had it worse: a failed scheduled/agent unload was only ever logged to stdout, with no dashboard-reachable signal at all - and the schedule itself reported `last_status: ok` regardless, since that status was stamped the instant dispatch started, before the actual (async, per-model) warmup/unload work had even begun. Node cards on GPU Nodes now show a "WARMUP FAILED" / "UNLOAD FAILED" badge (with a per-model tooltip) when either map is non-empty; scheduled warmup/unload now wait for every model's real outcome before the schedule records its status, so `last_status`/`last_error` on `GET /admin/schedules` reflects what actually happened, not just that the node was found.
- **A single dropped marbor agent poll (one TCP blip, one 5-second timeout) instantly blanked all of that node's agent-derived telemetry** (fan/RAM/disk/GPU/runtime status) on the GPU Nodes dashboard, unlike the node's own inference-runtime health, which requires several consecutive failures before flipping unhealthy. Agent polling now uses the same consecutive-failure hysteresis before clearing telemetry or firing an `agent_down` webhook, so a transient blip no longer blanks the dashboard for one cycle. Separately, a rolling-upgrade "agent build is newer than this marbor understands" warning was meant to log once per node but re-logged on every recovery from a transient poll drop - it now latches for real.
- **A marbor agent that started before its local inference runtime (vLLM/TGI/llama.cpp) was up (a routine boot-ordering race on restart or upgrade) permanently omitted that runtime's status and warm-models from every `/v1/status` response for the rest of the agent process's life**, even once the runtime came up moments later - runtime detection only ever ran once, at agent startup. The agent now retries detection on every refresh tick until it succeeds, at effectively no cost once a runtime is found (the retry path is never reached again afterward). Affects all GPU vendors identically (the race is about runtime startup timing, not GPU hardware).
- **Unloading a model on a llama.cpp router-mode node always 400'd "model is not found," even for a model genuinely present.** Callers send the Hugging Face `org/repo` identifier (the same format used everywhere else an HF-cache-sourced model is named), but the router's own model id is a bare filename stem with no substring relationship to `org/repo` at all - so an exact-match lookup could never succeed. Unload now resolves `org/repo` to the router's id by matching each loaded preset's `--model` path, refusing with a clear "ambiguous - N presets match" error (listing the candidate ids) when a repo has more than one quant loaded, rather than guessing.
- **The marbor agent install command shown in the admin UI baked the real, permanent bearer token directly into a copy-pasteable `curl | sh` / PowerShell one-liner**, so it ended up in `~/.bash_history`, SSH session logs, or any runbook/chat message the command got copied into - a durable, cross-machine leak of a token that authorizes pull/delete/unload on that node forever. The command now carries a short-lived (20-minute), single-use enrollment code instead (`ENROLL=<code> MESH=<url>`); `agent service install` exchanges it for the real token over a new `POST /admin/agent/enroll` call, so the permanent token only ever travels in an HTTP response body, never argv or a pasted command. `TOKEN=<token>` keeps working unchanged as a manual fallback for existing scripts.

## [0.17.4] - 2026-07-21

### Added
- **A keep-warm model that fails to warm now shows why, instead of silently never becoming resident.** Warmup ping failures were only visible as an unlabeled Prometheus counter bump - an admin watching the Warmup & Scheduling page had no way to tell a stuck model apart from one still warming up normally. Each node now tracks its last warmup-ping error per model (`GET /admin/nodes` / `GET /admin/nodes/{name}` gain `warmupErrors`), cleared the moment a later ping for that model succeeds. The Keep-Warm list on the Warmup & Scheduling page shows the error inline under the model name when present.

### Fixed
- **A dropped pull-progress connection permanently showed a model download as failed, even when it kept downloading successfully in the background.** The progress widget's SSE stream (`GET .../pull/progress`) can legitimately drop mid-download - a sleeping browser tab, a Wi-Fi blip, an intermediate proxy's idle timeout - none of which touch the actual pull, which runs as its own server-side goroutine independent of any admin's open tab. The widget treated the very first stream error as the pull itself failing, closed the connection for good, and showed "Lost connection to the progress stream" with no way back except Retry - which then hit a `409 pull already in progress` from the marbor's own dedup check, since the real download was still running (or had already finished) the whole time. The widget now resubscribes to the same job's progress (a stateless, idempotent GET) after a drop, up to 5 attempts with backoff, and treats a 409 on retry the same way - both cases pick back up on the job's real state instead of reporting a false failure.
- **Deleting a local model showed a garbled error and could silently double-fire.** `ollama rm`'s stderr spinner writes raw ANSI terminal escape codes (cursor hide/show, synchronized-update mode, cursor positioning) even when its output is captured rather than attached to a real terminal - these rode along uncleaned into any error surfaced through the marbor agent's shared subprocess-error path (`runDownload`, used by pull/delete/unload), rendering as garbled box characters in the Models page's delete confirm dialog. They're now stripped before the error ever leaves the agent. Separately, the delete confirm dialog could be dismissed (Cancel, X, or backdrop) while a delete was still in flight - a real delete of a large model over slow/network storage can take a minute or more with no other visual cue, so a user assuming it had stalled could close the dialog and reopen it, firing a second overlapping delete for the same model/node (the second one hitting a "not found" error once the first quietly succeeded). The dialog is now un-dismissible while a delete is in progress, and shows an explicit "this can take a minute" note instead of just a static "Deleting..." label.

## [0.17.3] - 2026-07-21

### Added
- **marbor agent Protocol v1: multi-GPU array, host capacity/identity, runtime version, and a resource-oriented API.** The marbor agent's wire format is renamed and restructured from a narrow "telemetry" schema into the permanent protocol every future node capability builds on. `GET /telemetry` becomes `GET /v1/status`, returning a `{agent, capabilities, host, gpu, runtime, health}` envelope; `POST /actions/pull_model` becomes `POST /v1/models` (pulling a model creates a model resource, capability renamed `actions.pull_model` → `models.pull`); `GET /metrics` is unchanged (left unversioned per Prometheus convention). New data: the `gpu` resource now reports **every** GPU on the host as an array (`gpu.devices[]`, with per-device `core_percent`) plus host-level `driver_version`/`cuda_version` - previously only the first detected GPU was ever reported. `host` gains `ram_total_mb`, `disk_total_gb`, `hostname`, `uptime_seconds`, `boot_time`. `runtime` gains `version` (from `ollama version` where available) and `status` (live reachability), and now also reports `warm_models`. `agent` gains `node_id` - a stable UUID generated once on first install and persisted locally, surviving agent binary upgrades and hostname/IP/DNS changes. The `marbor`-owned scheduler metrics (prefix/session-affinity hit rate, warm-residency confidence) that the original design sketch included are deliberately **not** part of the agent's payload - the node reports what it knows about itself, marbor reports what it computes centrally, through its own existing surfaces. GPU Nodes dashboard and the manage-agent modal now show the full per-GPU breakdown, host capacity, and runtime version/status where an agent reports them; demo mode's `gpu-node-01` shows a plausible 2-GPU node exercising the whole new shape. No dual-shape backward-compat decode was needed for this rename (no external fleet depended on the old shape yet); going forward the new envelope is the frozen v1 baseline additive changes build on.
- **A node's host/port is now editable after it's added**, instead of the only option being remove-then-re-add (which loses the node's history for no real reason). The GPU Nodes edit modal now includes Host/Port fields alongside the existing GPU Model/Runtime/VRAM overrides; changing either is gated behind a confirm dialog since it re-points live traffic to a different address immediately and resets that node's health/warm-state (`PATCH /admin/nodes/{name}` now accepts `url`).
- **marbor agent can now pull models locally on the node (`actions.pull_model`), and pull failures show the real reason.** Previously a model pull always went through marbor proxying straight to the node's Ollama HTTP API, which had no way to pass a Hugging Face token to a gated/rate-limited download and was bound by the marbor's own outbound HTTP client timeout for a transfer it wasn't a party to - failures surfaced as a generic "Bad Gateway" with no indication of why. `POST /admin/v1/nodes/{name}/pull` now dispatches to the node's marbor agent when one is enabled and reports the new `actions.pull_model` capability: the agent runs the locally-detected runtime's own download mechanism (`ollama pull`, `text-generation-server download-weights` for TGI, `huggingface-cli download` for vLLM/llama.cpp), with the marbor's configured Hugging Face token injected into that one subprocess's environment only - never written to disk, never logged. Both this path and the pre-existing direct path now surface the real upstream/agent error text instead of a bare status code. Pulling a raw Hugging Face repo id (e.g. one copied from a model page, ending in `-GGUF`) into the free-text "Pull Model from Registry" field is also now auto-prefixed `hf.co/` to match the format Ollama actually requires, instead of failing outright. Nodes without an agent, or an agent build predating this capability, are unaffected.
- **Model pulls now show live, cancellable progress instead of a blocking spinner.** A pull request returns immediately and runs in the background, tracked by node+model; a browser-download-style widget (bottom-right, on every page) shows real bytes transferred, speed, and time remaining for the direct-to-Ollama path, or an elapsed-time indicator when the download only reports through the marbor agent (never a fabricated percentage). Pulling several models at once - across one node or several - shows one card per pull, each labeled with its node so they stay distinguishable. Cancelling a pull (with a confirm step) actually tears down the in-flight download on the node, not just the dashboard's view of it; a failed pull can be retried in place, and a finished one dismissed.
- **Manual "Ping warmup" button on the Warmup & Scheduling page.** `POST /admin/warmup/ping` (trigger a marbor-wide warmup pass on demand) previously had no UI caller - only reachable via a direct API call. A button next to the page title now triggers it and reports the result inline.
- **Schedules now show when they last ran.** Previously the only record of a warmup/unload/drain/undrain schedule firing was a server log line - an admin had no way to tell from the dashboard whether a schedule had actually run, or silently done nothing (e.g. pointed at a since-removed node). Each schedule row on the Warmup & Scheduling page now shows a "last ran Xh ago" / "never ran" indicator, red with the failure reason on hover if the last dispatch failed. This is live in-memory state (resets on marbor restart, same as other runtime signals) exposed via `last_run_at`/`last_status`/`last_error` on `GET/POST/PATCH /admin/schedules`.
- **NVIDIA NIM added as a cloud fallback provider preset.** Settings > Cloud Providers now offers "NVIDIA NIM" alongside OpenAI/Anthropic/OpenRouter/Groq/etc, pre-filled with `https://integrate.api.nvidia.com/v1` and a default model - it uses the same OpenAI-compatible `Authorization: Bearer` schema every non-Anthropic preset already goes through, so no proxy/routing changes were needed, only the preset entry.
- **MLX runtime support (Apple Silicon).** `mlx` joins `ollama`/`vllm`/`tgi`/`llamacpp` as a valid node runtime (`mlx_lm.server`, ml-explore/mlx-lm). It speaks the same OpenAI-compatible wire format as vLLM/TGI/llama.cpp, so request translation needed no new code - only enum wiring (config validation, admin API, marbor agent model pulls via `huggingface-cli`, Model Advisor search recognizing `library:mlx`-tagged safetensors repos instead of mislabeling their quantization, and the GPU Nodes runtime dropdown/badge). mlx is reachable only via an explicit `runtime: mlx` config - it is deliberately never auto-detected, since `mlx_lm.server`'s `/v1/models` response has no reliable signature distinguishing it from llama.cpp's.
- **Model Advisor: manual "Browse format" dropdown decoupled from the node's declared runtime.** Previously Model Advisor's Hugging Face search always filtered GGUF-vs-safetensors strictly off the selected node's own runtime, so browsing MLX/vLLM/TGI safetensors repos was only possible by switching the node itself. A new dropdown (defaulting to the node's actual runtime) lets an admin browse any format - e.g. check GGUF repos while the active node runs MLX - without changing what the node is configured to run. Pull availability and the advanced config modal still key off the node's real runtime, never the browse override, so nothing lets you pull a format the node can't actually load.
- **marbor agent GPU telemetry now supports AMD ROCm, Intel, and Apple Silicon, not just NVIDIA.** GPU vendor detection was already behind a `GPUCollector` interface with only one implementation (`nvidia-smi`); three more now plug into the same interface with no other code changes: `rocm-smi` (AMD, temperature/fan/power/utilization/VRAM/model), `xpu-smi` (Intel, temperature/power/utilization/VRAM/model - Intel Data Center GPUs are fanless so fan speed is never reported), and `system_profiler` (Apple Silicon, model name only - unprivileged `system_profiler` exposes no temperature/fan/power, and unified memory has no separate VRAM figure to report). A node's GPU model name is now also auto-filled from whichever vendor tool the agent detects the first time it reports one, replacing an empty or placeholder "Unknown GPU" label without ever overwriting a name an operator explicitly set. **AMD and Intel support have not yet been validated against real hardware** (built against each tool's publicly documented output format) - flagged here rather than silently shipped as equivalent-confidence to the NVIDIA path.
- **The GPU Nodes VRAM bar's data-source badge ("nvidia-smi", "declared", etc) went blank for every Node-Agent-sourced reading, on every vendor including NVIDIA.** The badge's label lookup only recognized `nvidia`/`api`/`declared`/`none`, but the real value marbor reports for any remote node with a marbor agent enabled is `agent` - a value the lookup (and the TypeScript type itself) didn't even list, so the badge silently rendered nothing instead of naming the source. It now shows the actual tool the agent detected - `nvidia-smi`, `rocm-smi`, `xpu-smi`, or `system_profiler` - falling back to a plain "agent" label only if the vendor is somehow unreported.

### Added
- **marbor agent can now list models already downloaded on a node, not just currently loaded (`models.list`, `GET /v1/models`).** Previously the only visibility into a node's models was "Loaded Models" (currently warm/resident) - there was no way to see what's downloaded-but-idle. Ollama nodes report this via their own `GET /api/tags`; vLLM/TGI/llama.cpp/MLX have no HTTP endpoint for "everything downloaded" (their own `/v1/models` reports only the currently-loaded model), so the agent scans the local Hugging Face cache directory instead, the same one marbor agent model pulls populate. `GET /admin/nodes/{name}/models` dispatches to the node's agent when it reports the capability - a node without an agent, or an older agent build, returns a clear "not supported" response rather than a fabricated empty list. The Manage marbor agent modal on GPU Nodes now shows a "Locally Available Models" section when the node's agent supports it.
- **marbor agent can now remove a locally-downloaded model from a node (`models.delete`, `DELETE /v1/models/{name}`).** Ollama nodes delete via `ollama rm`; vLLM/TGI/llama.cpp/MLX (no first-class delete primitive of their own) remove the model's directory from the same local Hugging Face cache `models.list`/model pulls already use, with the resolved path validated to stay inside that cache directory before anything is removed. `DELETE /admin/nodes/{name}/models/{model}` dispatches to the node's agent when it reports the capability - a node without an agent, or an older agent build, returns a clear "not supported" response rather than silently reporting a delete that never happened. The Manage marbor agent modal's "Locally Available Models" list now has a delete button per model, behind a confirm dialog since this removes real files from disk (re-pulling later re-downloads the full model).
- **marbor agent can now unload a model from VRAM through the same multi-runtime path as pull/list/delete (`models.unload`, `POST /v1/models/{name}`).** The existing manual "unload" action (GPU Nodes dashboard, `POST /admin/nodes/{name}/unload`) only ever called Ollama's own runtime HTTP API directly (`keep_alive:0`) - it silently did nothing on vLLM/TGI/llama.cpp/MLX nodes. It now dispatches through the node's marbor agent when one reports the `models.unload` capability (Ollama, via `ollama stop <model>` run agent-side); nodes without an agent, or an agent build predating this capability, fall back to the unchanged direct path. Verify-before-build found no genuine per-model "unload without a full restart" primitive in vLLM's (process-scoped sleep-mode, dev-flag gated), TGI's (never shipped one, project now archived), or MLX's (no unload endpoint in the official server) current released versions - those runtimes return a clear "not supported" error rather than a fabricated success.
- **marbor agent's `models.unload` now also supports llama.cpp's router mode** (following up on the above). Router mode (`llama-server` started without a model path, e.g. via `--models-dir`/`--models-preset`) exposes a genuine `POST /models/unload {"model": "<name>"}`, confirmed against the official server docs. Router mode is opt-in, not the default llama-server deployment shape, and the agent's existing "llamacpp" runtime signature can't tell a router-mode instance apart from a plain single-model one (both answer the OpenAI-compatible `/v1/models`) - so the agent first confirms router mode is genuinely running via a read-only `GET /models` probe (a route only router mode exposes) before ever attempting the unload, refusing with a clear "router mode not detected" error otherwise rather than guessing. No new config or protocol field was needed: the runtime's local address was already resolved by the agent's existing runtime-detection code, just not previously threaded through the unload path.
- **`agent_down`/`agent_up` webhook events for marbor agent reachability.** Previously the only webhook alerts (`node_down`/`node_up`) tracked a node's inference runtime (Ollama/vLLM/etc) - if a node's marbor agent (the telemetry sidecar) stopped responding while the node itself stayed healthy and serving inference, the only visible sign was the GPU Nodes dashboard quietly reverting fan/temp/CPU/RAM/disk to `-`, with no alert. The same webhook mechanism (HMAC-signed, fires on genuine transitions only - not on an agent that was never configured or was deliberately disabled) now also fires `agent_down`/`agent_up` per node.
- **`marbor.db` now refuses to start if its recorded schema version is newer than the running binary supports**, instead of silently limping forward. Previously `migrate()` was forward-only with no version marker at all - rolling back to an older marbor binary after upgrading (e.g. reverting a bad release) had zero guardrail; the old binary would just run its smaller migration list against a DB that may already have newer columns/tables it doesn't understand, risking a silent no-op at best or silent data corruption at worst. A `schema_version` value is now stamped into the existing `settings` table on every successful boot; if a binary opens a DB whose stamped version is higher than its own, it now fails loudly with a clear error instead of proceeding.
- **Rolling prefix locality routing preference (opt-in, default off).** When a new request's conversation shares a rolling hash with a recently completed request from the same API key, the router softly prefers the node that served the earlier one, on the theory that the same node is more likely to still have that conversation's KV cache warm - real-hardware measurement of that effect is pending, this only changes routing preference, not any measured outcome. The hash is scoped per API key (or a reserved anonymous identity for unauthenticated requests) so two different keys sending the same conversation never share routing state, and only a one-way hash of the conversation plus the serving node name is ever retained - never raw prompt or message content. It is a weak, additive placement signal (`Settings` → Routing → "Prefix Locality Hints", configurable weight, default 10) - strictly weaker than warm residency and queue depth, so it can never cause a cold start or override a real capacity/health decision, and it only ever records a routing preference after a request has fully and successfully completed. Hit/miss counts are visible on the Warmup page and via Prometheus (`marbor_prefix_locality_hits_total`/`_misses_total`).

### Changed
- **Removed the redundant per-card "Pull to {node}" input on the Models page.** Every model card duplicated the top-right "Pull Model" button's functionality with its own single-node pull field, targeting only the first healthy node with no node choice - the top-right modal already covers the full catalog with a real node selector. Cards now go straight from "Loaded on" to "Delete from node."
- **"Delete local model" moved from the GPU Nodes page to the Models page, with multi-node selection.** The delete action (and the "Locally Available Models" list it lived in) was previously buried inside GPU Nodes' "Manage marbor agent" modal - a model-management action organized by node instead of by model. It now lives on each model's card on the Models page: a single-node model deletes directly behind a confirm dialog, a model resident on multiple nodes gets a node dropdown first so deleting the wrong node's copy isn't a one-click accident. `DELETE /admin/nodes/{name}/models/{model}` is unchanged; a node whose agent doesn't support `models.delete` still returns a clear "not supported" error rather than a silent no-op.

### Security
- **marbor agent bearer token was embedded in the service definition's command line on every platform** (systemd `ExecStart`, launchd `ProgramArguments`, Windows `sc.exe binPath`) - readable by any local user on the node via the world-readable unit file/plist or via `ps`/Task Manager while running. Token now flows through each platform's environment mechanism instead (systemd `EnvironmentFile`, launchd `EnvironmentVariables`, Windows service registry `Environment`), none of which appear in argv.
- **Admin session cookie was forwarded to backend GPU nodes and cloud providers.** The proxy stripped only the `Authorization` header before forwarding a request upstream or to a cloud fallback - every other browser header, including the `MARBOR_session` httpOnly cookie on any request that shared a client with the admin dashboard, passed through unchanged. `Cookie` is now stripped at both proxy paths alongside `Authorization`.
- **Request log's "source IP" trusted client-supplied `X-Forwarded-For`/`X-Real-IP` headers unconditionally**, so a directly-reachable marbor (no trusted reverse proxy in front of it) could have its logged client IP forged by any caller. A new `proxy.trust_proxy_headers` setting (Settings > Proxy, default off) gates whether these headers are trusted; disabled, the log always uses the real TCP peer (`r.RemoteAddr`).

### Fixed
- **Model Advisor's per-variant "Pull" button was permanently disabled in Demo Mode**, same root cause as the Models page fix above - gated on `isLive`, which Demo Mode always sets to `false`. Gate is now `(!demoMode && !isLive) || ...`, matching the rest of Demo Mode's pull surfaces.
- **Demo Mode's "Pull Model from Registry" node selector only ever offered one fake node ("gpu-0"), regardless of how many nodes the rest of Demo Mode shows.** `openPullModal`'s demo branch built its own single-node stub instead of using the same `mockGPUNodes` list every other page in Demo Mode already draws from. It now uses `mockGPUNodes` directly, so the selector shows all 5 mock nodes (with their real health states) like everywhere else in the demo.
- **The UI's fallback version display (`__APP_VERSION__`, shown whenever `/admin/health` hasn't returned a live version - always true in Demo Mode) was stuck on `0.16.0` while the shipped product was already several releases ahead.** `ui/package.json`'s `version` field is bumped on every release now (see CLAUDE.md's updated release checklist) so this fallback stays accurate.
- **The top-right "Pull Model" button on the Models page was permanently disabled in Demo Mode.** It was gated on `isLive`, which Demo Mode always sets to `false` (there is no backend to be "live" against) - even though the pull modal already had a full demo-mode branch that worked end-to-end once opened. The button is now gated on `!demoMode && !isLive`, so Demo Mode can actually demonstrate the pull flow.
- **A small model could be permanently, wrongly reported as "cannot make room" on a node with plenty of real free VRAM, if a much larger sibling keep-warm model was mid-reload at the same time.** Headroom/reservation math (`estimateModelSizeBytes`) estimated a not-yet-loaded model's VRAM need from its on-disk weights size - a reasonable guess before the model has ever been observed loaded, but wrong for any model that doesn't fully fit in VRAM and gets split across GPU+CPU by the runtime (a real case: an 8B Q4_K_M model was ~9.6GB on disk but only used ~3.3GB of actual VRAM once loaded). Every time that model reloaded, its in-flight reservation used the full 9.6GB disk figure instead of its real ~3.3GB footprint, poisoning every other keep-warm model's headroom check on the same node for as long as the reservation lived - a sibling model a fraction of that size could look permanently unfittable even on a node with room to spare. The real, previously-observed VRAM footprint (from the last `/api/ps` poll while the model was actually resident) is now cached per (node, model) and used ahead of the on-disk size the moment it's known, matching the State Hierarchy (live observation beats a derived guess) - the on-disk size is now only ever used before a model's real footprint has been seen for the first time.
- **A scheduled or manual "unload" on a model that's also on a node's keep-warm list got silently undone by the very next warmup ping.** `pingWarmupModels` re-pings every keep-warm model on a fixed interval (default 5m) regardless of why it went cold, so an operator running a "unload gemma4 at night" schedule (or clicking manual unload from GPU Nodes) would see the schedule fire and succeed in the log, only to find the model back in VRAM within minutes - with no error anywhere, since the reload was the warmup ticker doing its job as designed, just fighting the operator's own decision. A manual/scheduled unload now suppresses that model's warmup pings on that node until it's explicitly re-armed - by a "warmup" schedule targeting it, by `WarmModels`, or by toggling the node's keep-warm config again. Auto-eviction under VRAM pressure (LRU headroom) is unaffected and still re-warms on the next tick as before, since that path is meant to be temporary.
- **The Model Fit / Model Advisor VRAM-source badge falsely claimed "nvidia-smi" for any AMD/Intel/Apple Node-Agent-sourced reading.** `handleModelFit`'s source-labeling fell through to a hardcoded `"nvidia-smi"` for any raw source other than the marbor's own local `nvidia`/`declared`, so a ROCm or Intel node's VRAM total displayed the wrong tool as its origin. It now names the actual detected tool (`nvidia-smi`/`rocm-smi`/`xpu-smi`/`system_profiler`), and the GPU Nodes and Model Advisor badges color any of these as a live reading instead of only recognizing the literal string `"nvidia-smi"`.
- **MLX's Advanced Settings showed two controls that silently did nothing, and hid four that actually work.** `seed` and `response_format` were injected into every non-Ollama runtime's request unconditionally, but mlx_lm.server's documented `/v1/chat/completions` schema (SERVER.md) has no equivalent for either - setting them for an MLX model had no effect. Meanwhile `top_k`, `min_p`, `repetition_penalty`, and `logit_bias` - all real, documented mlx_lm.server fields - were hidden from the Advanced Settings modal and never injected, because MLX had been declared with no extra fields beyond the strict-OpenAI base set. Both sides are now corrected against the live mlx-lm source: `seed`/`response_format` are excluded for MLX, and the four real extras are shown and injected.
- **GPU Nodes' CPU stat never showed a real number.** The admin API never serialized the Node
  Agent's collected host CPU utilization at all - `CPUPercent` was polled and stored on the
  node's in-memory state but silently dropped before reaching the JSON response, so the
  dashboard's CPU tile always rendered blank regardless of platform or agent status. It's now
  serialized (`cpuPercent`, gated by `agentPresent` the same way as fan speed/RAM/disk, `--`
  when no agent is reporting) and demo mode only shows a value on the one mock node that has an
  agent, matching how fan/RAM/disk are already demoed. The manage-agent modal copy also now
  mentions CPU usage alongside fan speed, RAM, and disk free.
- **Keep-warm models on a VRAM-constrained node flipped resident/evicted at random instead of settling on a predictable winner.** When two or more models were configured to always stay warm on the same node (via the Warmup page toggle) but didn't fit in VRAM together, the warm loop built its per-node model set from a Go map - whose iteration order is randomized every tick - so which model warmed last (and therefore survived LRU eviction) changed from one warmup cycle to the next. The "keep warm" list order is now treated as an explicit priority hierarchy (first in the list = highest priority): the warm set is built in list order and a higher-priority model can never be evicted to make room for a lower-priority one. The same model now reliably wins every time on a node that can't hold the full set, instead of thrashing. The Warmup page's "Models to keep warm" list is now an explicit, reorderable priority list (numbered rows with up/down controls, same pattern as the cloud-provider priority list in Settings) instead of an unordered toggle grid, so that priority is something you can see and set on purpose rather than an invisible side effect of click order.
- **The time-of-day scroll wheel in the schedule/expiry time picker was over-sensitive and janky on desktop.** Mouse-wheel input translated raw scroll delta directly into scroll position against a hardcoded row-height guess, so a single wheel notch could skip past several values, and snap-back animations could visibly stutter. Wheel input is now normalized to one value per notch (measured against actual row height), decoupled from touch/drag scrolling which still behaves natively.
- **Settings and API Keys pages could freeze permanently mid-navigation** (URL/sidebar updates, page content doesn't) on any build, including the public GitHub Pages demo. `useCurrency`'s cross-tab sync effect called `setPref()` with a freshly-allocated object on every sync, which never equals the previous value by reference - so any two mounted `useCurrency()` instances (the always-mounted budget banner plus whichever of Settings/API Keys was open) endlessly re-triggered each other's sync listener, permanently starving React's ability to commit the next route transition. The sync listener now bails out to the previous object reference when the value hasn't actually changed, matching the value-equality guard already used elsewhere for polling effects.
- **marbor agent's model-pull subprocess (`ollama pull`, `huggingface-cli download`, etc) could panic with `$HOME is not defined`** when the agent ran as a service (systemd or similar) whose own environment had no `HOME` set - the child process inherited that gap and ollama's own CLI has no fallback for it. The agent now guarantees `HOME` is present in the subprocess's environment, resolved from the OS user record when missing, before spawning any pull.
- **GPU Nodes' CPU percentage showed the raw unrounded float** (e.g. `2.3383084577114466%`) instead of a readable two-decimal figure.
- **Loaded models' VRAM size was wildly wrong on the GPU Nodes page** (e.g. a real 3.1GB model showing `3249746.0GB`) - the display divided the model's raw byte count by 1,024 (bytes→KB) and labeled the result "GB" instead of dividing by 1,024³. Now uses the same byte-formatting helper already used correctly elsewhere on the same page.
- **A node added without typing a GPU model name permanently stored the literal string "Unknown GPU" as its label**, rather than treating it as an empty/unset display fallback - so it could never later be auto-filled by real data. Adding a node with the field left blank now sends no model name at all; "Unknown GPU" is purely a display fallback shown when nothing (operator or agent) has set a real one yet.
- **A scheduled "unload" from the Warmup & Scheduling page silently did nothing on vLLM/TGI/llama.cpp/MLX nodes.** The manual unload button (GPU Nodes dashboard) already dispatches through the node's marbor agent when it reports `models.unload`, but the scheduled unload action still only ever called Ollama's own `keep_alive:0` HTTP API directly - worse than the manual case, since a scheduled action has no admin API response to surface an error through, so the no-op was completely invisible. Scheduled unload now shares the same agent-vs-direct decision as the manual path: a node with an agent enabled and reporting `models.unload` dispatches through it; a node with no agent (or an agent that doesn't report the capability) falls back to the direct path completely unchanged.

### Removed
- **Peer health monitor (`ha` config block, `GET /admin/ha/peers`, "Peer Health Monitoring" Settings card) removed.** It only ever polled other marbor instances' `/health` endpoints for passive observability - no failover, no shared state, no leader election, and it never provided real HA despite the name. marbor is a single-instance control plane by design (see Architecture Laws); this module's premise (multiple marbor peers) doesn't fit that model, and marbor agent already covers the real multi-machine need (one marbor, many GPU nodes). Anyone who had `ha.enabled: true` set can safely drop it - the `ha` config block and its `ha_*`/`ha_peers` settings keys are silently ignored (not read, not migrated).
- **`GET /admin/config` (and its `/admin/v1/config` alias) removed.** Dead route - no UI caller, no docs, no test coverage. It returned a masked dump of the running config, but Settings.tsx already exposes every editable config field through `/admin/settings`; this was a leftover from before that page existed. `POST /admin/config/reload` (config reload) is unaffected.

## [0.17.0] - 2026-07-17

### Added
- **marbor agent (v1): real GPU/host telemetry for remote nodes, read-only, with a vendor-agnostic collector architecture and a self-installing persistent service.** A new `marbor agent` subcommand runs a small HTTP server on the GPU node itself, reporting temperature, fan speed, power draw, VRAM, CPU, RAM, and disk free via a versioned JSON schema (`GET /telemetry`) plus a Prometheus-compatible `GET /metrics` derived from it. Telemetry is collected by a background goroutine on a fixed interval (5s default, `--refresh-interval` to tune) into a cached snapshot - `/telemetry` and `/metrics` only ever read the cache, so nvidia-smi never runs on the request path regardless of poll frequency or how many things scrape the node. The payload carries a `last_updated` timestamp so a stale/stuck collector is visible rather than silently serving old numbers as if fresh. Marbor polls it on the existing node poll cycle - no new transport, no WebSockets/gRPC.
  - GPU collection is behind a `GPUCollector` interface (nvidia-smi is the first implementation, selected once at startup from a candidate list) so AMD ROCm, Apple Silicon, and Intel can be added later as new implementations without touching the agent's HTTP server, wire schema, or the marbor-side poller. A CPU-only node (or an unsupported vendor) gets an explicit "no GPU" result, never a fabricated reading.
  - The agent installs itself as a persistent, auto-restarting OS service - `marbor agent service install` registers with systemd (Linux), launchd (macOS), or a native Windows Service via `sc.exe` (Windows), no new dependencies, no manual unit files or `sc.exe` calls for the operator to write. Re-running install (e.g. after an upgrade, or to rotate the token) reconfigures and restarts the existing service in place.
  - The one-line install command generated from the GPU Nodes page (Linux/macOS via `install.sh`, Windows via the new `install.ps1`) now actually downloads the binary and runs the full service install, instead of assuming the binary already exists and running only in the foreground.
  - Each node gets its own opaque bearer token (AES-256-GCM encrypted at rest, distinct from the marbor's client-facing API keys - a leaked agent token can only feed fake telemetry to one node, not access inference or other secrets). Enable/regenerate/disable an agent from the GPU Nodes page; nodes without an agent installed keep showing `-` for these fields exactly as before, never a fabricated estimate.
  - Agents advertise a `capabilities` list (`["telemetry"]` today) instead of the marbor assuming every agent build supports every feature - future capabilities (models, diagnostics, actions, maintenance) get added to this list in the same change that implements them. Every `/telemetry` response also reports `agent_version`, `schema_version`, `platform`, `architecture`, `gpu_vendor`, and the locally-detected inference `runtime` (if any), surfaced in the GPU Nodes manage-agent modal and as Prometheus labels (`nodeagent_info`) - makes debugging a mixed-version, mixed-vendor fleet possible from the telemetry data alone.
  - **Protocol frozen for rolling upgrades.** Verified (with tests) that an older marbor and newer agents, or a newer marbor and older agents, can coexist in the same fleet indefinitely: unknown fields are always ignored on decode, and every field already treats "missing" as "unknown," never a fabricated zero. Marbor now logs once per node if an agent ever reports a newer `schema_version` than it understands, purely for operator visibility. This is the last planned change to the `/telemetry` wire format before real-world usage.
- **API key expiry is now editable after creation, with time-of-day precision.** Previously `expires_at` could only be set at key creation and had no field at all in the edit modal - once created, a key's expiry could never be added, changed, or cleared. `PATCH /admin/keys/{name}` now accepts `expires_at`, and both the create and edit forms use a combined date/time picker (was date-only on create) so expiry can be scoped to the hour/minute, not just the day.

### Fixed
- **System Audit Trail's action-type dropdown blurred the rows behind it when opened, on both desktop and mobile.** The desktop table wrapper was the only one in the app still using a translucent `bg-card/30 backdrop-blur-sm` background instead of the solid `bg-card` every other page's data table uses, and the mobile card view directly beneath the filter bar had the same translucent-plus-blur combination on each card - so an open dropdown panel bled a frosted-glass effect onto whatever sat underneath it. Both now use a solid `bg-card`, matching the rest of the app.
- **Mobile viewport pass across the whole admin UI.** Every page with a data table (Requests, System Audit, API Keys, GPU Nodes, Users, Routing, Analytics, Dashboard) previously fell back to raw horizontal table-scroll below 768px; each now gets a proper stacked card view with full action parity instead. The date/time picker used site-wide (filters, key expiry, schedules) could render its popup panel past the right edge of a narrow viewport with no clamping - it's now portaled and clamped to stay on-screen at any width. Several pages (Metrics, ModelAdvisor, Warmup, Settings) had individual flex/grid rows that didn't collapse to a single column on narrow screens and now do.
- **Status/action badges (System Audit, GPU Nodes, Requests) used dark-mode-only accent colors** (e.g. bare `text-amber-400`) with no light-mode variant, so they lost contrast against a light background. Now theme-aware (`text-amber-600 dark:text-amber-400` pattern) like the rest of the UI.
- **Past dates were selectable in the API key expiry picker**, only rejected after a round-trip to the server. The picker now grays out and disables any day before today.
- **A key's `expires_at` was silently dropped on marbor restart.** It was applied in-memory at creation but never written to `marbor.db` (`runtime_keys` had no `expires_at` column) and never re-loaded into the auth middleware at boot - a key created with an expiry effectively lost it the next time marbor restarted.

### Security
- **Secrets are now encrypted at rest in `marbor.db`.** Cloud provider API keys, marbor-issued API keys, the LiteLLM key, HuggingFace token, and webhook secret were previously stored as plaintext columns/settings - readable by anything with access to the SQLite file (backups, misconfigured storage, a copied `.db`). They're now AES-256-GCM encrypted, with the key held in a separate `marbor.db.key` file (0600) generated on first boot, or supplied via `MARBOR_ENCRYPTION_KEY` (base64 32-byte key) for operators who want to manage it themselves. Existing installs migrate transparently on first boot after upgrading - no manual step, no re-entering keys.

## [0.16.0] - 2026-07-16

### Added
- **Custom Select, date picker, and time picker components across the whole UI**, replacing native browser `<select>`/date/time inputs - consistent theming, no more browser-default styling breaking out of the dark UI, dropdowns that flip position instead of clipping out of modals, and styled native checkboxes/number-input spinners.
- **Admin-configurable audit log retention** (`Settings` → Global Warmup & Audit → "Audit Log Retention (days)"). Replaces a hardcoded "keep the last 10,000 rows" trim that ran a `DELETE` on every single proxied request. Now a periodic sweep (startup + every 12h) removes `audit_log` rows older than the configured window; set to `0` to keep every entry forever instead.
- **Separate, independent retention setting for the System Audit page** (admin action trail - who changed what) via "System Audit Retention (days)", next to the request-log one. This table previously had no pruning at all. Defaults to `0` (forever) rather than the request log's 30 days, since it's low-volume and security-sensitive.
- **Requests page key/node filters are now a searchable, autocompleting list** (backed by the marbor's actual current API keys/nodes) instead of free text requiring an exact guess, and a precise custom "From" date/time filter joins the existing "Until" one instead of only three canned lookback presets. Every text/date filter box also gets an inline clear (×) once it has a value.
- Cloud fallback providers now try in operator-defined priority order (editable via up/down reorder controls in Settings), falling through to the next provider on connection failure instead of always using the same one.
- LiteLLM integration, when enabled, now takes over cloud fallback entirely - the per-provider priority list is disabled in the UI and ignored at request time, since LiteLLM already manages its own provider ordering and retries.
- **Anthropic cloud fallback provider actually works now.** Selecting "Anthropic" as a cloud provider previously reached a hardcoded 501 for `/api/chat`, `/api/generate`, and `/v1/chat/completions` because those get translated to an OpenAI-shaped path (`/v1/chat/completions`) that Anthropic doesn't expose. Marbor now translates requests to Anthropic's native `/v1/messages` schema (system prompt pulled out of `messages[]`, `max_tokens` required field, `x-api-key`/`anthropic-version` auth instead of `Authorization: Bearer`) and translates the response back to OpenAI shape - streaming included - so it flows through the existing cloud-fallback and Ollama-NDJSON pipeline unchanged. Embeddings still 501s (Anthropic has no equivalent).

### Fixed
- **The "Test Key" button for cloud providers (Anthropic, OpenRouter) accepted garbage keys.** The test call was a no-op that always reported success regardless of whether the key actually worked - it now genuinely gates on the provider's own auth response.
- **Requests page key/node filters silently returned zero rows on partial input.** They matched by SQL exact-equality while the model filter used substring `LIKE` - typing anything but the full, exact key/node name looked broken. Both now match by substring, consistently with model.
- **`system_audit_log` (System Audit page) grew completely unbounded** - unlike the request audit log, it had no row cap or pruning of any kind. Now covered by the periodic prune above.

### Changed
- **BREAKING: `config.yaml` is gone. marbor is now fully DB-first (SQLite/`marbor.db`).** A fresh install starts blank-slate: the binary opens/creates `marbor.db`, prints a banner pointing at the dashboard, and everything - nodes, API keys, ports, routing knobs, Docker auto-discovery, HA peer monitoring, webhooks, thermal watchdog, cloud providers, model context windows - is configured from the admin dashboard (`admin`/`admin` default login, forced password change on first login) or the admin REST API. `main.go` no longer has `--config`/`CONFIG_PATH`/`-validate`; it has `--db` (SQLite path, default `marbor.db`, or `MARBOR_DB_PATH` env) and a new `--seed-node "name=...,url=...,runtime=..."` (repeatable) that writes a node directly to the DB and exits, used by `install.sh`'s discovery wizard. Config hot-reload (`kill -HUP` / `POST /admin/v1/config/reload`) now re-syncs live state from SQLite instead of re-reading a file. `Settings.tsx` gained full parity coverage for every field that previously had no UI (~20 fields) plus an editable Cloud Providers CRUD table and a Model Context Windows editor.
- **`install.sh`'s network probe is now the default flow, not opt-in.** After install, it always scans the local subnet for Ollama/vLLM/TGI/llama.cpp, prints a numbered list of what it found, and interactively prompts which to add (`comma-separated numbers`, `all`, or `skip`) - selected nodes are seeded via `--seed-node`. No config file, API key, or admin-token generation happens in `install.sh` anymore. `FORCE_PROBE=1` re-runs the wizard against an existing `marbor.db`.
- **`uninstall.sh`** no longer references `config.yaml` (`KEEP_CONFIG` env var removed) - only `marbor.db`, the pidfile, systemd unit, and log file are ever touched.
- `config.example.yaml` deleted (no file-based config left to document).

## [0.15.1] - 2026-07-14

### Changed
- **Model configuration overrides are now keyed by `(model, node)` instead of model alone.** The same model name can be resident on nodes with different runtimes (Ollama/vLLM/TGI/llama.cpp) or simply different VRAM budgets, and a single shared-by-model-name profile couldn't express either case. Config resolution moved from before routing to right after node selection (it can't be known which node's profile applies until a node is actually chosen). New `GET /admin/model-config/capabilities` endpoint is the single source of truth - for each runtime, exactly which fields take effect - read by both the injection code and the UI's field filtering, so they can't drift apart. The Advanced Settings modal keeps one card per model with a node-selector inside it.
- **The model-config parameter matrix was corrected and substantially expanded**, verified against each runtime's actual current source/API (not memory or older docs): removed 7 fields that don't exist in current Ollama (`flash_attention`, `offload_kv_cache_to_gpu`, `rope_frequency_base/scale`, `use_mlock`, `tensor_parallelism`, `tfs_z`) - these were being silently dropped by Ollama's own server on every request. Added the 3 real Ollama fields that were missing (`num_keep`, `main_gpu`, `draft_num_predict`). Expanded llama.cpp from 7 to 17 real extra fields (DRY/XTC repetition control, `logit_bias`, `n_probs`, `min_keep` - its server README confirms these are accepted on its OpenAI-compatible endpoint too) and vLLM from 3 to 10 (`length_penalty`, `stop_token_ids`, `min_tokens`, etc.) via a new "Runtime-Specific Extras" UI section.
- `Reset this node to defaults` and the equivalent destructive actions across the admin UI (suspend/delete/reset-password on Users; drain/toggle-predictive-prewarm on GPU Nodes; routing strategy change; schedule delete; toggle predictive warmup engine; config reload) now confirm before firing, matching the pattern already used for Remove Node/Unload Model/Revoke API Key.

### Fixed
- **`system` prompt override wasn't actually applied for vLLM/TGI/llama.cpp** - only Ollama ever received it. Now prepended as a leading `{"role":"system",...}` chat message for every runtime, never overwriting a client-supplied system message.
- **`install.sh`'s network probe misidentified the marbor's own admin port as a TGI backend.** Two bugs: the TGI check only verified an HTTP 200 on `/info` rather than the response content, and the marbor's own admin server serves its embedded dashboard as a catch-all for any unmatched path - so any marbor instance (including itself) false-positived as a TGI node. Separately, the self-IP skip list broke on any host with more than one local network interface (e.g. any Docker host), so the subnet scan probed the marbor's own LAN IP anyway.
- **The "Reset password" action on the Users page fired the actual password reset (revoking all sessions) immediately on click**, before any confirmation - the confirm-looking modal only showed the result afterward. Now gated behind a real confirm step first.

### Added
- **Real vLLM/TGI/llama.cpp mock servers in the demo stack** (`cmd/mocknode`, renamed from `cmd/mockollama` since it's no longer Ollama-only - one binary, a `RUNTIME` env var selects the personality). Each implements exactly what the marbor's own runtime-detection/health probes call, verified against that code, not each project's full real API. `make demo` now brings up all 5 backend nodes (2× Ollama, vLLM, TGI, llama.cpp) by default.

## [0.15.0] - 2026-07-14

### Added
- **Advanced model configuration overrides**: operators can now set a persisted default parameter profile per model - all 30 Ollama/LM-Studio-class knobs across load-time/engine (`num_ctx`, `num_gpu`, `flash_attention`, `num_batch`, `num_thread`, `use_mmap`/`use_mlock`, RoPE, `ttl`, tensor parallelism), inference-time sampling (`temperature`, `top_p`, `top_k`, `min_p`, `typical_p`, `tfs_z`, `max_tokens`, `seed`, `stop`, repeat/presence/frequency penalties, mirostat, `response_format`), and meta/orchestration (`system`, `template`, per-model `rpm`/`tpm` caps). New `GET/PUT/DELETE /admin/model-config` + `GET /admin/model-configs` endpoints back an "Advanced Settings" modal wired into Models, GPU Nodes, and Model Advisor. Inference-time defaults are injected into outgoing requests only where the client didn't already specify them (works across all runtimes via `/v1/*`); load-time/engine params apply only on Ollama-native `/api/*` requests and are visibly disabled in the UI for models resident on vLLM/TGI/llama.cpp nodes, since those runtimes take such settings only as launch-time flags, not per-request. `flash_attention`, `offload_kv_cache_to_gpu`, `tensor_parallelism`, and `logit_bias` are stored/exposed but not yet injected anywhere (no per-request hook exists in any supported runtime today).
  > Superseded in 0.15.1 above: this profile shape and field list were reworked (keyed by `(model, node)`, and the field list corrected against each runtime's real current API) the same day.
- **Requests page filters**: added `node`, `status` (success/client_error/server_error category), and `until` (upper time bound) filters to the server-side `/admin/audit` endpoint and Requests page filter toolbar, alongside the existing model/key/cloud/since filters.
- **Dismiss button on the cloud-spend warning banner** - closes for the current session instead of staying pinned until the underlying spend drops below the threshold.
- **`uninstall.sh`**: removes the binary, the systemd service (if installed), and a background (nohup) instance; prompts before deleting `config.yaml`/`marbor.db` (kept by default, including in non-interactive/piped runs - `KEEP_DB=0`/`KEEP_CONFIG=0` to remove without prompting).
- **`install.sh` post-install health checks**: after starting (either mode), validates `config.yaml` (`-validate`), confirms the proxy/admin/metrics ports are actually responding (not just that the process exists), and reports reachability of configured backend nodes.
- **`install.sh` upgrade reporting**: prints old → new version on reinstall/upgrade via the binary's own `-version` flag; warns when the binary on disk was upgraded but the running background process hasn't been restarted yet.
- **`install.sh` idempotent background start**: re-running the installer while a `nohup`-started instance is already running no longer starts a competing second process - it detects the existing one (via a new `marbor.pid`) and re-verifies its health instead. A stale pidfile (process no longer running, e.g. after a crash) is cleaned up and a fresh instance starts normally.

### Changed
- **`SERVICE=1` is now a generic service-mode abstraction**, not systemd-specific: it dispatches through a `detect_service_manager` step (systemd on Linux today; launchd on macOS is a planned but not-yet-implemented backend) and falls back to the existing background mode with a clear message on any host without a supported service manager, instead of only degrading silently on non-systemd Linux.
- **`install.sh` download failures** now print an actionable message (connectivity/firewall/no-release-yet) instead of a bare `curl`/`wget` error trace.

## [0.14.3] - 2026-07-06

### Changed
- **License: relicensed from MIT to Apache-2.0** (adds an explicit patent grant). Added a `NOTICE` file; every license reference (JSON-LD, `llms.txt`, README badge, site, docs footers) updated to match.
- Website, README, and `llms.txt` repositioned around the **self-hosted, multi-runtime inference control plane** (Ollama, vLLM, TGI, llama.cpp). Visible FAQ reconciled with the JSON-LD `FAQPage`; sitemap refreshed.
- `ROADMAP.md` rewritten as a technical, single-instance roadmap organized around the router-moat progression; pricing/commercial detail moved out of the public repo.
- Removed the internal `docs/design/` doc from the repository.

### Fixed
- **Warmup**: warming 2+ models on one node raced concurrent cold-load goroutines, and headroom accounting didn't see in-flight (not-yet-polled) loads, so a second model would falsely be evicted or never load.
- **Scheduler**: `fireSchedule` logged "fired" success even when the target node was missing or a warmup/unload schedule had zero models, making broken schedules indistinguishable from working ones. Now validates node existence + non-empty model list at creation/patch time (400), and logs skips explicitly.
- **Scheduler timezone mismatch**: schedules never fired for operators outside UTC - the scheduler evaluated `HH:MM` against the server/container clock (typically UTC in Docker), while the admin UI's time input implicitly meant the operator's local time. Added a persisted, configurable timezone; the scheduler and predictive prewarmer now evaluate against it, and the Warmup page shows a live server clock so operators can see exactly what time schedules are evaluated against.
- **HuggingFace model pull returning Bad Gateway**: the pull proxy waits for the whole download before responding; a hardcoded 5-minute client timeout killed any pull over 5 minutes (routine for multi-GB HF files). Raised to 2 hours.
- **Pinned models could still be unloaded**: the pin check only guarded auto-eviction, never the manual/scheduled unload path. Now blocked (409) on both, and the GPU Nodes page disables the unload button for pinned models.
- **Duplicate node registration**: nodes loaded from config.yaml, the DB store, and Docker discovery were merged with no URL-based dedup - only by exact name - so the same physical node could end up registered twice under different names, splitting its usage/eviction accounting.
- **`install.sh` self-discovery**: the port-8080 probe treated any 200 response on `/health` as a foreign llama.cpp node, so it discovered marbor's own admin API as a separate node; it now recognizes and skips itself via the unique `proxy_port` field, and skips the host's own IP during the subnet scan.
- **Sidebar/Dashboard version mismatch**: the sidebar showed a stale build-time version while the Dashboard showed the live one; the sidebar now fetches `/health` too.
- **Demo site server clock stuck on "Loading server time…"**: the static GitHub Pages demo (no backend) called the real `/admin/system-info` endpoint, which doesn't exist there, and swallowed the failure. It now returns mock system info like every other demo endpoint.

### Added
- **`SERVICE=1` install mode**: `install.sh` can now set up a systemd unit (`Restart=on-failure`) so marbor persists across reboots, in addition to the existing install-only and install+probe+run modes.
- **DCO (Developer Certificate of Origin) sign-off requirement** - CONTRIBUTING guidance, a PR-template checkbox, and a CI check.

## [0.14.2] - 2026-07-03

### Fixed
- Resolved 7 security, concurrency, performance, and UI issues surfaced by a codebase audit.
- Website polish: custom thin scrollbars on code/table blocks, code-block copy-button layout, mobile step layout, and split installer options with separate copy buttons.

## [0.14.1] - 2026-07-02

### Fixed
- GitHub Pages deploy pipeline - Actions workflow mode, tag-push handling, version derived from the git tag.
- `install.sh` subnet scan is now gated behind `PROBE=1`; `START=1` alone writes a localhost config without scanning.

## [0.14.0] - 2026-07-02

### Added
- **Persistent warm state** - the warm-model map is persisted and reconciled against live `/api/ps` on startup, so routing intelligence survives restarts.
- **`marbor bench`** - reproducible cold-vs-warm TTFT measured through the marbor proxy.
- **Weighted placement scoring** - multi-factor routing (warm / free VRAM / queue depth / health / success) with model pinning and post-failure node cooldown.
- **Predictive prewarming** - an in-memory transition ring buffer plus time-of-day patterns warm the next likely model before it is requested; accuracy metrics included.
- **Manual + scheduled model unload** - `POST /admin/nodes/{name}/unload` and a new "unload" schedule action.
- Installer subnet probing detects Ollama, vLLM, TGI, and llama.cpp nodes; optional Prometheus + Grafana provisioning.

### Changed
- Router decomposed into `placement` / `health` / `queue` behind interfaces (pure refactor, no behavior change).
- Go toolchain upgraded to 1.25.11 for standard-library security patches.

## [0.13.1] - 2026-07-02

### Fixed
- Admin login is now rate-limited per client IP to prevent brute-force.
- Hourly analytics are backfilled from SQLite on startup, so the dashboard shows continuous history after a restart.
- Docker auto-discovery uses the container's own network IP.

## [0.13.0] - 2026-07-01

### Added
- **Model lifecycle / LRU eviction** - eviction primitives plus operator-pinned never-evict models (B1), headroom-gated auto-eviction on the load path (B2), and a pinned-models control in the Warmup page (B3).

### Fixed
- Affinity deletions guarded under a write lock; `recover()` added to background goroutines.
- Warmup/schedule UI: edit/pause split, styling, and decluttering.

## [0.12.2] - 2026-07-01

### Added
- Scheduled warmup and a per-node warmup dashboard (W2 + W3).

## [0.12.1] - 2026-07-01

### Added
- Real per-node warmup with a keep-alive guard and a model-residency metric (W1).

### Changed
- Session affinity is gated behind `routing.session_affinity`; honest peer-monitor reporting; Anthropic overflow returns 501 where unsupported.
- Added the LIMITATIONS docs page; removed the unshipped SPCS section from the website.

### Security
- Block link-local / cloud-metadata node URLs (SSRF protection).
- Auth on by default, stronger key entropy, database file mode `0600`.

## [0.12.0] - 2026-07-01

### Added
- **Enterprise multi-user auth** - separate admin/user login paths, a user portal, and user management (approve, soft-delete, reset password).

### Security
- bcrypt-hashed passwords, DB-backed admin auth, and SQLite schema hardening.

## [0.11.1] - 2026-06-25

### Added
- Username/password login with sessions and change-password; installed-models view in the dashboard; API key name shown in the request log.

### Changed
- Auth on by default.

## [0.11.0] - 2026-06-25

### Added
- **SQLite persistence for all operational state.**
- **Node runtime auto-detection** (Ollama / vLLM / TGI / llama.cpp) with runtime badges in the UI.
- `vram_used_bytes` in the model catalog.

### Fixed
- Unique request IDs (fallback prevents empty/duplicate IDs); admin token validated before login, with a route guard.

## [0.10.0] - 2026-06-23

### Added
- **Multi-backend runtime support** - nodes now declare a `runtime:` field (values: `ollama`, `vllm`, `tgi`, `llamacpp`; default: `ollama`). The router is now runtime-agnostic via a `RuntimeProbe` interface in the new `internal/runtime` package.
- **RuntimeProbe interface** (`internal/runtime`) - defines `ListModels()` and `HealthCheck()` per backend. Implementations: Ollama (`GET /api/ps`), vLLM (`GET /health` + `GET /v1/models`), TGI (`GET /health` + `GET /info`), llama.cpp (`GET /health` + `GET /v1/models`).
- **Path-aware routing** - `/api/*` requests route exclusively to Ollama nodes. `/v1/*` requests route to any runtime. Non-Ollama backends that receive `/api/*` requests are skipped, not errored.
- **`runtime` field in admin API** - `GET /admin/nodes` now includes `runtime` on each node response.

### Changed
- Warmup (keepalive pings) skips non-Ollama nodes. Keep-alive is an Ollama-native feature; vLLM/TGI/llama.cpp do not expose it.
- VRAM reporting for vLLM, TGI, and llama.cpp nodes is not available via API. Use the existing `vram_total_mb` config field to declare capacity for these nodes. The dashboard labels the source as `declared`.

## [0.9.1] - 2026-06-20

### Changed
- Dashboard sidebar is now a slide-in drawer on mobile with a hamburger toggle and backdrop overlay. Desktop behavior unchanged.
- Main content area has correct top padding on mobile to clear the hamburger button.
- Fixed non-responsive `grid-cols-2/3` layouts in API Keys edit modal, GPU Nodes stat panel, and Routing rule form - all now collapse to single column on small screens.

## [0.9.0] - 2026-06-19

### Added
- **Model warmup** - proactive `keep_alive` pings on a configurable schedule keep loaded models hot in VRAM. Eliminates cold starts on first request after idle.
- **SIGHUP full config reload** - `kill -HUP <pid>` diffs the new config against the running state: adds new nodes/cloud providers/auth keys, removes deleted ones, leaves unchanged ones running. Zero-downtime reconfiguration without a process restart.
- **Structured JSON logging** via `log/slog`. `--log-format json` outputs one JSON object per line. Default remains plain text. Legacy `log.Printf` calls redirect through the same handler.
- **Node drain** - `POST /admin/nodes/{name}/drain` marks a node so the router skips it for new requests while in-flight requests complete. `DELETE` to undrain. Enables zero-downtime GPU maintenance. Dashboard shows an amber DRAINING badge.
- **Per-node request counter** - `requests_total` tracked atomically per node, exposed in `GET /admin/nodes`.
- **X-Request-ID forwarded upstream** - the proxy now sets `X-Request-ID` on requests forwarded to Ollama nodes for end-to-end log correlation.
- **Retry-After + quota reset headers** on 429 responses - clients see when to retry.
- **`POST /admin/v1/config/reload`** - HTTP-triggered config reload for container environments that cannot send SIGHUP.
- **`GET /admin/v1/config`** - returns the current live config (secrets masked). Settings page shows a Reload button.
- **`PATCH /admin/nodes/{name}`** - runtime metadata overrides: `vram_total_mb`, `gpu_model`.
- **`PATCH /admin/keys/{name}`** (`PatchKey`) - update rate limit, daily/monthly quotas, and model allow-list at runtime without rotating the key. Counters preserved.
- Queue depth metric card on the dashboard.

## [0.8.2] - 2026-06-19

### Added
- **Request queue** - instead of returning 503 when all nodes are at connection capacity, requests queue and retry as slots free. Configurable `routing.queue_max_depth` and `routing.queue_timeout_ms`.

## [0.8.1] - 2026-06-18

### Fixed
- Quota counters no longer drift when multiple requests race the monthly reset window.
- `nvidia-smi` is polled at the configured `routing.poll_interval_ms` interval, not on every health check.
- Docker auto-discovery now works on Windows (path normalization for Docker Desktop socket).

## [0.8.0] - 2026-06-18

### Added
- **VRAM-aware cold routing** - when no node has a model warm, the router places the request on the node with the most free VRAM rather than using least-connections. Nodes with unknown capacity or overcommitted VRAM are skipped; falls back to least-connections when all nodes are at capacity.

## [0.7.0] - 2026-06-18

### Added
- **Active/active HA** - each instance polls a configurable peer list (`ha.peers`) via `/health`. `GET /admin/ha/peers` returns cluster status. Run two instances behind any TCP load balancer; the proxy SPOF is eliminated.

## [0.6.0] - 2026-06-17

### Added
- **KV-cache / context affinity** - sticky session routing via `X-Session-ID` request header. Same conversation ID routes to the same node, keeping the KV cache warm for multi-turn inference. TTL-based eviction; falls back gracefully on node failure.
- **Warm-vs-cold benchmark** - `make bench` runs a reproducible first-token latency comparison (warm model in VRAM vs cold load). Numbers are measured on real hardware. See `bench/`.
- Security regression tests for five invariants: constant-time admin token compare, auth fails closed when enabled, cloud off by default when unconfigured, request body cap enforced, upstream host comes only from config.

## [0.5.0] - 2026-06-17

### Security
- Config file is now written `0600` (owner-only) and an existing file is chmod-ed back to `0600` on every save. It previously used `0644` and re-widened on every dashboard "Save Settings", exposing all API keys, the admin token, and cloud-provider keys in plaintext to any local user.
- The admin server no longer falls back to the literal token `admin` when no `admin_token` is set. It generates a `crypto/rand` token and logs it once at startup, closing a LAN-takeover path (guessable token + all-interfaces admin bind).
- `GET /admin/keys` no longer returns proxy keys in plaintext. It returns a masked prefix; the full key is shown only once at creation. The dashboard's reveal/copy-from-list controls were removed accordingly.
- Cloud-fallback now uses a dedicated transport with `ResponseHeaderTimeout` (from `routing.upstream_timeout_ms`) instead of `http.DefaultTransport`, so a hung provider can no longer leak goroutines/connections. No overall client timeout is set, so streaming is preserved.

### Added
- `routing.allow_management_endpoints` (default `false`).

### Changed
- **Behavior change:** the proxy now rejects Ollama model-management endpoints (`/api/delete`, `/api/pull`, `/api/push`, `/api/create`, `/api/copy`, `/api/blobs`) with `403` by default. Any authenticated key could previously delete or pull models on a backend node. Set `routing.allow_management_endpoints: true` to restore the old behavior for single-tenant deployments. Inference and read-only inventory paths (`/api/generate`, `/api/chat`, `/api/tags`, `/v1/*`, ...) are unaffected.

## [0.4.0] - 2026-06-16

### Added
- Cluster-wide remote-node VRAM telemetry. Each node reports a VRAM source (`nvidia`, `declared`, `api`, or `none`) so the dashboard is honest about where the number came from: `nvidia-smi` is read only for local nodes, remote nodes fall back to a declared `vram_total_mb` (new optional per-node config field) or the VRAM derived from Ollama's `/api/ps` (`size_vram`), and show `none` when nothing is known.

### Fixed
- API key expiry is now enforced. A key past `expires_at` (date `2006-01-02`, valid through end of day, or RFC3339) is rejected before rate-limiting; unparseable values are treated as non-expiring.
- Allow-list rejections (`403`) no longer consume a key's rate-limit/quota budget - the token is refunded, so a blocked model never exhausts a key into a `429`.
- Cloud overflow with `stream: false` now returns a single Ollama JSON object (not an NDJSON stream), matching native Ollama semantics.
- Several concurrency and correctness fixes: double connection-decrement on pre-byte proxy errors, two data races (`handleSummary`/`handleKeys` reading node and key state without the lock), a quota check-then-increment TOCTOU, and SSE response truncation on the cloud-translation path.
- Prometheus `model` label is now bounded (cap 256 distinct values, overflow folded to `other`, empty to `unknown`) to prevent client-controlled label cardinality blowup.

## [0.3.0] - 2026-06-14

### Added
- Per-key usage and hard quotas (`daily_limit` / `monthly_limit`) that persist across restarts via an atomic JSON state file (`auth.state_path`, default `usage-state.json`, `-` to disable). A restart no longer resets quotas or usage.
- Per-key model allow-lists enforced at the proxy: requests for a model outside a key's list return `403`.
- `GET /v1/models` returns an aggregated OpenAI-schema list of models across all healthy nodes.
- Cloud-overflow response translation: OpenAI/Anthropic responses are translated back to Ollama's NDJSON shape on the native (`/api/*`) path, including SSE-to-NDJSON streaming.
- Upstream retry/failover: a node that fails before sending any bytes is retried on an alternate node (`routing.max_retries`), then cloud, then 502 - never violates streaming.
- `routing.upstream_timeout_ms`: bounds the wait for upstream response headers (header phase only, not the stream body).
- CLI flags: `--version`, `--config <path>`, `--validate` (validate config and exit), and `--help`.
- 4 new Prometheus metrics (11 total): `marbor_retries_total`, `marbor_cloud_fallbacks_total`, `marbor_quota_rejections_total`, `marbor_panics_total`.
- Structured JSON access log on stdout, one line per request (`proxy.access_log`, default on). Logs key name (never the value), model, node, status, latency, request id.
- Configurable admin listener: `admin.bind_address` (default `:8080`; set `127.0.0.1:8080` to keep the dashboard off the network) and `admin.cors_origin`.
- `make demo`: spins up mock Ollama servers in-process, sends real traffic, shows a populated dashboard in <60s with no Ollama install required.
- VRAM fit indicator: GPU Nodes page shows green/yellow/red badges for each downloaded model per node based on available VRAM.
- Tokens and tok/s columns in live request log.
- `docs/PRODUCTION.md`, `docs/INTEGRATIONS.md`, and an honest "How It Compares" table in the README.

### Changed
- Zero-config first run promoted to top-level Quick Start path in README.
- Savings reference rate is now configurable via `savings.reference_cost_per_1k` (see v0.2.1).

### Security
- Admin token comparison is now constant-time (`crypto/subtle.ConstantTimeCompare`) to remove a timing side channel.
- Request bodies are capped at 32 MiB (`413` over the limit) to bound a memory-exhaustion DoS vector.
- Admin API no longer sends a wildcard `Access-Control-Allow-Origin: *`. CORS is off by default (same-origin only) and opt-in via `admin.cors_origin`, closing a CSRF/exfil surface on the mutating admin API.
- `ReadHeaderTimeout` set on the proxy, admin, and metrics servers (metrics server previously had no timeouts) to close a Slowloris DoS vector.
- Upstream/cloud error responses no longer leak the raw error (hostnames, ports, dial/TLS details) to the client; they return a generic `502 upstream unavailable` and log the detail server-side.
- Panic-recovery middleware returns a clean `500` and increments `marbor_panics_total` instead of dropping the connection. It re-raises `http.ErrAbortHandler` so streaming aborts are unaffected (R2).
- Usage-state persistence now `fsync`s the temp file and parent directory before the atomic rename, closing a power-loss window that could reset quota counters.
- Config validation rejects non-`http(s)` node and cloud `base_url` values at boot instead of failing per-request later.

### Performance
- Hourly analytics buckets are pruned to a 48h window, bounding memory growth on long-lived processes.

### CI
- CI now runs `go vet`, a `gofmt` gate, `go test -race`, and `govulncheck`.

---

## [0.2.1] - 2026-06-11

### Added
- Zero-config first run: `./marbor` auto-detects localhost:11434, generates crypto/rand API keys, starts on :11435, prints one curl example
- `savings.reference_cost_per_1k` config field: controls the $/1K token rate used for saved_usd calculation (default $0.002)
- `docs/SAVINGS-MATH.md`: formula, null semantics, restart reset, and known limitations documented
- `CloudModel` field in audit log entries: records cloud model used when default_model rewrites the request

### Fixed
- Savings math uses real parsed token counts (eval_count + prompt_eval_count for Ollama, usage.total_tokens for OpenAI); no more hardcoded 500-token estimate
- saved_usd and cloud_spent_usd return JSON null (shows "-" in UI) when requests exist but no token data was parseable
- Mid-stream abort (upstream node death) now records status=aborted in metrics, admin log, and audit instead of silently vanishing
- Cloud model rewriting visible in request log as "original -> cloud_model" for observability

### Tests
- Streaming integration tests: unbuffered delivery (R2 verified), token tracking from NDJSON tail, mid-stream node death records aborted, SSE passthrough
- Admin savings tests: custom reference rate flows to handleSavings and hourly analytics buckets
- Config tests: default rate applied via Validate(), custom rate loaded from YAML
- Proxy tests: cloud model mapping visible in live request log and audit entry

## [0.2.0] - 2026-05-23

### Added
- Analytics dashboard: 24-hour area chart (local vs cloud), cost savings stats, per-model breakdown table
- Model catalog: cross-node VRAM view with warm status badges and node chips, searchable
- Request log page: live feed with 3-second polling, filter by key/model/status, status indicators
- Rate limit response headers: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset on every response
- Webhook notifications: node_down and node_up events with HMAC-SHA256 request signatures
- Audit logging: append-only JSON-lines file with crypto/rand request IDs
- Docker auto-discovery: scans containers with OLLAMA_HOST env or ollama/ollama image, deduplicates by URL
- Grafana dashboard: 13-panel JSON at grafana/marbor.json, one-click Prometheus import
- Admin API versioning: all endpoints available at both /admin/ and /admin/v1/
- Cloud provider config: OpenAI and Anthropic fallback routing with per-request cost tracking
- Cloud savings widget on dashboard: shows estimated savings vs pure cloud this month
- GET /health endpoint: unauthenticated, returns 200 when proxy is ready
- X-Request-ID header on all proxy responses

### Fixed
- Auth token validation uses exact match (was substring - security vulnerability)
- Port derivation uses URL parsing (was hardcoded array index arithmetic)
- VRAM calculation uses real Ollama /api/ps data (was fake arithmetic)
- Random CPU/temperature metrics replaced with real nvidia-smi data
- Mutex protection added to auth rate limit maps (was a race condition)
- API key creation timestamp shows real value (was hardcoded date)
- UI auth prompt asks for token (was silently falling back to "admin")

## [0.1.0] - 2026-05-16

### Added
- Initial release
- GPU-aware warm-first routing across Ollama nodes
- Bearer token authentication with per-key rate limiting
- Admin dashboard (React + TypeScript, embedded in binary)
- Prometheus metrics on :9090
- Docker Compose example
- Single Go binary, zero runtime dependencies