"` / `"error ..."` outcome entry to a local SQLite database with a per-entry HMAC-SHA256 chain (with secret parameters redacted). Fail-closed: if the audit log cannot be verified or written, proxxx refuses the mutation with exit code 8 before sending any request. |
| **Cryptographic chain verification** | `proxxx audit verify` walks every entry, recomputes the HMAC chain from the keyed root, and reports the first broken link. CI-friendly: exits 0 on pass, 1 on violation — wire it into your compliance pipeline. |
| **Export for SIEM** | `proxxx audit export --export-format json` or `--export-format csv` — pipe into Splunk, Elastic, Wazuh, or any log aggregator without an agent. |
| **Supply-chain** | Every release ships: SHA-256 sidecar, sigstore keyless cosign signature (pinned to the exact workflow path, offline-verifiable, transparency-log proof embedded), and a CycloneDX SBOM from `Cargo.lock`. Audit with `cosign verify-blob` + `grype` / `trivy`. |
| **Self-diagnostic** | `proxxx doctor` validates config, cluster connectivity and credentials via the authenticated PVE client (`/version` + `/nodes`, PVE 8+/9 compatible), Telegram HITL (`getMe` status + `ok == true` with URL/token scrubbing), PBS (`/version` via `PbsClient`), SSH key, and audit-chain integrity (recomputing the HMAC chain, not just that the DB opens) in one pass. Exits 0 if all critical checks pass. |
| **Secrets hygiene** | All secret values live in `SecretString` — `Debug` prints `[REDACTED]` (not even the length), no `Display` and no `Serialize` so interpolating or serialising one is a *compile error*, and the heap bytes are zeroized on drop. HMAC and audit keys are stored at 0600 paths; proxxx refuses to use a key file with looser permissions. |
> **Note:** proxxx is a management tool, not a compliance product. `proxxx audit verify` provides integrity assurance for the local mutation log; it does not replace a SIEM or a formal audit trail required by a certification body. Use it as one control layer in a broader NIS2 / ISO 27001 implementation.
---
## Install
Pre-built binaries for **macOS Apple Silicon**, **Linux x86_64-musl**, and **Linux aarch64-musl** (Pi 4/5, Ampere, Graviton, Oracle Free Tier) are attached to each [tagged release](https://github.com/fabriziosalmi/proxxx/releases). All Linux artefacts are statically linked — no glibc, drops onto Alpine through RHEL.
### Homebrew (macOS & Linux)
```bash
brew install fabriziosalmi/proxxx/proxxx
```
The fastest path on a workstation. For a Proxmox VE node (Debian) use the [`.deb`](#debian--ubuntu--proxmox-ve--deb) below; for air-gapped installs or full supply-chain verification, download + verify the trio manually:
```bash
TARGET=x86_64-unknown-linux-musl # or aarch64-apple-darwin
VERSION=$(gh release view --repo fabriziosalmi/proxxx --json tagName -q .tagName | sed 's/^v//')
gh release download v${VERSION} --repo fabriziosalmi/proxxx \
--pattern "*-${TARGET}.tar.gz" \
--pattern "*-${TARGET}.tar.gz.sha256" \
--pattern "*-${TARGET}.tar.gz.cosign.bundle"
# 1. Checksum
shasum -a 256 -c proxxx-${VERSION}-${TARGET}.tar.gz.sha256
# 2. Sigstore keyless signature (offline; cert pinned to release.yml)
cosign verify-blob \
--bundle proxxx-${VERSION}-${TARGET}.tar.gz.cosign.bundle \
--certificate-identity-regexp 'https://github.com/fabriziosalmi/proxxx/.github/workflows/release.yml@.*' \
--certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
proxxx-${VERSION}-${TARGET}.tar.gz
# 3. (optional) Audit the CycloneDX SBOM
gh release download v${VERSION} --repo fabriziosalmi/proxxx \
--pattern "*.cdx.json" --pattern "*.cdx.json.sha256" --pattern "*.cdx.json.cosign.bundle"
shasum -a 256 -c proxxx-${VERSION}.cdx.json.sha256
cosign verify-blob \
--bundle proxxx-${VERSION}.cdx.json.cosign.bundle \
--certificate-identity-regexp 'https://github.com/fabriziosalmi/proxxx/.github/workflows/release.yml@.*' \
--certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
proxxx-${VERSION}.cdx.json # releases after v0.14.0 sign the SBOM too
grype sbom:proxxx-${VERSION}.cdx.json # or trivy / cyclonedx-cli
tar xzf proxxx-${VERSION}-${TARGET}.tar.gz
./proxxx-${VERSION}-${TARGET}/proxxx --version # names the tagged commit (releases after v0.14.0)
```
If you only want the binary fast (no verification), skip steps 2–3 and run just `shasum -a 256 -c …` from the snippet above. Production deployments should run all three — see the [Production checklist](https://fabriziosalmi.github.io/proxxx/guide/production-checklist).
Or build from source (needs Rust 1.95+):
```bash
git clone https://github.com/fabriziosalmi/proxxx.git
cd proxxx && cargo build --release
./target/release/proxxx --version
```
The Linux musl artifact is statically linked — runs on every distro from RHEL 6 to Alpine 3.x without GLIBC drama.
### Debian / Ubuntu / Proxmox VE — `.deb`
Each release also ships a `.deb` for **amd64** and **arm64**, signed with the same sigstore bundle. The binary is static-musl, so the package declares **no runtime dependencies** — drop it straight onto a Proxmox VE node (which is Debian):
```bash
VERSION=$(gh release view --repo fabriziosalmi/proxxx --json tagName -q .tagName | sed 's/^v//')
gh release download v${VERSION} --repo fabriziosalmi/proxxx \
--pattern "proxxx_${VERSION}-1_amd64.deb" # or _arm64.deb
sudo apt install ./proxxx_${VERSION}-1_amd64.deb # or: sudo dpkg -i proxxx_*.deb
proxxx --version
```
Verify its signature exactly like the tarball — `cosign verify-blob --bundle proxxx_${VERSION}-1_amd64.deb.cosign.bundle …` with the same `--certificate-identity-regexp` / `--certificate-oidc-issuer` as above.
## Quick start
```bash
proxxx init --interactive # 5-step wizard: prompts for URL, auth, TLS, optional
# SSH + Telegram, validates each input against the
# live cluster before write. Recommended for first
# run — wrong field caught here, never lands in TOML.
proxxx init # non-interactive variant: writes a commented
# starter config.toml; refuses to overwrite — pass
# --force if you mean it. Edit url / user /
# token_id / token_secret manually after.
proxxx ls nodes # validates the connection.
proxxx # TUI (no args). Press ? for the keymap; the
# bottom-row footer shows contextual binds always.
proxxx --help # full subcommand list.
proxxx version --json # build + capability metadata.
```
The starter `config.toml` carries inline comments for every secret-resolution path (CLI flag → env var → inline TOML → 0600 file → macOS Keychain). Optional sections — HITL via Telegram, SSH layer, PBS, alerts, policies — are commented out so the API-only operator doesn't have to delete anything.
A delete of a running guest is refused by the pre-flight before any request is sent; a stop that a policy puts behind human approval does not run when no approval channel is configured. A terminal recording of the real binary against a lab Proxmox VE 9.1 cluster, unedited.
## Daily-driver TUI
Run with no arguments. Vim keys, fuzzy search across the cluster (`/`), command palette (`:`), quick-open palette (`Ctrl+K`). 18 views over the same Elm-pattern reducer:
| `1` Dashboard | `2` Nodes | `3` Guests | `4` Storage |
| :---: | :---: | :---: | :---: |
| `H` Heatmap | `B` Backup board | `G` Config grep | `Q` Operation queue |
| `T` Audit timeline | `Z` Snapshot tree | `D` Drift compare | `W` Hardware passthrough |
Plus a read-only **fleet view** (`proxxx fleet`) that aggregates every configured profile into one screen — `↑↓` select cluster, `Tab` toggle guest pane, `Enter` drill into a cluster, `q` quit.
Multi-select + bulk ops with pre-flight risk preview. Operation queue with dry-run, diff preview, replay-as-script export (proxxx CLI / pvesh / curl / Ansible), and HITL approval gate (Telegram, policy-driven).
The terminal is restored on every exit path — happy, `?` early-return, panic. RAII `TerminalGuard` plus a flight-recorder panic hook installed in `main()` before the runtime starts.
## Pipeline-friendly CLI
```bash
# Read
proxxx ls guests --format json | jq '.[] | select(.status == "running") | .vmid'
proxxx ha preview --node pve1 # failover what-if
proxxx hw conflicts --node pve1 # PCI passthrough audit
proxxx perms root@pam --node pve1 # effective permissions
```
```bash
# Write — every destructive op routes through the pre-flight risk gate
proxxx start 100 101 102
proxxx delete 100 --yes
proxxx migrate 100 pve2 --yes
proxxx snapshot create 100 --name pre-upgrade
proxxx disk move 100 --disk scsi0 --storage ceph-rbd --yes
proxxx patch apply --reboot=auto --dry-run
```
```bash
# Console handoff
proxxx ssh 100 # interactive ssh into guest (system ssh +
# QGA / lxc-interfaces auto-discovery; falls
# back to [ssh.guests."100"] when explicit)
proxxx serial 100 --node pve1 # raw termproxy WebSocket
proxxx spice 100 --node pve1 # writes 0600 .vv, launches remote-viewer
proxxx novnc 100 --node pve1 # opens browser to web UI's noVNC
```
```bash
# GitOps loop — export, diff, apply with safety on top
proxxx state export > state.toml # snapshot cluster state
git add state.toml && git commit -m snapshot # version it
$EDITOR state.toml # declare intent
proxxx state diff state.toml # preview drift, exit 2 if any
proxxx state apply state.toml --dry-run # rehearse, never mutates
proxxx state apply state.toml --prune # pre-flight refuses Severe (exit 6)
proxxx state apply state.toml --prune --interactive # per-Severe [y/N] prompt
# Continuous reconciliation — the GitOps controller (Layers 1–3)
proxxx reconcile run --source git@host:cluster.git # CI drift gate, exit 2 on drift
proxxx reconcile converge --source git@host:cluster.git --dry-run # rehearse the converge
proxxx reconcile converge --source git@host:cluster.git --prune # push-mode: converge prod
# Unmanned: set `auto_converge = true` under [profiles.X.reconcile] + run `proxxx daemon serve`
```
The reconcile loop, end-to-end against a live Proxmox VE 9.1 cluster: drift detected → converged (safe by default) → back in sync. Every output line is verbatim.
```bash
# Cross-cluster fanout + cluster digest
proxxx find 100 # which profile owns VMID 100?
proxxx ls guests --all-profiles # every guest across every cluster
proxxx fleet # read-only TUI: ALL profiles in one screen
proxxx describe --output llm-context # paste at top of an LLM chat
proxxx accounting --group-by pool --timeframe month # CPU-hours / GiB·h / net-GiB
proxxx heatmap # per-node API RTT, color-bucketed
proxxx anomaly --threshold 3 # z-score outliers on CPU + mem%
proxxx logs tail --service pveproxy --since "1 hour ago" # cross-node journalctl
```
```bash
# Incident lockdown + upgrade pre-flight
proxxx incident freeze --reason "rotating token" --ttl 4h # halt every mutation cluster-wide
proxxx incident status --output json
proxxx incident thaw --reason "rotation complete"
proxxx upgrade-check --target 9.x # exits 1 on any block-severity finding
proxxx explain freeze-refusal # bundled error knowledge base
```
Freeze the fleet and every mutation is refused — before it touches Proxmox — until you thaw. Output verbatim from a live Proxmox VE 9.1 cluster.
```bash
# Console handoff + session recording
proxxx ssh 100 # interactive ssh into guest (auto-discovery)
proxxx serial 100 --node pve1 --record # raw termproxy WebSocket + asciinema cast
proxxx play-cast ~/.../sessions/-100-serial.cast --speed 2
proxxx spice 100 --node pve1 # writes 0600 .vv, launches remote-viewer
proxxx novnc 100 --node pve1 # opens browser to web UI's noVNC
```
```bash
# Long-running daemon — alerts + HITL + schedule under ONE process
proxxx daemon serve # every configured component, one SIGTERM
proxxx daemon serve --no-hitl # alerts + schedule only
proxxx schedule add --name nightly-snap --every 1d --cmd "vm snapshot 100 --yes"
# MCP for AI agents (stdio JSON-RPC, HTTP/SSE for streaming)
proxxx mcp serve # stdio + interleaved server-sent notifications
proxxx mcp serve-http --bind 0.0.0.0:8080 # HTTP/SSE transport
proxxx mcp tools --checksum # registry SHA-256 for audit pinning
```
Any MCP client (Claude, Cursor) gets 25 typed, audited tools over your cluster. Output verbatim from a live Proxmox VE 9.1 cluster.
Exit codes are stable contract — see [`docs/reference/exit-codes.md`](docs/reference/exit-codes.md) for the full table.
## Configuration
Default location follows the `directories` project-dirs convention:
| Platform | Path |
| :--- | :--- |
| Linux | `~/.config/proxxx/config.toml` |
| macOS | `~/Library/Application Support/dev.proxxx.proxxx/config.toml` |
Secrets resolve in order: CLI flag → `PROXXX_TOKEN_SECRET` env → inline TOML → `token_secret_file` (0600 enforced) → macOS Keychain (macOS only; service `proxxx`, account `/token_secret` then `token_secret`). **Inline beats the file reference** — when moving a secret into a file, delete the inline value or it keeps winning (proxxx warns when both are set). Loaded values live in `SecretString`: redacting `Debug`, no `Display`, no `Serialize`, zeroized on drop.
| Optional section | Unlocks |
| :--- | :--- |
| `[telegram]` | HITL approvals + alert routing |
| `[ssh]` | Patching orchestrator, `proxxx perms`, guest SSH |
| `[ssh.guests.X]` | Per-guest SSH overrides (optional — `proxxx ssh ` auto-discovers via QGA / lxc-interfaces by default; pin only when the agent's off, only loopback/link-local IPs are returned, or you want a stable DNS name) |
| `[pbs]` | PBS browse + restore |
| `[[alerts]]` | Alerting daemon — `node_offline`, `storage_above`, `replication_failing` |
| `[[policies]]` | HITL gating rules — match by tag / vmid / wildcard |
Per-profile, set `read_only = true` to refuse all mutations on that profile client-side (reads unaffected, exit 8) — pair with a `PVEAuditor` PVE token for a server-side lock. See the [configuration guide](docs/guide/configuration.md#read-only-profiles-read_only).
## Quality gate
Eight stages, run as both a pre-commit hook and the CI contract in [`.github/workflows/ci.yml`](.github/workflows/ci.yml).
| Stage | What | Time |
| :---: | :--- | :---: |
| 0 | secret regression scan | <1 s |
| 1 | `cargo fmt --all -- --check` | ~3 s |
| 2 | `cargo clippy --release --all-targets` | 10–60 s |
| 3 | `cargo audit --deny warnings` | 3–5 s |
| 4 | `cargo deny check` (license / banned crates / sources / wildcards) | 2–4 s |
| 5 | `cargo test --release --all-targets` (745 lib tests + 478 integration tests including ~25 proptest properties × 256 cases each) | 10–90 s |
| 6 | `tests/live/test_run.sh` (67 read-only probes against the live cluster) | ~30 s |
| 7 | `tests/live/test_mutation.sh` (34 mutation probes: LXC + cluster-level CRUD across 8 of the 10 state families + QEMU; opt-in QGA via `PROXXX_E2E_QGA_VMID=`) | ~60 s |
End-to-end wall time against a reachable cluster: **~340–480 s**.
```bash
git config core.hooksPath .githooks
chmod +x scripts/gate.sh .githooks/pre-commit .githooks/pre-push
cargo install cargo-audit --locked
cargo install cargo-deny --locked
```
The clippy `[lints.clippy]` block in [`Cargo.toml`](Cargo.toml) denies `unwrap_used`, `expect_used`, `panic`, `todo`, `await_holding_lock` in production code.
## Architecture
Pure Elm-pattern TUI over a typed REST client. The reducer is sync, total, and tested without a runtime.
```
crossterm key tokio::mpsc
user ─────────────────► event::map_key ─► Action
│
▼
app::update(state, action)
│
┌─────────────┴────────────┐
▼ ▼
AppState mutation Option
│
▼
check_hitl → preflight::assess
(Telegram round-trip) (queued operations)
│
▼
ProxmoxGateway / PbsGateway / SshPool
```
| Module | Responsibility |
| :--- | :--- |
| [`src/app.rs`](src/app.rs) | Pure reducer. No I/O, no async. ~70 `Action` variants, ~20 `SideEffect`. |
| [`src/api/`](src/api) | `ProxmoxGateway` trait, typed `ApiError` enum (9 categorical variants), reqwest client with 32 MiB body cap and rate limiter. The `types/` directory holds ~3100 LOC of serde-typed PVE responses split across 16 submodules. |
| [`src/api/error.rs`](src/api/error.rs) | `Unauthorized`, `Forbidden`, `NotFound`, `RateLimited`, `PayloadTooLarge`, `StorageHang`, `Transport`, `Parse`, `Other`. Callers `.downcast_ref()` for differentiated handling. |
| [`src/state/`](src/state) | GitOps loop — model + export + diff + apply + preflight, ten state families (pools, ACL, storage, backup-jobs, firewall-cluster, notifications, HA rules, HA resources, PCI/USB mappings), sorted TOML. |
| [`src/pbs/`](src/pbs) | PBS REST browse + `kill_on_drop(true)` supervision over `proxmox-backup-client restore`. |
| [`src/ssh/`](src/ssh) | `russh`, publickey only, dedicated TOFU `known_hosts` (separate from `~/.ssh/`), per-node connection pool. |
| [`src/app/cache.rs`](src/app/cache.rs) | SQLite-backed time-travel cache, drives `proxxx replay `. |
| [`src/app/preflight.rs`](src/app/preflight.rs) | 11 risk variants with per-op weighting and `--allow-risk` override. |
| [`src/hitl/`](src/hitl) | Real Telegram round-trip via `HitlCoordinator` + a single shared `getUpdates` poller. Deny on 120 s timeout, deny when Telegram unconfigured but a policy matched. |
| [`src/mcp/`](src/mcp) | JSON-RPC server with stdio + Streamable HTTP transports. Compile-time-fixed tool registry (25 tools). Surface SHA-256 pinned via `proxxx mcp tools --checksum`. |
| [`src/util/`](src/util) | `panic_hook` (flight recorder), `terminal_guard` (RAII raw-mode), `shutdown` (SIGTERM / SIGINT for daemons). |
## Documentation
- **VitePress site** — [`docs/`](docs/) and [the live build](https://fabriziosalmi.github.io/proxxx/). Local preview: `cd docs && npm install && npx vitepress dev`.
- [`CHANGELOG.md`](CHANGELOG.md) — what shipped, with the SemVer contract for CLI / JSON / config / MCP registry surfaces.
- [`pre-commit/`](pre-commit/) — four matrices distinguishing *implemented* from *verified end-to-end*:
[`01-feature-coverage.md`](pre-commit/01-feature-coverage.md) ·
[`02-error-handling.md`](pre-commit/02-error-handling.md) ·
[`03-security-invariants.md`](pre-commit/03-security-invariants.md) ·
[`04-resiliency-and-chaos.md`](pre-commit/04-resiliency-and-chaos.md) ·
[`ACCEPTED-RISKS.md`](pre-commit/ACCEPTED-RISKS.md) (AR-1..AR-6 residual risks)
- [`ARCHITECTURE.md`](ARCHITECTURE.md) — one-page module map + data flow + reducer/side-effect bus + process model.
- [`THREAT_MODEL.md`](THREAT_MODEL.md) — attack surfaces, mitigations, accepted risks, verification ladder.
- [`SECURITY.md`](SECURITY.md) — vulnerability reporting policy + scope + hardening snapshot.
- [`CONTRIBUTING.md`](CONTRIBUTING.md) — onboarding, the gate, live-cluster verification format.
- [`.cargo/audit.toml`](.cargo/audit.toml) — supply-chain advisory ignore policy.
- [`deny.toml`](deny.toml) — cargo-deny policy: license whitelist + banned crates + source lock.
## Live cluster harness
[`tests/live/`](tests/live/) drives the release binary against a real PVE cluster — separate from the cargo integration tests in [`tests/`](tests/) which use `wiremock`.
| File | Tracked | Purpose |
| :--- | :---: | :--- |
| [`test_run.sh`](tests/live/test_run.sh) | ✓ | 67 read-only probes covering the full CLI surface; logs to `test_run.log` |
| [`test_mutation.sh`](tests/live/test_mutation.sh) | ✓ | 34 mutation probes with `trap EXIT` cleanup: LXC 9999 (create → start → snapshot → stop → delete), cluster-level CRUD across 8 of the 10 state families (pool / ACL / storage-defs / backup-job / firewall-cluster alias+group+ipset / notifications matcher / HA rules / HA resources — PCI/USB passthrough mappings not yet in the harness), QEMU 9998 from alpine ISO, opt-in QGA round-trips via `PROXXX_E2E_QGA_VMID=` |
| `test_*.log` | — | Generated by the harness |
## Honest non-goals
Design boundaries: proxxx will not ship these. The rate at which the surface grows is limited too: see `CONTRIBUTING.md` → Surface freeze.
- **No GUI.** Proxmox already has a web UI; proxxx is for terminal users who want CLI / TUI / scripting parity.
- **No frame rendering** for graphical SPICE or VNC. proxxx hands off to `remote-viewer` / `virt-viewer` (SPICE) or the system browser (noVNC). It never holds pixel buffers.
- **No re-implementation of Perl algorithms in Rust** where the Perl on the node is the ground truth. `proxxx perms` shells out to `pveum user permissions` over SSH and parses, since the `pve-access-control` evaluator is canonical. The API-side `proxxx access permissions` is also available — same typed tree from `/access/permissions`, no SSH dependency — for the common case where the evaluator's full expansion isn't needed.
- **No new dependencies for trivial things.** Three-line per-platform `Command::new` beats pulling `opener` for a launcher.
- **No multi-cluster _writes_ from one TUI.** The interactive per-cluster TUI is single-profile-per-process by architectural decision (switch with `--profile`); every mutation targets exactly one cluster. The read-only `proxxx fleet` view *does* aggregate every configured profile into one pane — but it stays strictly read-only and drills into a single-profile TUI (`Enter`) to act. The boundary is writable aggregation, not aggregation itself.
- **No Ceph cluster writes.** Operators reach for the `ceph` CLI directly on the node where the kernel module is loaded; proxxx wraps Ceph reads (status, metadata, flags) but not destructive ops (osd add/down, mon create, pool prune).
- **No SDN config writes.** PVE SDN is opt-in cluster config that few clusters enable, and the wire shape changes between PVE versions. Skipped rather than ship a fragile surface.
- **No browser-only auth flows.** U2F/WebAuthn registration and OIDC's redirect-callback dance both need a browser to drive them. proxxx exposes the API-driven primitives (token CRUD, password change, ACL editing) but stays out of `/access/openid/*` and `/access/tfa/u2f` — there's no terminal UX for those that beats the web UI.
- **No snapshot rollback as a destructive trigger.** The snapshot-tree TUI shows a read-only rollback impact preview (what would be discarded + time delta); the actual rollback runs through `qm rollback` / `pct rollback` or the PVE web UI. Read-only inspector views never expose destructive entry points by design.
## License
MIT. Copyright © 2026 Fabrizio Salmi. See [`LICENSE`](LICENSE).
