# ProxMate — Production Deployment Runbook

A step-by-step guide to deploying a production-ready ProxMate on your own
infrastructure: HTTPS, tenant network isolation, SMTP, Keycloak SSO, and all
two-step-authentication methods. Follow it top to bottom.

> **Why HTTPS is non-negotiable here:** passkeys (WebAuthn), `Secure` session
> cookies, and the OIDC SSO redirect all require a secure context. The whole
> deployment is built around **one HTTPS domain** behind a reverse proxy.

---

## 0. Target topology

```mermaid
flowchart LR
  browser([Browser])
  subgraph server [Your server]
    caddy["Caddy reverse proxy<br/>(:443, auto-TLS)"]
    backend["proxmate-backend<br/>127.0.0.1:4000 · Express + SQLite"]
    frontend["proxmate-frontend<br/>127.0.0.1:3000 · Next.js"]
  end
  pve[("Proxmox VE cluster")]

  browser -- HTTPS --> caddy
  caddy -- "/api/*" --> backend
  caddy -- "/*" --> frontend
  backend -- "Proxmox API token" --> pve
```

- **One public domain** (e.g. `proxmate.example.com`). Frontend at `/`, API at `/api`.
  Same-origin ⇒ first-party cookies, passkey RP ID = the domain, SSO callback on the
  same domain. This avoids every cross-origin cookie headache.
- The backend reaches the **Proxmox cluster** over your management network using an
  API token. ProxMate never stores your Proxmox password.

---

## 1. Prerequisites

**On the server that will host ProxMate:**
- [ ] Linux host with Docker Engine + Docker Compose v2 (`docker compose version`).
- [ ] A domain/subdomain (`proxmate.example.com`) with an **A/AAAA record** pointing at the server's public IP.
- [ ] Ports **80 + 443** open to the internet (for Caddy + ACME). The app ports (3000/4000) stay bound to `127.0.0.1`.
- [ ] Outbound reachability to your **Proxmox API** (`https://<pve-host>:8006`) and, if used, your **Keycloak** and **SMTP relay**.

**On the Proxmox side:**
- [ ] A dedicated **API token** for ProxMate (see §2.2). 
- [ ] Decided on your **tenant isolation** model (see §2.1) — ideally a dedicated VLAN/SDN bridge.

**You'll generate/collect along the way:** `ENCRYPTION_KEY`, the Proxmox token id+secret, SMTP creds, and Keycloak client id+secret.

---

## 2. Network, VLAN & tenant isolation (Proxmox side)

ProxMate gives every tenant VM a per-VM Proxmox firewall (`policy_in=DROP`,
mac/ip-filter on, DNS to the gateway allowed, RFC1918 dropped, internet allowed).
**These rules only take effect once the Proxmox *cluster* firewall is enabled** —
ProxMate can do that for you safely from the admin UI (§7.3).

### 2.1 Choose an isolation model

- **Good — shared bridge + per-VM firewall + cluster firewall on.** Works on a
  default `vmbr0`. ProxMate's per-VM rules block tenant-to-tenant traffic once the
  cluster firewall is enforced.
- **Best — a dedicated VLAN or SDN zone for tenant VMs.** Put tenant NICs on an
  isolated VLAN (e.g. tag 50) or a Proxmox **SDN VNet** so tenants are L2-isolated
  from your management network by design.
  - Datacenter ▸ SDN ▸ Zones → add a **VLAN** (or **Simple**) zone; add a **VNet**
    (e.g. `tenants`, VLAN tag 50); **Apply**.
  - Set ProxMate's default bridge to that VNet in admin **Settings ▸ Proxmox tab ▸ VM defaults**
    after first run.
- Keep the **Proxmox management interface (8006) and SSH on a separate VLAN/subnet**
  the tenants can't reach.

### 2.2 Create the ProxMate API token (avoid the #1 pitfall)

Proxmox tokens default to **Privilege Separation ON**, which gives the token *no*
permissions even for root — storage lists come back empty and VM creation 403s.

```bash
# On a Proxmox node (as root). Create a user + token, OR use root@pam.
pveum user token add root@pam proxmate --privsep 0       # privsep OFF
# → copy the displayed token VALUE now; it is shown only once.
```
You'll enter `root@pam!proxmate` as the **Token ID** and that value as the **Secret**
during OOBE (§7).

Whatever token you use must be able to set guest tags (`VM.Config.Options`): ProxMate
tags every guest it manages with the machine it belongs to, and acts only on guests
carrying their own machine's tag. On Proxmox VE 8+ it also needs `SDN.Use` (checked
whenever a guest NIC is attached, plain bridges included) and `SDN.Audit` to list SDN
VNets; the connection test warns when either is missing. A privsep-off token has all
of these.

**Least privilege (recommended).** Instead of a root token, `deploy/proxmox-token.sh` creates a
dedicated `proxmate@pve` user, a `ProxMate` role holding only the privileges ProxMate
uses, and a privilege-separated token `proxmate@pve!proxmate` with the role granted on
`/` to both the user and the token. Copy the script to a Proxmox node, read it, then:

```bash
bash proxmox-token.sh --dry-run     # show the pveum commands without running them
bash proxmox-token.sh               # as root; prints the token secret once
#   --with-passthrough  also allow GPU/PCI passthrough through resource mappings
```

It picks the guest-agent privileges for your release (`VM.Monitor` on Proxmox VE 8,
`VM.GuestAgent.Audit` + `VM.GuestAgent.Unrestricted` + `VM.GuestAgent.FileWrite` on 9). Re-running it updates the
role and ACLs and never re-issues an existing token. The connection test names any
privilege that is still missing; a correctly created token passes it with no warnings
(`VM.Audit` and `Datastore.Audit` are the two that block outright). By default the grant
is on `/`: the cluster firewall, cluster status and cloud-image downloads are checked there
(the pool-scoped mode below keeps only `Sys.Audit` and `Sys.Modify` on `/`).
The role has been run end to end on a Proxmox VE 9.2 cluster: every ProxMate feature
worked through it. On 9.2, `VM.GuestAgent.Unrestricted` already covers guest file writes
(the IDE bootstrap and pushing a file into a VM), so `VM.GuestAgent.FileWrite` is only belt
and braces there. The old `--with-ide` option is still accepted and changes nothing.

**Pool-scoped token (optional).** With the grant on `/`, the boundary between guests is
ProxMate's ownership-tag check, not a Proxmox permission. `--pool <name>` instead grants the
guest privileges on a Proxmox resource pool, so Proxmox itself refuses the token guests outside it:

```bash
bash proxmox-token.sh --dry-run --pool proxmate   # prints the ACL plan and the commands
bash proxmox-token.sh --pool proxmate             # creates the pool if missing
#   --snippet-storage <id>   optional: Datastore.Allocate there, so ProxMate can enable snippets
#   --import-storage <id>    optional: Datastore.Allocate there, so ProxMate can delete imported images
#   --with-cluster-firewall  let ProxMate enable the cluster firewall (see below)
#   --drop-root-grant        remove the earlier grant on / (refused while the pool is empty; --force)
```

| Path | Role | Privileges |
| --- | --- | --- |
| `/pool/<name>` | `ProxMate-Guest` | every `VM.*` privilege above, `Pool.Audit`, `Pool.Allocate` |
| `/nodes` | `ProxMate-Infra` | `Sys.Audit`, `Sys.AccessNetwork` (8.2+, cloud-image downloads) |
| `/storage` | `ProxMate-Infra` | `Datastore.Audit`, `.AllocateSpace`, `.AllocateTemplate` |
| `/storage/<id>`, only for `--snippet-storage` / `--import-storage` | `ProxMate-Storage` | the same plus `Datastore.Allocate` |
| `/sdn/zones` | `ProxMate-Infra` | `SDN.Audit`, `SDN.Use` |
| `/mapping` (with `--with-passthrough`) | `ProxMate-Infra` | `Mapping.Audit`, `Mapping.Use` |
| `/`, not propagated | `ProxMate-Cluster` | `Sys.Audit`; `Sys.Modify` only with `--with-cluster-firewall` |

By default no storage gets `Datastore.Allocate`, which deletes any volume on it. ProxMate writes
cloud-init snippets through its mount and needs it only to switch a storage's snippets content
type on (do that once as root instead), and to delete a downloaded cloud image after building a
template (without it the image stays under the import storage's `import/` folder for you to
remove). The script warns when a storage you name also holds disk images, container volumes or
backups. Re-running it removes `ProxMate-*` grants an earlier run left on other paths (an old pool
or storage); `--dry-run` shows them.

Uploaded backups need a shared backup storage in this mode (or a single node): ProxMate checks the
upload on the share and reads its guest config from the file itself, because Proxmox would only
answer those questions about an archive whose guest does not exist yet with `Datastore.Allocate`
on the whole backup storage. `.vma` and container `.tar` backups, plain or `.zst`/`.gz`, are read;
`.lzo` and old-format QEMU `.tar` ones are refused. In both modes the restored guest's config is
read back from Proxmox before it is used. A guest carrying settings ProxMate won't restore first
loses its protection and autostart and is then deleted; if Proxmox refuses the delete, the refused
settings are removed where the API allows it, every NIC is set to link down, and the machine is
held for an admin.

`Sys.Modify` on `/` also opens cluster backup jobs, the cluster firewall's groups and IP sets and
other cluster settings, so it is opt-in: while the token holds it the pool does not fully contain
it. Without it, turn on tenant isolation's cluster firewall once by hand as root (Datacenter →
Firewall: add an accept rule for your management network on ports 8006 and 22, then enable it);
the connection test warns until it is on. Before Proxmox VE 8.2 there is no `Sys.AccessNetwork`,
so the script grants `Sys.Modify` on `/` for cloud-image downloads and says so.

Then set **Proxmox pool** to the same name (setup wizard, or Admin → Settings → Proxmox
connection): every guest ProxMate creates, clones, restores, imports or adopts through "Import
existing guests" is placed in it. Setting it does not move existing guests. When switching an
existing install over:

1. Run the script with `--pool <name>`. It keeps the token's earlier `ProxMate` grant on `/`.
2. Set the pool in ProxMate and run the connection test. It warns that the token still holds guest
   privileges on `/`, and lists the guests ProxMate manages that are outside the pool.
3. Click **Move managed guests into the pool**. Only guests carrying a ProxMate ownership tag
   (machines and Template Store templates) are moved, one at a time; a template in another pool is
   moved only after you confirm. Re-test until nothing is listed.
4. Run `bash proxmox-token.sh --pool <name> --drop-root-grant` and test again: the `/` warning is
   gone. A guest left outside the pool is now invisible to ProxMate (its row reads as unconfirmed);
   as root, `pveum pool modify <name> --vms <vmid> --allow-move 1` puts it in.

With a pool-scoped token ProxMate only sees guests in the pool: the Cluster Balancer, alerts and
"Import existing guests" work on those alone. Templates you register by hand must be in the pool,
since cloning checks `VM.Clone` on the template. Backups of a guest that is gone (deleted, or
destroyed by a rebuild that then failed) are not reachable through the pool: the token cannot list
them, read their config or delete them. Restore one as root, or rebuild the machine first (its VMID
back in the pool makes its backups reachable again). This mode has
not yet been run end to end on a live cluster.

---

## 3. Get the code onto the server

```bash
git clone <your ProxMate repo URL> proxmate
cd proxmate
```

(Or copy the repo across with `rsync`/`scp`. Everything below runs from the repo root.)

---

## 4. Configure environment

```bash
cp .env.docker.example .env
openssl rand -hex 32          # copy the output into ENCRYPTION_KEY
nano .env
```

Fill in the **PRODUCTION** block (replace the domain). The key values:

| Variable | Production value | Why |
|---|---|---|
| `ENCRYPTION_KEY` | the 64-hex string you generated | Encrypts Proxmox token, JWT secret, SSO/SMTP secrets at rest. **Keep it stable** — changing it makes stored secrets unreadable. |
| `FRONTEND_URL` | `https://proxmate.example.com` | CORS allow-list + redirect targets. |
| `NEXT_PUBLIC_API_URL` | `https://proxmate.example.com/api` | Baked into the browser bundle at **build time**. |
| `BACKEND_PUBLIC_URL` | `https://proxmate.example.com` | Builds the SSO callback `…/api/auth/sso/callback`. |
| `WEBAUTHN_RP_ID` | `proxmate.example.com` | Passkey relying-party (domain only, no scheme/port). |
| `WEBAUTHN_ORIGIN` | `https://proxmate.example.com` | Passkey origin (full https origin). |
| `COOKIE_SECURE` | `true` | Send `Secure` cookies (HTTPS). |
| `TRUST_PROXY` | `1` | One proxy hop (Caddy) → real client IP for rate-limit + audit. |
| `REAL_IP_HEADER` | *(optional)* default `cf-connecting-ip` | Header the audit log reads the client IP from. Honoured **only when `TRUST_PROXY` ≥ 1**; `x-forwarded-for` is resolved through the trusted hop count rather than read verbatim. Behind plain Caddy/nginx set `x-forwarded-for`; set it empty to always record the socket address. |
| `BIND_ADDR` | `127.0.0.1` | Only the local reverse proxy can reach the app ports. |
| `PROXMATE_BACKUP_DIR` | *(recommended)* e.g. `/mnt/backups/proxmate-db` | **Where ProxMate's own database snapshots land on the HOST** (§12). The container-side path is fixed at `/var/backups/proxmate`; this chooses the host directory that gets mounted there. Creating a folder on the host does **nothing** unless it's set here — the backend runs in a container and can only write to paths mounted into it. Changing it requires `docker compose up -d backend` (a mount is only established at container-create time), not a restart. Default: `./backups` next to `docker-compose.yml`. |
| `BACKUP_DOWNLOAD_DIR` | *(optional)* e.g. `/backups` | **Enables tenant backup downloads.** Mount your backup share (the same NFS/CIFS/PBS-dir Proxmox writes vzdumps to) into the API container and point this at it. Tenants then get a **Download** button on each MateState that emails them a single-use, 1-hour link; ProxMate streams the file off this mount (the Proxmox API can't stream vzdump bytes). Requires SMTP. Leave unset to keep the feature off. |
| `RESTORE_UPLOAD_MAX_GB` | *(optional)* default `50` | Size cap for **restore-from-upload** (below). `0` disables the cap. |
| `SNIPPET_DIR` + `SNIPPET_STORAGE` | *(optional)* e.g. `/snippets` + `local` | **Enables on-demand cloud-init snippet writing.** Point `SNIPPET_DIR` at a snippets directory that is both mounted into the API container **and** backed by a Proxmox storage every node can read (an NFS export is ideal), and set `SNIPPET_STORAGE` to that storage's id (with the `snippets` content type enabled). ProxMate then writes the exact cloud-init extras a deploy needs, atomically, at deploy time — no per-node file placement. Leave unset to fall back to manual snippet pre-placement. |
| `METRICS_TOKEN` | *(recommended)* a random string | Guards `GET /metrics`. **In production `/metrics` returns 404 unless this is set** — there is no localhost exemption, so a Prometheus scrape from `127.0.0.1` gets a 404 too. Scrapers send `Authorization: Bearer <token>`. |
| `ALLOW_PRIVATE_OUTBOUND_URLS` | *(optional)* `true` | Admin-configured **webhooks** and **cloud-image URLs** are blocked from targeting private/loopback/metadata addresses (SSRF guard). Set `true` only if you legitimately point at a **LAN** service (self-hosted Mattermost/ntfy, a local image mirror). The scheme + no-credentials checks always stay enforced. |
| `MATESTATE_CRON` / `ACCESS_EXPIRY_CRON` | *(optional)* 5-field cron | Override the guest-backup schedule (default Sundays 03:00) and the compute-access-window sweep (default hourly at :20). |
| `APPDB_BACKUP_CRON` / `BALANCER_CRON` / `TEMPLATE_REFRESH_CRON` | *(optional)* 5-field cron | Override the app-database snapshot schedule (default nightly 02:30), the Cluster Balancer and the cloud-image template refresh. An unset or invalid expression uses the built-in schedule. |
| `USAGE_DIGEST_CRON` | *(optional)* 5-field cron | Override when the weekly usage digest goes to users who opted in (default Mondays 08:00). Like every schedule here it is read in the backend's local time, which is **UTC in the Docker image**. An invalid expression logs a warning and uses the default. |
| `AUTH_RATE_LIMIT_MAX` / `_WINDOW_MS` | *(optional)* default `20` / `900000` | Login / register / invite throttle per IP. |
| `AUTH_LOCKOUT_MAX` / `AUTH_LOCKOUT_MINUTES` | *(optional)* default `10` / `15` | Per-account lockout after consecutive failed passwords (auto-unlocks). |
| `SHARE_RATE_LIMIT_MAX` / `_WINDOW_MS` | *(optional)* default `30` / `3600000` | Sharing a VM by email, per signed-in user. |
| `GUEST_FILE_RATE_LIMIT_MAX` / `_WINDOW_MS` | *(optional)* default `30` / `600000` | Pushing files into a VM, per signed-in user. |
| `GUEST_LOG_RATE_LIMIT_MAX` | *(optional)* default `30` | Reading the IDE install log, per signed-in user per minute. |
| `GUEST_FILE_PUSH_MAX_BYTES` | *(optional)* default `1048576` | Largest file a push accepts; capped at 8 MiB (`8388608`). |
| `PUBLIC_TOKEN_RATE_LIMIT_MAX` / `_WINDOW_MS` | *(optional)* default `60` / `900000` | Public token lookups (invite links, downloads), per IP. |
| `ALERT_COOLDOWN_MIN` | *(optional)* default `60`, minimum `5` | Minutes before a firing alert rule can fire again. |
| `RECONCILE_STUCK_MINUTES` | *(optional)* default `15`, minimum `1` | Minutes a machine left creating, restoring or rebuilding (or a backup left running) by a restart must sit untouched before ProxMate settles it. Operations still running are never touched. |
| `PROXMOX_CA_CERT_FILE` | *(optional)* e.g. `/certs/pve-ca.pem` | Keep Proxmox TLS verification on against a private CA. A container path: mount the PEM in `docker-compose.override.yml`. |
| `IDE_GUEST_PORT` | *(optional)* default `8080` | Port `code-server` listens on inside tenant VMs for the ProxMate IDE (§16). The proxy, the guest install and the firewall pinhole all use it. |
| `SELF_UPDATE_ENABLED` / `UPDATE_REPO` | *(optional)* | The one-click update button (§14.3) and the GitHub repo the release check reads. `GITHUB_TOKEN` raises the GitHub API rate limit for that check. |
| `PROXMATE_MODULES` / `LOG_LEVEL` | *(optional)* | Optional modules to enable ([`backend/src/modules/README.md`](backend/src/modules/README.md)) and the log level (default `info`). |

`docker-compose.yml` forwards every setting listed in `.env.docker.example` to the
backend; the file documents each one with its default.

> **`SNIPPET_DIR` and `BACKUP_DOWNLOAD_DIR` are CONTAINER paths.** Setting the variable is only
> half the job — the host directory must also be **mounted into the backend container**, or the
> path simply won't exist inside it. Add the mount to a `docker-compose.override.yml` next to
> `docker-compose.yml` (copy `docker-compose.override.example.yml`, which has both lines ready to
> uncomment) and keep its container-side path identical to the variable. (This is the same container-boundary trap described for `PROXMATE_BACKUP_DIR`
> in §12.)

> **Restore from old build (upload a MateState backup).** When `BACKUP_DOWNLOAD_DIR` is
> mounted **read-write** (`rw`, not `:ro`), the create-VM wizard gains a **"Restore from old
> build"** source: a tenant uploads a vzdump archive they previously downloaded and ProxMate
> restores it as a new machine — the migration path between clusters or ProxMate instances.
> The upload is streamed onto the mount (the Proxmox API can't accept backup uploads), the
> embedded config is quota-checked via `vzdump extractconfig` before anything is restored,
> volumes are remapped onto the default disk storage, the guest gets fresh MAC addresses, and
> the tenant-isolation firewall is applied before first boot. The uploaded file is removed
> after the restore. Mount the share `:ro` to keep downloads but disable uploads.
> **Cloudflare note:** the free Cloudflare plan caps request bodies at ~100 MB, so multi-GB
> uploads through a Cloudflare Tunnel will be rejected at the edge — upload from the LAN /
> Tailscale origin instead, or raise the plan limit.

> If you ever change `NEXT_PUBLIC_API_URL`, you must **rebuild** the frontend image
> (it's compiled in, not read at runtime).

---

## 5. TLS + reverse proxy (Caddy)

Caddy terminates HTTPS on the host and routes to the two localhost-bound containers.

```bash
# Install Caddy (Debian/Ubuntu example): https://caddyserver.com/docs/install
sudo cp deploy/Caddyfile /etc/caddy/Caddyfile
sudo nano /etc/caddy/Caddyfile      # set your domain + ACME email
sudo systemctl reload caddy
```

Caddy will obtain and auto-renew a Let's Encrypt certificate as soon as DNS points
at the box and ports 80/443 are reachable. (Prefer nginx/Traefik? Mirror the same
routing: `/api/*` → `127.0.0.1:4000` with WebSocket upgrade, everything else →
`127.0.0.1:3000`.)

### Alternative: Cloudflare Tunnel (no open ports)

If you can't (or don't want to) open 80/443 — e.g. behind CGNAT — front ProxMate with
a **Cloudflare Tunnel** instead. TLS terminates at Cloudflare's edge and `cloudflared`
forwards plain HTTP to a **single local origin**, so you still need one merge-proxy that
joins `/api` and `/` onto one host:port. Run a small Caddy "merge-proxy" (no TLS):

```
# deploy/Caddyfile.proxy
:8184 {
	handle /api/* { reverse_proxy 127.0.0.1:4000 }
	handle       { reverse_proxy 127.0.0.1:3000 }
}
```
Start it (`auto_https off`), then in the Cloudflare Zero Trust dashboard point the public
hostname (`proxmate.example.com`) at `http://<host>:8184`. WebSockets (the noVNC console)
pass through automatically on proxied hostnames. The `.env` values are identical to the
reverse-proxy setup above (single HTTPS origin), and **no inbound ports** are exposed on
the host. In particular keep **`TRUST_PROXY=1`**: the audit log only uses Cloudflare's
`CF-Connecting-IP` header (the `REAL_IP_HEADER` default) when a proxy is trusted, so with
`TRUST_PROXY` unset or `0` every entry records the merge-proxy's address instead of the
client's.

---

## 6. Build & launch

```bash
docker compose up -d --build
docker compose ps
docker compose logs -f backend     # watch "Applying database migrations…" then "Starting ProxMate API…"
```

The backend entrypoint runs `prisma migrate deploy` against the SQLite DB on the
named volume `proxmate-data`, so all auth migrations (cookies/CSRF, password reset,
2FA, passkeys, SSO, invite-2FA) apply automatically on first boot.

Browse to **https://proxmate.example.com** — you should land on the setup wizard.

---

## 7. First run (OOBE)

### 7.1 Create the owner + connect Proxmox
1. The setup wizard creates the **first admin (owner)** — choose a strong password.
2. Enter the **Proxmox host** (`https://<pve-host>:8006`), **Token ID**
   (`root@pam!proxmate`), and **Secret** from §2.2. Leave "verify SSL" off if your
   Proxmox uses a self-signed cert.
3. Set **VM defaults** (storage, network — point this at your tenant VLAN/VNet if you
   made one; Linux bridges, Open vSwitch bridges and SDN VNets are listed, and a name
   can be typed in — ISO storage). Every step after the admin account needs that
   admin's session.

### 7.2 Smoke-test the core
- Dashboard shows live cluster capacity (admins see cluster stats).
- Create a tiny test VM (or deploy from a template / cloud-init image) → start →
  open the **noVNC console** (this exercises the WebSocket relay through Caddy) → delete.

### 7.3 Turn on tenant isolation enforcement
- Admin **Settings ▸ Proxmox tab ▸ Tenant network isolation** → click **Enable enforcement**. ProxMate
  first adds management allow-rules (web 8006 + SSH 22 on the auto-derived
  `suggestedMgmtCidr`) **before** flipping the datacenter firewall on, so you don't
  lock yourself out. Confirm the suggested CIDR matches your mgmt network.
- Verify you can still reach Proxmox (8006) and SSH, then confirm tenant VMs can't
  reach each other / your mgmt subnet.

---

## 8. Email (SMTP)

Enables password-reset emails — without it, resets fall back to an admin-approval queue.
SMTP also gates a lot more than resets, so it's worth configuring even if you're happy
approving resets by hand:

- **Compute access-window warnings** (§12b) — the 7-day and 1-day notices before a tenant's
  access lapses, plus the `access.expired` notification to admins. Without SMTP the
  suspension still happens, just silently.
- **Invites** sent by email, **account-lockout** alerts, backup-failure and
  provisioning-error notifications, and the **single-use backup download links**
  (`BACKUP_DOWNLOAD_DIR`, §4) — which cannot work at all without it.
- **Tenant activity emails** — a tenant hears when an admin sets a VM up in their account, or when a
  VM is shared with them or their share level changes — and the opt-in **weekly usage digest**
  (`USAGE_DIGEST_CRON`, default Mondays 08:00). Without SMTP these are skipped with a log line.

All of it renders from one branded template, so verifying a single send covers the lot.

1. Admin **Settings ▸ Notifications tab ▸ Email (SMTP)** → enter host, port (587 STARTTLS or 465 TLS),
   username, password, and a From address. Point it at any relay (e.g. the same one
   your Proxmox notifications use).
2. Click **Save**, then **Test** (verifies the connection/credentials).
3. Confirm end-to-end: log out → **Forgot your password?** → check the inbox for the
   reset link → reset → log in.

---

## 9. Single sign-on with Keycloak (OIDC)

### 9.1 In Keycloak
1. Create (or pick) a **realm**, e.g. `proxmate`.
2. **Clients ▸ Create client** → OpenID Connect, Client ID `proxmate` → Next.
3. **Client authentication: ON** (confidential), **Standard flow** enabled → Save.
4. **Valid redirect URIs:** `https://proxmate.example.com/api/auth/sso/callback`
   (this exact URL is shown in ProxMate's SSO settings card — copy it from there).
5. **Credentials** tab → copy the **Client secret**.
6. **Group→admin mapping (optional but recommended):**
   - Create a group, e.g. `proxmate-admins`, and add your admin user to it.
   - Add a **Client scope / protocol mapper** of type **Group Membership**:
     Token Claim Name = `groups`, **Full group path = OFF**, include in ID token.
   - This puts `"groups": ["proxmate-admins"]` in the ID token.
7. Your **Issuer URL** is `https://<keycloak-host>/realms/proxmate`
   (it must serve `…/realms/proxmate/.well-known/openid-configuration`).

### 9.2 In ProxMate
Admin **Settings ▸ Access tab ▸ Single sign-on (OIDC)**:
- **Issuer URL:** `https://<keycloak-host>/realms/proxmate`
- **Client ID / Client secret:** from Keycloak
- **Scopes:** `openid profile email`
- **Groups claim:** `groups` · **Admin group:** `proxmate-admins`
- **Auto-create accounts (JIT):** off = only existing/invited users may sign in
  (recommended for invite-only); on = any Keycloak user is provisioned on first login.
- Tick **Enable SSO**, set the button label, **Save**, then **Test** (runs discovery).

### 9.3 Test the SSO flow
- Log out → the login page shows your **SSO button** → sign in via Keycloak → you
  land on the dashboard.
- **Hybrid check:** a user invited in ProxMate (local account) who signs in via
  Keycloak with the **same email** is linked automatically.
- **Admin mapping:** a Keycloak user in `proxmate-admins` becomes an admin (members
  are promoted; the mapping never auto-demotes an existing admin/owner).

---

## 10. Two-step authentication — full test matrix

Run these against the production HTTPS site (passkeys won't work otherwise).

| # | What | How | Expected |
|---|---|---|---|
| 1 | **TOTP enroll** | `/security` → Set up 2FA → scan QR (Google Authenticator/1Password/Authy) → enter code | 2FA on; **10 recovery codes** shown once |
| 2 | **TOTP login** | Log out → log in with password | Prompted for a 6-digit code; correct → in, wrong → rejected |
| 3 | **Recovery code** | At the 2FA prompt, enter a recovery code | Works once; reusing the same code is rejected |
| 4 | **Passkey enroll** | `/security` → Passkeys → Add a passkey | Browser biometric/security-key prompt; passkey listed |
| 5 | **Passwordless passkey login** | Log out → **Sign in with a passkey** | Authenticate with the device; lands on dashboard (skips password + TOTP) |
| 6 | **Invite-enforced 2FA** | Create an invite with **Require two-step authentication** ticked → register through it | New user is corralled to `/security` and **can't use VMs** until they enroll TOTP or a passkey |
| 7 | **SSO exemption** | An SSO-linked user with require-2FA | Signing in **through SSO** isn't forced into local 2FA (their IdP handles it); a **password** login on the same account still needs the second factor |

---

## 11. Security hardening checklist

- [ ] **Host firewall:** only 80/443 (and your admin SSH) open to the world; app
      ports stay on `127.0.0.1`.
- [ ] **`ENCRYPTION_KEY`** backed up in a secret manager — losing it orphans every stored secret
      (the Proxmox token, SMTP creds, tenant AI keys, TOTP secrets). A database backup **without
      this key cannot be restored usefully** — treat the key and the DB backups as a pair, stored
      in different places than the host itself.
- [ ] **`.env` permissions:** `chmod 600 .env`; never commit it.
- [ ] **Proxmox isolation enforcement ON** (§7.3) and verified.
- [ ] **Backups (built-in):** set `PROXMATE_BACKUP_DIR` in `.env` to where snapshots should land
      on the **host**, then confirm the directory under **Admin → Settings → Maintenance →
      App-database backups**. ProxMate snapshots its own DB nightly (`VACUUM INTO`, consistent on
      the live DB) with rolling retention. Use **Back up now** to prove the path works.
      Schedule override: `APPDB_BACKUP_CRON` (default nightly 02:30). See §12.
- [ ] **Backups (manual alternative):** snapshot the `proxmate-data` volume —
      `docker run --rm -v proxmate_proxmate-data:/data -v "$PWD":/backup alpine tar czf /backup/proxmate-db-$(date +%F).tgz -C /data .`
- [ ] **Updates:** `./deploy/update.sh` onto a release tag (§14); migrations apply on boot.
- [ ] Consider enforcing **2FA on the owner/admin** account immediately after setup.

---

## 12. App-database backups

ProxMate can snapshot **its own database** (users, VM records, settings, encrypted
secrets) on a schedule. This is separate from MateStates, which back up the guests.

### The one thing that trips people up

ProxMate runs **in a container**. It can only write to directories that were
**mounted into** that container from the host. Creating a folder on the host and
typing its path into the app does *not* work — the container has its own
filesystem, so `/srv/backups` on the host and `/srv/backups` in the container are
unrelated paths.

That is why the host directory is chosen with an environment variable, not in the
app: the mount has to exist before the container starts.

### Setup

1. In `.env`, point `PROXMATE_BACKUP_DIR` at the host directory you want:

   ```bash
   PROXMATE_BACKUP_DIR=/mnt/backups/proxmate
   ```

   Prefer a disk that is **not** the one holding the database, ideally an
   off-host mount (NFS/CIFS) so losing the host doesn't lose the backups with it.
   The default is `./backups` next to `docker-compose.yml`, which works out of
   the box but lives on the same machine.

2. Make sure the directory is writable by the container's non-root user
   (uid 1000), then recreate the backend so the mount is picked up:

   ```bash
   sudo install -d -m 0700 -o 1000 -g 1000 /mnt/backups/proxmate
   docker compose up -d backend
   ```

3. In **Admin → Settings → Maintenance → App-database backups**, the directory
   should already read `/var/backups/proxmate` — the fixed container-side path
   your host directory is mounted onto. Set **Keep** (rolling retention) and
   press **Back up now** to prove the whole path works.

Snapshots are named `proxmate-appdb-YYYYMMDD-HHMMSS.db`. Retention only ever
deletes files matching that pattern, so anything else in the directory is safe.

Each snapshot is written mode `0600`, owned by uid 1000 (the container's `node` user),
so on the host only root and the account with uid 1000 can read it — use `sudo` to
copy one if your own account has a different uid. Treat the files as secret:
besides the encrypted secrets they contain the session table, so anyone who can
read a recent snapshot can act as a signed-in user. Keep the directory (and any
share it lives on) restricted to uid 1000 and root.

Schedule override: `APPDB_BACKUP_CRON` (default nightly 02:30).

### Restoring

A snapshot is a plain SQLite database file. To restore:

```bash
docker compose stop backend
docker run --rm -v proxmate_proxmate-data:/data -v /mnt/backups/proxmate:/src alpine   cp /src/proxmate-appdb-20260729-023000.db /data/proxmate.db
docker compose up -d backend
```

Migrations run automatically on start, so restoring an older snapshot onto a
newer ProxMate is fine.

> **The backup is useless without the matching `ENCRYPTION_KEY`.** Every stored
> secret — the Proxmox API token, SMTP credentials, TOTP secrets, tenant AI keys —
> is encrypted with it. Restoring a database with a different key leaves the app
> running but unable to decrypt any of them. Back the key up separately from the
> database, in a password manager or secret store.

Take a copy of the current database before restoring over it, so a wrong choice
of snapshot is reversible.

> **Get `ENCRYPTION_KEY` off this host.** The key never changes, so one copy in a
> password manager or secret store closes the risk permanently. Snapshots stored on a
> second disk in the same machine do **not** protect you: lose the chassis and every
> secret inside those backups is undecryptable. Do this now rather than at restore time.
> Admins see a banner on every page until one of them confirms the key is backed up;
> the confirmation (who and when) is audited, and a changed key brings the banner back.

---

## 12b. Compute access windows (time-boxed tenants)

Invites can grant access for a **fixed term** or **never expire**. The clock is anchored at
sign-up (not at invite creation), and admins can override any user's window afterwards under
**Settings ▸ Access**.

What operators need to know before handing out invites:

- **Expiry suspends, it never deletes.** When a window closes the tenant's VMs are stopped and
  sign-in is refused, but nothing is destroyed — extending the window restores access intact.
- **Enforcement is everywhere**, not just at login: the console WebSocket, the IDE proxy, and
  both API-token families all check it, so an expired tenant can't keep working through a
  still-open tab or a saved token.
- **Admins never expire.**
- A sweep runs **hourly at :20**, with a catch-up shortly after boot — so a window that lapsed
  while the host was down is still enforced on the way back up. Override the schedule with
  `ACCESS_EXPIRY_CRON` (§4).
- **Warning emails (7-day and 1-day) and the admin `access.expired` notification require SMTP
  (§8).** Without it, tenants get suspended with no warning.

---

## 13. Final verification checklist

- [ ] `https://proxmate.example.com` loads over a valid certificate.
- [ ] OOBE done; Proxmox connected; a test VM created, console opened, deleted.
- [ ] Tenant isolation enforcement on; mgmt (8006/SSH) still reachable; tenants isolated.
- [ ] SMTP test passes; password-reset email round-trips.
- [ ] SSO button works; hybrid email-link + admin-group mapping verified.
- [ ] TOTP, recovery codes, passkeys, and invite-enforced 2FA all behave per §10.
- [ ] Audit log (admin ▸ Audit) shows logins, 2FA/passkey/SSO events.

---

## 14. Updating ProxMate

ProxMate ships an in-app updater. **Admin ▸ Settings ▸ Maintenance tab ▸ Updates** shows your running
version and a **Check for updates** button that reads the latest **GitHub Release**;
if a newer one exists it shows the release notes and lets you decide whether to apply it.

Because a container can't rebuild and restart itself, the actual update is done by a
**host-side script** (`deploy/update.sh`). You can run that by hand, or wire up the
**opt-in one-click** button which hands the job to the host via a systemd unit.

```mermaid
flowchart TD
  admin([Admin]) -->|Check for updates| app[ProxMate backend]
  app -->|GET releases/latest| gh[(GitHub Releases)]
  admin -->|Install update| app
  app -->|write update-request.json| ctrl[/control dir<br/>bind mount/]
  ctrl -. watched by .-> pathunit[systemd .path]
  pathunit -->|flag appeared| svc[proxmate-updater.service]
  svc -->|deploy/update.sh| steps["git checkout TAG<br/>docker compose build<br/>docker compose up -d<br/>(migrations run on boot)"]
  steps -->|write update-status.json| ctrl
  app -->|poll status| ctrl
```

### 14.1 Publishing a release (so there's a "Latest" to find)

The check compares `backend/package.json`'s `version` against the latest GitHub Release tag.
To cut one:

```bash
# bump backend/package.json "version" (e.g. 0.2.0), then:
git commit -am "release: v0.2.0"
git tag v0.2.0
git push origin main --tags
```

`.github/workflows/release.yml` turns the pushed `v*` tag into a published Release with
auto-generated notes. (Manual alternative: `gh release create v0.2.0 --generate-notes`.)

> Set `UPDATE_REPO` in `.env` if you track a fork instead of the upstream repo.

### 14.2 Manual update (no host changes needed)

```bash
cd /opt/proxmate
git fetch --tags
./deploy/update.sh            # newest vX.Y.Z tag
./deploy/update.sh v0.2.0     # or a specific tag
```

The script checks out the tag, runs `docker compose build` + `up -d`, and the backend
applies DB migrations on boot (`docker-entrypoint.sh`). **Back up the DB first** (the
`proxmate-data` volume) so you can roll back. Rollback = `./deploy/update.sh <previous-tag>`.

> **Keep local changes out of tracked files.** Extra mounts and any other compose
> customisation go in `docker-compose.override.yml` (untracked, merged automatically by
> `docker compose`; start from `docker-compose.override.example.yml`). Settings go in `.env`.
> The script refuses to update — and reports which files — when a tracked file such as
> `docker-compose.yml` has local edits, rather than discarding them. Move those edits into
> the override, `git checkout -- docker-compose.yml`, and run it again.

### 14.3 Enabling the one-click "Install update" button

1. **Mount the control dir + enable the flag** — create the directory owned by the
   backend's unprivileged user (uid 1000), so the app can write its request there:
   ```bash
   sudo install -d -m 0775 -o 1000 -g 1000 deploy/update-control
   ```
   (the backend's entrypoint also hands an existing `/control` mount to that user on
   start). Then in `docker-compose.override.yml` (copy
   `docker-compose.override.example.yml`), uncomment `- ./deploy/update-control:/control`
   under the backend `volumes`, and in `.env` set:
   ```
   SELF_UPDATE_ENABLED=true
   ```
   Then `docker compose up -d` to apply.
2. **Install the host updater unit** (as root; edit `PROXMATE_DIR` in the unit if your
   checkout isn't `/opt/proxmate`):
   ```bash
   cp deploy/proxmate-updater.service deploy/proxmate-updater.path /etc/systemd/system/
   systemctl daemon-reload
   systemctl enable --now proxmate-updater.path
   ```
3. Now **Install update** in the UI drops a request flag the unit picks up; the card polls
   the host's status file and shows progress, then a **Reload** when it's done. If any
   step fails (fetch, checkout, build, restart — or the run is killed), the card shows an
   error naming the step; the full output is in `journalctl -u proxmate-updater`.
   The updater runs git (including the fetch) as the checkout's owner, not root, so if you
   track a private fork its credentials must work for that account without a prompt.

> **Security note:** the app itself never runs Docker or git — it only writes one JSON
> flag in the bind-mounted control dir. All privileged work lives in `update.sh`, run by
> the host's systemd. Leave `SELF_UPDATE_ENABLED=false` (the default) to keep the apply
> path entirely manual.

If an update reports *"SELF_UPDATE_ENABLED is true but the backend has no /control
mount"*, the mount went missing (see §14.4): add it to `docker-compose.override.yml` and
run `docker compose up -d`.

### 14.4 Upgrading from 1.0.x and earlier

The update into this release is performed by the updater **already on your host** — the
1.0.x `deploy/update.sh`, which force-checks-out the new release. Anything you changed in
`docker-compose.yml` (typically the uncommented `/control`, private-CA, snippets or
backup-download mounts) is **silently discarded** by that run, and the stack restarts
without those mounts. The new updater refuses to run over such edits, but it only takes
over after this update. So, **before** you update:

1. **Move your compose edits into the override file.** See what you changed, copy those
   lines into `docker-compose.override.yml`, then restore the stock file:
   ```bash
   cd /opt/proxmate
   git diff docker-compose.yml                     # your local edits
   # The template is new in this release, so take it from the release tag:
   git fetch --tags
   git show <new-tag>:docker-compose.override.example.yml > docker-compose.override.yml
   nano docker-compose.override.yml                # uncomment the same mounts
   git checkout -- docker-compose.yml
   sudo docker compose config >/dev/null && echo ok   # the merged file still parses
   ```
   The old updater's `docker compose up -d` also reads the override, so the mounts survive
   this update — whether you use the in-app button or not.
2. **Update** as usual (**Install update**, or `sudo ./deploy/update.sh <new-tag>`). If the
   card stays on *queued*, run `sudo ./deploy/update.sh <new-tag>` by hand; files that an
   earlier `sudo` run left owned by root are handed back to the checkout's owner first.
3. **Re-install the systemd units** — the service now marks killed runs as failed
   (`ExecStopPost`):
   ```bash
   sudo cp deploy/proxmate-updater.service deploy/proxmate-updater.path /etc/systemd/system/
   sudo systemctl daemon-reload
   ```
   (Re-apply your `PROXMATE_DIR` / path edits if your checkout isn't `/opt/proxmate`.)

Also changed in this release:

- **Guest ownership tags.** ProxMate now tags every guest it manages (`proxmate-<id>`) and
  acts only on guests carrying their own machine's tag, so the Proxmox token must be allowed
  to set tags (`VM.Config.Options`; a privsep-off or Administrator token is). The first start
  tags existing guests; machines it can't match are listed under **Settings → Maintenance →
  Unconfirmed machines** for an admin to adopt or remove (see the admin guide §7).
- **Images move to Node 22.** The first build pulls `node:22-alpine`, so it needs network
  access and takes a little longer than a normal update.
- **`TRUST_PROXY` ≥ 1 is now required for the real-IP header.** The audit log only
  believes `REAL_IP_HEADER` (`cf-connecting-ip` by default) when a proxy is trusted, and
  resolves `x-forwarded-for` through `TRUST_PROXY`'s hop count. Behind Caddy, nginx or a
  Cloudflare Tunnel keep `TRUST_PROXY=1`; with it unset or `0` the audit log records the
  socket address.
- **The backend refuses to start in production without `ENCRYPTION_KEY`.** Docker installs
  already require it (compose fails without it); this affects bare `node`/`npm start` runs,
  which previously generated a key they could lose on restart.

**If the old updater already dropped your mounts** (the in-app button now fails, or the
Proxmox CA / snippets / downloads stopped working): put the mounts in
`docker-compose.override.yml` as in step 1 and run `docker compose up -d`. The next run of
the new `update.sh` also warns when self-update is enabled but `/control` is not mounted.

---

## 15. Rack kiosk mode (touch panel)

ProxMate has a full-screen, touch-friendly **kiosk** view designed for a small panel mounted
on/near the cluster (e.g. a Raspberry Pi Touch Display 2 at 1280×720). It shows an at-a-glance
**command center** — cluster CPU/memory/storage gauges, per-node health + quorum, VM
running/stopped counts, trend sparklines, a live activity feed with tap-to-filter, and a **VMs**
tab of live per-guest cards.

> **The kiosk is monitoring only (since v0.8.6).** The VM cards carry **no power controls** and no
> links off the panel — an unattended screen anyone can walk up to must not be able to stop a
> tenant's VM, or navigate out of kiosk mode and thereby walk around the exit gate. The panel also
> shows no tenant identities (no owner emails or display names).

- **Enter from the app:** **Admin ▸ Monitor ▸ "Kiosk mode"**. The tap requests browser
  fullscreen (a user gesture is required) and routes to the chromeless kiosk.
- **Exit requires re-authentication.** Tapping **X (close)** opens a full-screen unlock gate
  offering a passkey, the admin-set exit PIN, or the account password; only on success does it
  leave fullscreen and route to **Admin ▸ Monitor**. The **fullscreen toggle** switches
  full-screen on and off.
- **Admin-only:** the kiosk shows cluster-wide data, so it's gated to admins (and the underlying
  `/api/admin/*` feeds are admin-gated server-side).
- It keeps the panel awake via the **Screen Wake Lock API** and hides the cursor for a true
  appliance feel.

```mermaid
flowchart LR
  M["Admin ▸ Monitor<br/>'Kiosk mode' button"] -->|requestFullscreen + route| K["/kiosk<br/>(chromeless, admin-only)"]
  K -->|"exit"| M
  P["Pi boots Chromium<br/>--kiosk https://HOST/kiosk"] --> K
```

### 15.1 Auto-launch on the Raspberry Pi (boot straight into kiosk)

`/kiosk` is a stable deep link. For the panel to come up in fullscreen kiosk after a reboot, point
Chromium at it with the `--kiosk` flag (browsers can't self-trigger OS-level fullscreen without a
user gesture, so the flag does it). Example for a Pi running a desktop session — create
`~/.config/autostart/proxmate-kiosk.desktop`:

```ini
[Desktop Entry]
Type=Application
Name=ProxMate Kiosk
Exec=chromium-browser --kiosk --noerrdialogs --disable-infobars --incognito https://proxmate.example.com/kiosk
X-GNOME-Autostart-enabled=true
```

Notes:
- The browser must already hold a logged-in **admin** session (sign in once on the panel; the
  `proxmate_session` cookie persists). The panel stays signed in by design — set an **exit PIN**
  (or enrol a passkey) so leaving kiosk mode still demands a credential, since anyone can physically
  touch the screen. The kiosk itself is read-only, so a passer-by can look but not act.
- Use the externally reachable origin (the same one in `NEXT_PUBLIC_SITE_URL` / your tunnel host).
- To keep the Pi display from blanking at the OS level too, disable DPMS/screen-blanking in your
  Pi's display settings (the in-app Wake Lock covers the browser, but the OS may still blank).

---

## 16. ProxMate IDE (optional, in-guest editor + AI agent)

If you want the per-VM browser IDE, the full setup + security model is in
[`docs/proxmate-ide.md`](docs/proxmate-ide.md). The production essentials:

- **Reachability:** the ProxMate host must reach tenant VM IPs on **TCP :8080** (or
  `IDE_GUEST_PORT`, §4). On a flat
  network this is automatic; on a non-flat one you provide the routing (Tailscale subnet route,
  VPN, static route). ProxMate can't create it.
- **`BACKEND_PUBLIC_URL` must be your https origin** — the in-guest AI agent's gateway URL is built
  from it directly. Since v0.8.0 the configured value wins outright; deriving the scheme from
  `X-Forwarded-Proto` proved unreliable behind a proxy that rewrites it (an http URL gets 301'd,
  which breaks the agent's POST). Header derivation remains only a fallback for bare dev setups.
- **Firewall:** with isolation on, ProxMate opens a managed, infra-scoped pinhole for that port on each
  IDE VM, scoped to **`ide_ingress_cidr`** — set it to the address ProxMate's traffic arrives from
  (the backend host on a flat LAN; the subnet-router node's LAN IP when routed).
- **Guest specs:** IDE VMs need the **qemu-guest-agent** running, **>= 8 GB RAM** (`ide_min_ram_mb`),
  and AVX (ProxMate sets `cpu: host` automatically; reboot to apply).
- Enable + configure models in **admin Settings ▸ IDE tab**.
- **Owner-only:** the IDE opens only for a VM's owner — not for admins or people the VM is
  shared with. IDEs installed by 1.0.x reinstall once, automatically, on their owner's next open.

Prefer an agent to do it? [`DEPLOY_WITH_CLAUDE.md`](DEPLOY_WITH_CLAUDE.md) covers the IDE in §7.

---

## 17. Troubleshooting

**"EACCES: permission denied, mkdir …" when saving the app-database backup
directory.** The path isn't mounted into the backend container — ProxMate can
only write to directories mounted in from the host, and creating the folder on
the host is not enough on its own. Set `PROXMATE_BACKUP_DIR` in `.env` and
recreate the backend (`docker compose up -d backend`); the in-app field should
then read `/var/backups/proxmate`. See §12.

| Symptom | Likely cause / fix |
|---|---|
| Passkey enroll/login fails silently | Not HTTPS, or `WEBAUTHN_RP_ID`/`WEBAUTHN_ORIGIN` don't match the domain in the browser bar. RP ID = domain only, ORIGIN = full `https://…`. |
| Logged out immediately / cookies not sticking | `COOKIE_SECURE=true` requires HTTPS; make sure you're on the HTTPS origin and `FRONTEND_URL`/`NEXT_PUBLIC_API_URL` use the same domain. |
| SSO: "redirect_uri mismatch" | The redirect URI in Keycloak must be **exactly** `${BACKEND_PUBLIC_URL}/api/auth/sso/callback`. |
| SSO: discovery/Test fails | Issuer must serve `/.well-known/openid-configuration`; check the client secret and that the realm name is right. |
| Proxmox storage empty / VM create 403 | Token has **Privilege Separation ON** — recreate with `--privsep 0` (§2.2). |
| Rate-limit/audit shows the proxy IP, not the client | Set `TRUST_PROXY=1` (already in the production `.env`); the audit log ignores `REAL_IP_HEADER` without it. |
| Console (noVNC) won't connect | Ensure the reverse proxy forwards WebSocket upgrades on `/api/*` (Caddy does automatically). |
| Changed `NEXT_PUBLIC_API_URL` but the browser still calls the old URL | Rebuild the frontend image: `docker compose up -d --build frontend`. |

---

*Companion doc: the root `completed-tasks.md`.*