# exomem — remote deployment This guide covers the **remote tier**: running exomem as an always-on HTTP service behind a public HTTPS endpoint so you can reach your vault from **claude.ai** on the web or mobile as a custom connector. The local-only path (Claude Code over stdio, no cloud) is in [../QUICKSTART.md](../QUICKSTART.md); start there if you don't need mobile access. This personal remote profile is still one user and one vault. The managed hosted service uses the same single-vault runtime as a private tenant cell behind a separate shared control plane; it is not made multi-tenant by pointing several accounts at the server configured below. Exomem Cloud operators should also read the Cloud runbooks in [runbooks/](runbooks/) and the [Substrate control-plane contract](substrate-control-plane-contract.md). Throughout, replace `` / `example.com` with your own hostname. For the guided ≤15-minute bring-up, start with [remote-quickstart.md](remote-quickstart.md); this document is the reference. For structured logs, metrics, and diagnosing a failure after the fact, see [observability.md](observability.md). For host-configured content ceilings between owner clients, see [connector boundaries](connector-boundaries.md). ## Architecture ``` ┌──────────────────┐ HTTPS ┌──────────────────────────────┐ │ claude.ai │ ─────────▶ │ public edge (CDN / tunnel) │ │ (mobile/web │ bearer │ kb.example.com │ │ backend) │ │ TLS terminated at edge │ │ 160.79.104.0/21 │ └──────────────────────────────┘ └──────────────────┘ │ │ tunnel ▼ ┌────────────────────────────────────┐ │ host: Linux / Windows │ │ │ │ FastMCP @ 127.0.0.1:8765 │ │ GitHub OAuth (single user) │ │ ↓ │ │ MCP tools (find, get, note, …) │ │ ↓ │ │ /Knowledge Base │ └────────────────────────────────────┘ ``` **Why a public endpoint, not a tailnet-internal one?** claude.ai's MCP client fetches the connector URL *from Anthropic's cloud infrastructure* (egress range `160.79.104.0/21`), not from your phone. A purely internal hostname is therefore unreachable. The auth boundary is not network membership but **GitHub OAuth**, locked down to a single GitHub login via a custom `SingleUserGitHubVerifier` wrapping FastMCP's `OAuthProxy`. claude.ai discovers the OAuth endpoints at `/.well-known/oauth-authorization-server`, registers itself via Dynamic Client Registration at `/register`, and walks the standard authorize → token → use flow. **Downtime.** A single-host deployment without an always-on box accepts downtime: when the host is asleep, mobile writes fail with a connection error — fall back to editing the vault directly (the local capture path). Run on a box that stays up (or a cheap VPS) if you want reliable mobile access. ### Managed hosted topology The managed service has a different ingress and ownership boundary: ```text client -> Substrate public auth / gateway / Home -> authoritative account mapping + internal entitlement -> one private Exomem cell -> one tenant vault + isolated state/log/secret roots ``` The public gateway derives the destination from the authenticated account. No public body, URL, query, cookie, or header may select a tenant, cell, private endpoint, credential, or vault path. Each cell accepts only its unique private gateway/operator identity and exposes content-free readiness. Optional embedding and media workers start only from provider-neutral cell policy and soft-fail to durable capture and lexical recall. Deployment supplies private networking, process/container isolation, encrypted tenant volumes, secret injection, backup object storage, and KMS integration. That is encryption at rest and owner-scoped access—not zero-knowledge or end-to-end encryption. Search and indexing require plaintext in cell memory, so operators must minimize and audit volume access and keep content, queries, paths, and credentials out of operational logs. The gateway and cell negotiate a versioned protocol and are tested as a pair. Optional response fields may be added compatibly; command, parameter, or stable error removal requires a coordinated rollout. Pin the cell release per tenant, take a verified canonical snapshot before a file-format migration, and canary new gateway/cell pairs. Rollback stops routing, drains the affected cell, and starts the previous protocol-compatible image against the same vault. It never rolls canonical Markdown/media backward; disposable sidecars can be rebuilt. Paddle is not installed in, called by, or trusted by an Exomem cell. Substrate owns Paddle checkout, webhooks, portal, reconciliation, and the provider-neutral entitlement projection consumed before routing and at cell startup. Paddle MCP is an operator/development aid for that control-plane lane, not production runtime infrastructure. ## 1. Install dependencies ```powershell cd /path/to/exomem # Install Python deps (creates .venv automatically). # --extra embeddings pulls torch + sentence-transformers for HYBRID search. # --extra media pulls faster-whisper + Pillow + pymupdf + markitdown for # SERVER-SIDE media extraction (auto transcribe/OCR/parse uploaded binaries → # searchable). On Windows the [media] extra also pins the CUDA-12 runtime # (cublas/cudnn/cudart) that ctranslate2 needs alongside torch's cu132 build. uv sync --extra embeddings --extra media ``` Media extraction needs two **system** tools (not pip-installable): - **Tesseract OCR** (images): `winget install --id UB-Mannheim.TesseractOCR -e`. The installer doesn't add it to PATH; the server auto-discovers it at `%ProgramFiles%\Tesseract-OCR\`, or set `EXOMEM_TESSERACT_CMD`. - **ffmpeg** is bundled by PyAV (pulled via faster-whisper), so audio/video decode works without a separate install. Verify the GPU media path: `uv run python scripts/verify-media-gpu.py`. The `media` extra reads PDF, Word, Excel, PowerPoint, HTML, EPUB, OpenDocument (text, spreadsheet, presentation) and RTF files. Plain text, email and calendar files need no extra. OCR detects each image's script with Tesseract's OSD model first. It then reads the image with the installed script models for that script and the installed language packs written in it. A page with too little text for detection reads with `EXOMEM_OCR_DEFAULT_LANGS` (for example `eng+jpn`), or with Tesseract's default English when the variable is unset. Tesseract 4.x does not name its model directory, so on it OCR skips script detection and reads each image once in Tesseract's default language. To read Japanese, install the `jpn` and `jpn_vert` packs and the Japanese script models (Debian: `tesseract-ocr-jpn`, `tesseract-ocr-jpn-vert`, `tesseract-ocr-script-jpan`, `tesseract-ocr-script-jpan-vert`). OCR leaves each word with a confidence under `EXOMEM_OCR_WORD_CONFIDENCE_FLOOR` (default `70`, scale 0 to 100) out of search. The sidecar keeps those words in a `## Low-confidence OCR` section. Set the variable to `0` to keep every word searchable in each image that OCR reads from then on. `EXOMEM_MEDIA_ENGINES` limits which engines run, for example `documents,ocr`. Unset, a personal install runs every engine and a Cloud cell runs none. cellctl renders the variable into a Cloud cell, for one cell or as the fleet default. HEIC photos are not decoded; see `docs/runbooks/cloud-media.md`. Lean / CPU-only boxes can skip all of this — set `EXOMEM_DISABLE_MEDIA_EXTRACTION`; uploads still work, just without server-side searchable-text extraction. CPU also works (no GPU needed), just slower — pick a smaller Whisper model with `EXOMEM_WHISPER_MODEL=base`. The CUDA-12 wheels above are Windows + GPU only, unused on CPU. To make **existing** Evidence/media files searchable (one-shot back-fill; new uploads are handled automatically), run: ```powershell uv run python -m exomem backfill-media --dry-run # preview (writes nothing) uv run python -m exomem backfill-media # do it (CPU or GPU) ``` It writes a sidecar if missing, extracts text (OCR/ASR/PDF), and CLIP-embeds images. Idempotent. Flags: `--no-ocr` (sidecar + CLIP only), `--no-clip`, `--vault `. Media discovery, binary hashing, extraction compute, and the usual sidecar/transcript/failure index fanout run outside the global vault mutation boundary; only the revalidated per-artifact canonical commit is fenced. Lock telemetry can still show short named `background_media_clip_commit` and `background_media_scene_frame_commit` holders while CLIP vectors or persisted scene frames commit. Those are the measured residual paths, not a stuck scan. ## 2. Set up a public HTTPS URL You need a hostname for step 3. Pick **one**: **Option A — Cloudflare Tunnel.** Needs a domain you own in Cloudflare; more burst-tolerant under load. Prereq: `winget install --id Cloudflare.cloudflared`. ```powershell cloudflared tunnel login pwsh -File scripts/setup-cloudflared.ps1 -Hostname kb.example.com -TunnelName exomem-host # -> https://kb.example.com (script makes the tunnel + DNS + auto-start service). ``` In the Cloudflare dashboard for this hostname: Bot Fight Mode **OFF** + no WAF managed ruleset (Security Level low); the edge caps requests at ~100s. **Option B — ngrok.** No domain needed; every ngrok account gets one free static dev domain. Claim it in the ngrok dashboard, then: ```powershell ngrok config add-authtoken ngrok http 8765 --url https://.ngrok-free.dev ``` For an always-on Windows service instead of a foreground process, use `pwsh -File scripts/setup-ngrok.ps1 -Domain .ngrok-free.dev -Port 8765`. Free-tier limits: 120 requests/min, ~20k requests/month, and a one-time browser interstitial on first visit (click through once; it does not recur). Fine for one person; upgrade or switch to Cloudflare if you outgrow the cap. **Why not Tailscale Funnel?** Funnel is beta, has non-configurable bandwidth caps, and its shared relay throttles claude.ai's connector request bursts — producing a "looks disconnected" failure that has nothing to do with exomem itself. It is no longer the recommended no-domain default (ngrok's static domain is more burst-tolerant), but existing Funnel deployments keep working. If you're already running it: ```powershell tailscale funnel --bg --https=443 http://127.0.0.1:8765 tailscale funnel status # note the URL, e.g. https://..ts.net ``` ## 3. Create a GitHub OAuth App (one-time, ~3 min) At → **OAuth Apps** → **New OAuth App**: | Field | Value | |---|---| | Application name | `exomem` | | Homepage URL | `https://kb.example.com` | | Authorization callback URL | `https://kb.example.com/auth/callback` | Save the generated **Client ID** and **Client Secret**. ## 4. Populate `.env` Create `.env` in the repo root: ``` EXOMEM_BASE_URL=https://kb.example.com EXOMEM_GITHUB_USERNAME= EXOMEM_GITHUB_USER_ID= GITHUB_CLIENT_ID= GITHUB_CLIENT_SECRET= # Required root for durable opaque sessions. Rotation revokes every session. # Generate: python -c "import secrets;print(secrets.token_urlsafe(48))" EXOMEM_JWT_SIGNING_KEY= # Required: vault root — the folder that contains Knowledge Base/ EXOMEM_VAULT_PATH= ``` `EXOMEM_BASE_URL` must match your public hostname (from step 2) exactly — no trailing slash, no `/mcp` suffix. `EXOMEM_GITHUB_USERNAME` is case-insensitive but must be the *login*, not the display name. `EXOMEM_GITHUB_USER_ID` is the positive numeric `id` returned by `GET https://api.github.com/users/`; authorization must match both values. `EXOMEM_VAULT_PATH` is **required**: claude.ai connects over HTTP and passes no environment, so the service resolves the vault solely from this line in `.env` at startup. ## 5. Sanity-test locally ```powershell # Configuration profile check uv run python -m exomem doctor --profile remote # stdio (no auth needed) uv run python -m exomem --transport stdio # Ctrl-C to stop # HTTP (OAuth required) uv run python -m exomem --transport streamable-http --host 127.0.0.1 --port 8765 # In another terminal: # curl.exe -i http://127.0.0.1:8765/mcp → expect 401 # curl.exe -i http://127.0.0.1:8765/.well-known/oauth-authorization-server # → expect JSON metadata ``` Once the tunnel is up (step 2) and `.env` is populated (step 4), `exomem doctor --profile remote --probe` runs the equivalent checks above — plus the public OAuth-discovery and bare well-known endpoints through the live tunnel — in one command. ## 6. Install as a service (auto-start on boot) **Option 0 (container):** `docker compose --profile cloudflared up -d` (or `--profile ngrok`) runs the lean server plus the tunnel as one supervised unit. Use `docker compose -f compose.yaml -f compose.ml.yaml ...` for CPU hybrid search, or `docker compose -f compose.yaml -f compose.cuda.yaml ...` for NVIDIA/Linux CUDA capability. The CUDA image remains CPU-default at idle unless you explicitly set performance mode. See [docker.md](docker.md). The native services below suit Windows live vaults (Docker Desktop/WSL2 bind mounts miss live file-watch events) and hosts where Docker isn't wanted. macOS is not a host exomem can serve: the held-filesystem substrate every governed write acquires a reserved-path root through has no darwin backend, so the server refuses there. Pick your platform — both run the same `streamable-http` server and differ only in the OS service manager. The release commands create or update a separate PyPI-backed service venv, load the repository `.env` into the service manager, doctor-gate the selected profile and remote configuration, start the service, and verify that local `/mcp` returns the expected OAuth `401`. The default `standard` profile is multimodal but not permanently model-resident. Evidence work is durable and serialized through one disposable child; after `EXOMEM_MEDIA_IDLE_SECONDS` (default 300) without work, that child exits and returns its RAM plus MPS/MLX/CUDA state. After the deadline, maintainers can verify the persistent-core envelope with `python scripts/verify-resource-envelope.py` (targets: one service, zero media children, <=512 MiB pre-cache RSS, and <1% idle CPU). GPU acceptance is checked separately with `nvidia-smi` or Activity Monitor: the idle core must not be a CUDA compute process and targets <200 MiB GPU delta. On macOS the JSON also reports Activity Monitor's physical-footprint metric via `proc_pid_rusage`; use `--max-memory-mb` for an optional physical-footprint limit. That limit is evaluated only when every server PID has a native sample, while `--max-rss-mb` keeps its existing RSS meaning. **macOS (launchd) — does not work today:** Kept for the darwin held-filesystem backend that does not exist yet. The service starts and every governed write refuses at the reserved-path root, so this is a record of the intended shape rather than a route to follow. ```bash bash scripts/install-service.sh --release # Re-run after a package or .env change to update and restart the service. # Developer checkout mode: bash scripts/install-service.sh --repo-dev --profile standard # Uninstall: launchctl bootout gui/$(id -u)/com.exomem && rm ~/Library/LaunchAgents/com.exomem.plist ``` **Linux (systemd --user):** Add `--seamless` to a release install to preserve existing client connections during later worker upgrades. First enablement requires one reconnect; see [managed service upgrades](managed-service-upgrades.md) for setup and recovery. ```bash bash scripts/install-service.sh --release # The installer attempts to enable user linger so the service survives logout. # Re-run after a package or .env change to update and restart the service. # Developer checkout mode: bash scripts/install-service.sh --repo-dev --profile standard # Uninstall: systemctl --user disable --now exomem && rm ~/.config/systemd/user/exomem.service ``` **Windows (NSSM):** ```powershell # Prereq: NSSM must be installed and on PATH. Easiest: # winget install NSSM.NSSM # or download from https://nssm.cc/download and add nssm.exe to PATH # (or pass -NssmPath "C:\path\to\nssm.exe" to the script below). # The script self-elevates; approve the UAC prompt. pwsh -File scripts/install-service.ps1 -Release # The installer creates/updates the PyPI service venv and verifies /mcp -> 401. # It is idempotent: re-running against a registered service reconfigures it in # place and reuses the venv the service is already installed against. # Developer checkout mode is still available when .venv already exists: # pwsh -File scripts/install-service.ps1 # Uninstall: # nssm stop exomem && nssm remove exomem confirm # Restart (after .env edits): elevated shell required # Start-Process -Verb RunAs -Wait sc.exe -ArgumentList 'stop','exomem' # Start-Process -Verb RunAs -Wait sc.exe -ArgumentList 'start','exomem' ``` **Upgrading after a release — run this, not a bare `uv pip install`:** ```powershell pwsh -File scripts/upgrade.ps1 # Explicit policy: -CliSync auto|always|never (default: auto) ``` Requires PowerShell 7+; Windows PowerShell 5.1 is refused up front rather than allowed to fail obscurely partway. The service runs a PyPI-backed venv that is **not** the repo checkout, so `git pull` never touches it. `upgrade.ps1` locates that venv from the NSSM registry, prints the `target:` version it resolved before touching anything, upgrades it, **fails if the venv did not actually end up on that target** (a `uv` that exits 0 having applied nothing is the failure this catches — #578), repairs the CUDA torch build, gates on `doctor`, restarts, and then asserts the **live** `/health` version matches what it just installed. It then records a non-secret managed-install manifest and, in the default `auto` mode, aligns an existing uv-managed `exomem`/`kb` command to that exact verified release. `auto` never installs a command that was absent; `always` may install it; `never` leaves CLI management alone. `-SkipRestart` stages only the service venv and deliberately defers CLI alignment until a later run verifies the live release. No elevation is needed. The CLI and service need release parity, not dependency parity. The uv-tool CLI stays lean while a `standard` or `media` service keeps its embedding/media extras; duplicating Torch and model dependencies into the command environment is wasteful. Check the resolved command and its paired service identity without loading those optional stacks: ```powershell exomem --version --json kb --version ``` On macOS/Linux, `bash scripts/upgrade.sh` provides the same behavior with `--cli-sync auto|always|never`. A second service on the same machine (its own unit, port and vault, such as a client cell beside a personal one) upgrades with `--unit-file `. The script reads that service's vault from the unit's own environment, never from the checkout's `.env`, and refuses to continue when the unit names none: pass `--vault` then. For a unit that is not the default one, `--cli-sync auto` leaves the uv-tool CLI alone, and the managed-install manifest keeps naming the service it already names. A CLI sync keeps an operator-owned launcher (a regular file where uv would put its symlink) in uv's bin directory. ### Windows upgrade: legacy idempotency runtime DACL Exomem 0.47.0 began requiring a protected, principal-private DACL on Windows idempotency runtime state. An install upgraded from 0.46.0 or earlier can already have a runtime directory whose inherited ACL does not meet that contract. Exomem does not silently rewrite it: the service and `exomem doctor` fail closed and print the exact offending path plus a one-line `icacls.exe` repair for the current runtime identity. Stop the process that owns the reported runtime, open an elevated PowerShell as that intended owner, copy the complete command from the error, run it, then restart the process. The command resets and protects only the named entry; it is not recursive. Do not delete the idempotency database to bypass validation. A command generated by a normal user is not a repair for a LocalSystem-owned path: run the diagnostic under the identity that owns the runtime. The trustee set is exact. LocalSystem state permits only LocalSystem and Administrators; normal-user state permits that user's SID plus LocalSystem and Administrators. One `EXOMEM_WRITER_LEASE_STATE_DIR` therefore cannot serve both an NSSM LocalSystem service and a direct user process. More importantly, that directory also anchors the process-safe mutation lock. Do not point the two identities at separate state roots and then mutate the same vault concurrently. Route ordinary mutations through the service. Before direct CLI maintenance such as `exomem index`, stop the service, use a user-private state root, finish the command, and restart the service. `exomem mode` only updates machine configuration and does not enter the vault mutation path. Releases carrying semantic parser changes bump the parser generation used by the lexical, vector, graph, pack, and count sidecars. Canonical Markdown is not rewritten. After upgrading, quiesce vault mutations and explicitly run `exomem maintain --reconcile` on the host to refresh stale derived rows; service restart alone does not promise this rebuild. Anonymous unit references whose normalized content changed become explicitly stale rather than resolving to a nearby unit; authored `^anchors` remain the stable reference mechanism. A full `maintain_memory(mode="fix", rebuild_embeddings=true)` rebuild is optional when exact vector regeneration is required. Upgrading by hand is the trap this replaces: `uv pip install --upgrade exomem[...]` ignores `[tool.uv.sources]` (only `uv sync` reads it), so it silently pulls the PyPI **CPU** torch wheel over the CUDA build and moves embeddings/media onto the CPU with no other symptom. `doctor` now fails on that state when an NVIDIA driver is present (set `EXOMEM_ALLOW_CPU_TORCH=1` to accept CPU deliberately). ### Renaming an existing `kb-mcp` service Boxes provisioned **before the `kb-mcp` → `exomem` rename** still run the Windows service under the old name `kb-mcp`. The code (and `install-service.ps1` / `restart.ps1`, which now default to `exomem`) are renamed, but the installed NSSM service isn't — it must be re-registered once per box. `restart.ps1` falls back to the legacy `kb-mcp` name automatically so it keeps working until you migrate; to finish the rename, run these in an **elevated** PowerShell (the old service must be removed first or it collides with the new one on port 8765): ```powershell sc.exe stop kb-mcp nssm remove kb-mcp confirm pwsh -File scripts/install-service.ps1 # installs + starts 'exomem', re-grants no-UAC rights ``` The cloudflared tunnel/funnel targets the **port** (127.0.0.1:8765), not the service name, so it keeps working across the rename. Verify with `sc.exe query exomem` (RUNNING) and `sc.exe query kb-mcp` (should not exist). ## 7. Add to claude.ai 1. claude.ai → Settings → Connectors → **Add custom connector** 2. **Name**: `Knowledge Base` (or whatever) 3. **Server URL**: `https://kb.example.com/mcp` (this host's public hostname) 4. Leave **OAuth Client ID** and **OAuth Client Secret** blank — claude.ai uses Dynamic Client Registration against your `/register` endpoint. 5. Save. claude.ai opens a GitHub login window → log in (only the user in `EXOMEM_GITHUB_USERNAME` is allowed) → approve consent → redirects back to claude.ai. The tools appear in the palette. ## Machine-local state placement (and file-sync agents) Machine-local derived state — the semantic/lexical/graph index stores, the graph epoch and receipt records, the deferred-index queue, media-job and governance sidecars, due/review projections, and their rebuild scratch files — lives **outside the vault**, in a per-user, per-vault state root: 1. `EXOMEM_STATE_ROOT` environment override (absolute path, used verbatim), else 2. `%LOCALAPPDATA%\exomem\state` on Windows, else 3. `$XDG_STATE_HOME/exomem/state` (default `~/.local/state/exomem/state`) on POSIX, with one subdirectory per vault, keyed by the vault's resolved path. The vault itself carries only user content, so a file-sync agent watching it (Syncthing, Dropbox, OneDrive, iCloud) has no index store, epoch file, lock, or rebuild scratch to hash, hold, or replace — the failure class that has caused whole-vault rebuild chains and warm-up stalls when live state synced between machines. The structured collection store checks this before it writes its live database. Store custody stays unverified while the vault or the state directory is inside a file-sync root. Custody recognizes a root from the evidence in the `sync-providers` vocabulary registry: client metadata files, cloud document paths, platform variables, Windows sync folder names and vault sync plugins. The built-in sync plugin's enabled bit does not identify a connection or sync root, so the shipped pack does not treat that bit as root evidence. The single-host adapter still checks configured roots, root metadata, platform paths, and separation between the vault and live state. An overlay that explicitly declares core-plugin evidence retains its conservative refusal. No plugin toggle or declaration grants custody. If your sync client is not in the shipped pack, add it with `schema_memory(subject="sync-providers", operation="save")` or in `_Schema/sync-providers.yaml`. You can also list sync roots as absolute paths in `EXOMEM_COLLECTION_STORE_SYNC_ROOTS`. A vault entry can only add evidence; it cannot change or remove a shipped entry. Ordinary service and stateful CLI startup is read-only at this boundary. It never creates the state root, copies or removes a legacy file, resumes an interrupted transition, or upgrades a manifest. Until an explicit offline migration has completed, it refuses with `STATE_MIGRATION_OFFLINE_REQUIRED`. A complete manifest plus even one reintroduced legacy duplicate refuses the same way. `exomem doctor` names the local paths and conflict without exposing them in the public refusal. Every desktop upgrade is one fail-closed transition: **stop -> prove stopped/PID gone -> install target**, then **offline migrate -> doctor -> start -> prove new PID/version**. The stop proof checks both the SCM `Stopped` state and that the exact worker PID captured before the stop no longer exists; checking the NSSM parent after stopping cannot detect an orphan. Run the migration only from the target interpreter: ```powershell pwsh -File scripts/upgrade.ps1 -Vault "C:\path\to\vault" # Or, inside a manually proven stop window: & $ServicePython -m exomem maintain --vault "C:\path\to\vault" ` --migrate-state --offline ``` The migration lock serializes two new migrators; it cannot exclude an older Exomem release that does not know the lock exists. That is why the proven stop window is mandatory. `-SkipRestart` is unavailable for this transition. Any failure leaves the service stopped, including migration, doctor, start, new-worker, and live-version failures. Fix the failure and rerun the whole transition; never start an old installed release against state already migrated by the target. If doctor reports two authorities, repeat the explicit offline command with `--adopt-state external` or `--adopt-state vault` only after deciding which copy to keep. Belt and braces for vaults that carried historical state or may run an older exomem again: keep sync-agent ignore patterns for the state families, e.g. in Syncthing's `.stignore`: ```text .graph-sync*.json .graph-commit-receipts .graph-reset-* .graph-rebuild-* .graph-coordination .authorization-projections *.sqlite *.sqlite-wal *.sqlite-shm *.sqlite-journal .lexical.sqlite.rebuild-* .deferred-index.json .media-jobs.json .idempotency.json .idempotency.jsonl .voice_profiles.json .due-state.json .review-state.json .media-worker.lock ``` After migration these names no longer exist under the vault, so the patterns are inert on current versions — they only protect against re-introduction. ## Deploying on a second machine (multi-host) Each machine has its own Exomem process and local vault replica. Replication moves files; it does not decide which process may write. Choose the operating model that fits the downtime, automation, and operational complexity you actually want: | Model | Normal operation | What it gives you | Trade-off | |---|---|---|---| | Single host | One service and one public URL | Lowest operational overhead | The service is unavailable while that host is off or asleep. | | [Manual cold standby](runbooks/cold-standby.md) | Desktop normally active; laptop is demand-start and promoted by an explicit guarded command | One stable connector URL and a recoverable copy without an always-on lease/edge control plane | Planned handoff has bounded downtime; ambiguous/partitioned evidence refuses; forced recovery can lose or fork unreplicated state. | | Automatic writer-lease HA | Two replicas, coordinator/lease, and a stable edge route | Automatic active-writer selection and continued reads/failover under the documented HA contract | More moving parts: coordinator, lease, shared OAuth state, edge routing, and release-parity operations. | None is universally preferred. Choose single host when downtime is acceptable, manual cold standby when you prefer a deliberate recovery procedure, and automatic HA when its availability benefit is worth operating the coordination stack. ### Independent second host with its own URL If the second machine is an independent deployment rather than manual cold standby or automatic HA, give it its own public hostname, OAuth app, `.env`, and connector. A GitHub OAuth App accepts one callback URL, so create a separate app whose callback is `https:///auth/callback`; put that app's client ID and secret in the second host's `.env`. Set its own `EXOMEM_BASE_URL` and `EXOMEM_VAULT_PATH`, and add a second connector pointing at that host's `/mcp` URL. Hybrid `find` also needs the embeddings stack on each host independently. Run `uv sync --extra embeddings` there and verify the GPU path with `uv run python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_arch_list())"`. For an RTX 50-series GPU, expect `True`, `sm_120`, and the startup log line saying the embedding model is ready on CUDA; otherwise `find` falls back to keyword/BM25. ### Automatic single-writer lease and laptop takeover One managed service can also use the included coordinator locally for collection store admission. Set the explicit single-host topology in that service's environment: ```dotenv EXOMEM_WRITER_LEASE_TOPOLOGY=single-host EXOMEM_WRITER_LEASE_URL=http://127.0.0.1:8770 EXOMEM_WRITER_LEASE_VAULT_ID=main EXOMEM_WRITER_LEASE_REPLICA_ID=local EXOMEM_WRITER_LEASE_TOKEN= ``` Under this topology the service keeps the writer lease while it is idle: no other replica can reach a loopback coordinator, so an idle release would hand authority to nobody. Set `EXOMEM_WRITER_LEASE_IDLE_SECONDS` to a value of at least the TTL to release an idle lease anyway. Owner offline steps still need the service stopped first. Run the coordinator under the service manager on the same host, bound to `127.0.0.1`, with its database outside the vault. Give the coordinator the same writer credential through `EXOMEM_LEASE_COORDINATOR_TOKEN`. Keep its distinct operator credential in `EXOMEM_LEASE_COORDINATOR_OPERATOR_TOKEN` in the protected coordinator environment and offline operator storage. Ordinary service requests receive only the writer credential. A managed service that moves file collections into the store at an upgrade reads the operator credential from `operator.env` beside `service.env`, for its migrator alone, as `docs/managed-service-upgrades.md` describes. Keep the existing vault identifier, replica identifier, signing key, and FastMCP home stable. The source template is `scripts/exomem-lease-coordinator.service`. Render `__VENV_PYTHON__` with the installed interpreter, `__COORDINATOR_ENV_FILE__` with the protected coordinator environment file, and `__PORT__` with the loopback port. Render `__COORDINATOR_STATE_DIRECTORY__` and `__COORDINATOR_DATABASE__` with the persistent external directory and database path. Keep these private inputs outside the repository. The environment file must allow only its owner to read it; the unit's `UMask=0077` protects files the coordinator creates. Run systemd's native unit verification on the rendered file before installing it as `exomem-lease-coordinator.service` in the user manager. The template restarts a failed coordinator and binds only to loopback. It does not install or enable itself. Single-host accepts only a credential-free literal loopback HTTP origin, including IPv6 loopback. Paths, queries, fragments, and hostname aliases refuse. Its coordinator requests bypass proxies and refuse redirects to another origin. This restriction prevents a shared remote coordinator from silently using local session identities. A false refusal delays setup for the deployment owner. The service keeps its existing durable sessions and FastMCP OAuth storage when `EXOMEM_OAUTH_STORAGE_URL` is absent. Supplying that URL still selects shared storage and requires its existing credentials and namespace. Omitting the topology retains multi-host behavior and its shared OAuth requirement. `exomem doctor --profile ha --probe` checks the local coordinator and lease configuration without requiring an HA edge or two replica origins. It checks authenticated lease status and requires HTTP 401 for a credential-free status request. Other results fail the diagnostic, including unavailable transport. This detects missing bearer enforcement; a false refusal delays setup for the deployment owner. The store still requires its capability fence, head checks, mutation lock, verified custody, and supported reader admission. This setup does not enable store creation or migrate an existing collection. The collection store requires this coordination. On an install without a configured coordinator, including a stdio install, the migration preflight reports the vault `blocked`. Every store step refuses with `COLLECTION_STORE_COORDINATION_REQUIRED` and points to this section: migration, enrolment, a summary collection create, serving a store and adopt-local. File-mode Records and Planning collections are unaffected. The lease is replication-agnostic. It decides which replica may mutate; it does not copy vault files. This desktop/laptop example uses Syncthing, but self-hosters can use shared storage, Unison, rsync automation, Git-based replication, or another external mechanism. Followers can only serve content that mechanism has delivered, and its convergence/recovery behavior remains an operator concern. Network reachability is a third, separate layer. In a LAN-only Syncthing setup, an isolated laptop can take the lease and write its last local vault copy, but its changes do not reach the desktop until the machines share a network again. If takeover must work while the laptop is away from home, the coordinator must also be reachable from both networks (for example through public TLS, a small VPS, a managed linearizable service, or a private overlay network). Never put the only coordinator on the desktop whose shutdown is meant to trigger failover. Run one strongly consistent coordinator. The included reference service uses a transactional SQLite database on one always-on node: ```powershell $env:EXOMEM_LEASE_COORDINATOR_TOKEN = '' uv run python -m exomem.lease_coordinator --host 0.0.0.0 --port 8770 ` --database C:\path\to\writer-leases.sqlite ``` Expose that service over private TLS or a protected tunnel. SQLite is consistent on a single coordinator node but not highly available; for HA, implement the same HTTP contract on a linearizable store such as a Durable Object, transactional SQL, Consul, or etcd. A reference Cloudflare edge worker for this shape used to ship in `deploy/cloudflare-ha/`. **It has been removed.** Leaving retired routing infrastructure deployed cost a total MCP outage: the worker had no way to represent a legitimate single-node deployment, so once the second replica was decommissioned it refused the only healthy origin on every request, and the origin never learned it was being refused (#581, #550). Recover it from git history if you genuinely run two replicas; do not resurrect it by default. Nothing about the contract above depends on that worker. The coordinator and state contracts are plain HTTP, so any reverse proxy, load balancer or edge that implements them will do. Configure both Exomem hosts with the same vault ID and coordinator, but unique replica IDs. Desktop example: ```dotenv EXOMEM_WRITER_LEASE_URL=https://lease.example.com EXOMEM_WRITER_LEASE_VAULT_ID=personal-main EXOMEM_WRITER_LEASE_REPLICA_ID=desktop EXOMEM_WRITER_LEASE_TOKEN= EXOMEM_WRITER_LEASE_TTL=30 EXOMEM_WRITER_LEASE_PREFERRED=1 # Required when both replicas serve one stable OAuth/MCP hostname. EXOMEM_OAUTH_STORAGE_URL=https://exomem.example.com EXOMEM_OAUTH_STORAGE_NAMESPACE=personal-main EXOMEM_OAUTH_STORAGE_TOKEN= ``` The coordinator's `EXOMEM_LEASE_COORDINATOR_TOKEN`, each replica's `EXOMEM_WRITER_LEASE_TOKEN`, and `EXOMEM_OAUTH_STORAGE_TOKEN` must be the same non-empty value. `exomem doctor --profile ha --probe` proves that anonymous state reads receive `401` and an authenticated absent-key read succeeds; a coordinator that accepts anonymous state access is not safe for session authority. Laptop uses the same values except `EXOMEM_WRITER_LEASE_REPLICA_ID=laptop` and omits the preferred flag. Both remain readable. The first write acquires the lease; the server renews it while alive. After a crash or sleep, another host can take over after the TTL. Writes fail closed with `WRITER_COORDINATOR_UNAVAILABLE` if authority cannot be confirmed, but reads continue locally. Check any replica with: ```powershell uv run python -m exomem coordination_status --json ``` That command probes the configured vault's local process-safe OS boundary, not just the calling process's memory. A held boundary includes only bounded holder diagnostics and `verified: true` when the runtime-state sidecar is bound to the current lock generation. `verified: false` means an external holder is real but its identity could not be safely attributed. Vault paths, content, credentials, and tenant identifiers are never included. This local boundary coordinates service and worker processes on one host; it does not replace the remote writer lease between replicas. REST callers should attach one stable `Idempotency-Key` to every mutation attempt; CLI callers can set `EXOMEM_IDEMPOTENCY_KEY`. Exact same-key/same-payload retries return the stored canonical terminal, including its original request and receipt identity; reusing a key with different input is rejected. Explicit completed and recognized committed-failure receipts are retained locally for 24 hours. The default successful product response is compact and decisive (`ok`, `status="committed"`, `mutated`, path(s), request/receipt identity, and warning count); `response_detail="full"` or `"legacy"` changes only presentation and is excluded from mutation identity. MCP does not expose the REST idempotency header through its tool schema. Exomem therefore derives a privacy-safe local replay identity from the verified principal, hashed bearer, or MCP session plus the exact command and arguments. Its completed-result window is 600 seconds (10 minutes). Each MCP call also gets a UUIDv4 `request_id` generated locally unless a valid `x-exomem-request-id` arrives; that ID is correlation, not a substitute for the replay key. Never change the payload or create a fresh explicit identity to recover a pending or committed-uncertain acknowledgement: retry only with the same identity as directed, or reconcile. A REST write without an `Idempotency-Key` carries no replay identity: two identical requests are two writes, and a busy boundary refuses it with a retryable `MUTATION_BUSY` after one bounded acquire wait, without waiting out contention. Send one `Idempotency-Key` per intended write to have the server wait out ordinary contention and to let a resend attach to the first attempt. The bundled Studio does this for every action. Expected MCP operation refusals such as `MUTATION_WARMING` and `MUTATION_BUSY` are normal tool content: inspect top-level `success: false` and the structured `error` object, then follow its retry fields. They are not transport failures. A `receipt_id` is diagnostic, not a caller-supplied cross-session replay key; effective retry identity remains the explicit idempotency identity where supported or the same bounded principal/bearer/session scope plus unchanged command payload. Idempotency records and credentials stay in per-machine runtime state outside the synced vault. Both the 24-hour explicit window and the 600-second inferred window bound completed and recognized committed-failure receipts per replica. Pending or committed-uncertain markers fail closed until local reconciliation rather than silently expiring into another write. The lease prevents concurrent writers, but neither cache supplies cross-replica exactly-once durability if a retry reaches another replica after the original disappears. On Windows this runtime state is also principal-private. A LocalSystem service and a normal-user CLI cannot share one exact DACL or DPAPI receipt envelope. Keep direct user maintenance quiescent with the service stopped; separate state directories are an ownership boundary, not permission to create two concurrent mutation locks for one vault. The reference Cloudflare edge therefore treats `tools/call` as an ambiguous side-effect boundary. `ORIGIN_TIMEOUT_MS` remains the short connectivity window for discovery/initialization traffic, while `MCP_TOOL_TIMEOUT_MS` defaults to 60 seconds for actual tools. While a lease holder exists, a tool call is sent only to that replica and is never replayed to the passive origin after a timeout or 5xx response. Once the lease expires, the edge probes both origins, chooses one runtime-compatible replica, and forwards the next tool call exactly once. Runtime admission uses the content-free `/health/ready` contract: supported behavioral contract, stateless HTTP transport, expected replica identity, healthy writer coordination, and takeover eligibility. Admission is cached in the Durable Object against the active lease holder and fencing token, so steady-state tool calls do not pay another origin probe. The active origin renews its lease independently while a long tool runs; correctness does not rely on ordering `MCP_TOOL_TIMEOUT_MS` against `EXOMEM_WRITER_LEASE_TTL`. Both replicas must also use the same stable `EXOMEM_BASE_URL`, GitHub OAuth client ID/secret, `EXOMEM_GITHUB_USERNAME`, `EXOMEM_GITHUB_USER_ID`, and `EXOMEM_JWT_SIGNING_KEY`. Durable session records are Fernet-encrypted on the replica before leaving it. Session and auth-generation reads are always direct to the coordinator: there is no read-through mirror or stale local fallback that could resurrect a revoked session. ### When files change before a migration finishes `exomem collections migrate --apply` publishes its store before it installs the marker. Until the marker is installed, the files stay canonical. If an apply stops in that window and a writer outside the writer lease, such as a sync client or an editor, then changes an imported file, the next apply does not admit the published store, because admitting it would lose that change: - On a vault that no store served before, the apply imports the change into a new store and moves the coordinator's fence to it at the next generation. Its report names the replaced store under `superseded`, and the new store keeps that store's replica as `.foreign-*` evidence. - On a vault whose store already served collections, the apply first restores that store, with its rows and its head, and then migrates again with the change. Its report describes the restore under `restored`. When the migrated store's replica or head already left this host, the restored store continues as a new tenure, as adopt-local continues one. To finish the migration: 1. Keep the Exomem service stopped on every host that serves the vault. 2. Run `EXOMEM_LEASE_COORDINATOR_OPERATOR_TOKEN= exomem collections migrate --vault --apply`. 3. If the apply reports `status: restored`, wait until the writer lease expires, then run step 2 again. 4. If the apply reports `COLLECTION_STORE_SYNC_PENDING`, wait for the vault replica, start and stop the service once, then repeat step 2. 5. If the apply reports `COLLECTION_STORE_DIVERGED`, run `exomem collections adopt-local --vault --why "" --dry-run`, apply that preview with `--preview-id`, then run step 2 again. 6. If the apply reports `COLLECTION_STORE_MIGRATION_STALE`, put back the files of each collection that it names, then run step 2 again. 7. Start the Exomem services. The apply reports `marker_admitted` when the migration is finished. The apply never adopts a replica itself. The service adopts the vault replica when it starts, so step 4 starts it once on this host. `status: restored` means that the restore finished but could not hand back its lease. `COLLECTION_STORE_SYNC_PENDING` means that the coordinator's head or the vault replica is neither the restored store nor the migrated store, usually because a newer vault replica has not reached this host yet. `COLLECTION_STORE_DIVERGED` means that the restored store records a divergence that adopt-local must resolve first. `COLLECTION_STORE_MIGRATION_STALE` means that the apply cannot prove that the migrated store is the restored store plus only the migration's own imports, so it restored nothing. The stores, the marker, the fence and the files stay as they were. When the apply found the changed files, the refusal names the manifest of each collection that holds them. Put back those files with the bytes that the migration imported, for example from a backup or the vault's sync history. An apply that stops during a restore, after the migrated store reached the vault replica or the coordinator, leaves the restored store refused with `COLLECTION_STORE_SYNC_PENDING` until the next apply finishes the restore. An apply that finished its restore and then stopped, for example on a failed round-trip proof, leaves the restored store serving. The services may run meanwhile, and the next apply migrates that store with the writes it served. ### Summary collections A summary collection is a NEW Records collection whose rows live in the SQLite collection store outside the vault. It holds large preserved exports, such as a health export, and answers daily count, sum and average queries from rollups. Existing Records and Planning collections stay in their files; a summary collection never migrates them. The store requires writer-lease coordination (see "Automatic single-writer lease and laptop takeover"). A summary collection whose derived rows are missing or were built under an earlier layout rebuilds them from its import log at the service's next start. The rebuild gives way to requests between batches. The owner creates a vault's first summary collection offline, which enrols the vault's store with the coordinator: stop the service, then run `exomem collections create --manifest-path --manifest-file --why ""` with the service's lease environment and `EXOMEM_LEASE_COORDINATOR_OPERATOR_TOKEN` set. Later summary collections are created through `manage_memory` with `records.create` on the running service. To turn summary collections off for one vault, stop the service and run `exomem collections rollback --vault ` with the same environment. The rollback is one-way: it keeps every row, history entry and preserved source, pauses import jobs and refuses summary reads with `COLLECTION_STORE_DISABLED`. A later release does not turn the vault's summary collections back on. ### A release that changes the collection store's schema A release can change the schema of the live collection store. 0.119.0 changed it from 10 to 11, but 0.119.0 cannot migrate a store that an earlier release wrote. From 0.120.0, the first service that holds the writer lease migrates the store before it serves the store's collections. The migration keeps every row and the store head, so the coordinator's recorded head still matches it. Until a writer has migrated the store, the new release does not serve its collections: - In a handoff, the earlier release's worker keeps the lease and serves until it stops. The new worker refuses store reads and writes with `WRITER_LEASE_REQUIRED`, and it migrates the store once it acquires the lease. - An offline owner command that only reads the store, such as `exomem collections backup`, refuses with `COLLECTION_STORE_SCHEMA_PENDING`. Start the service once to migrate the store. `exomem collections create` holds the writer lease itself, so it migrates the store before it creates. File collections and knowledge writes are not affected. If the migration fails, it rolls back and leaves the store at its earlier schema. The service status then reports the store as `schema_pending`, with the error in its reason. The service refuses the store's collections with `COLLECTION_STORE_SCHEMA_PENDING`. A shutdown or an idle release hands the writer lease back at once, not at its expiry. The service tries the migration again on the next use, at least 10 seconds later. A rollback to the earlier release is safe in that state. Releases 0.114.0 through 0.118.x read the store only up to schema 10, so they cannot open a migrated store. 0.119.0 and later read schema 11, so they stay valid rollback targets. The managed supervisor refuses an upgrade target that cannot read the live store when the vault's marker routes collections to the store, or the store records a pending intent. It compares the newest store schema that the target reads with the schema of the live store, and it refuses before the serving worker stops. The serving worker keeps serving. The supervisor runs this check, so the supervisor's release decides it, not the worker's. A supervisor started on 0.120.0 or earlier does not have the check. After a seamless upgrade replaces its worker, it still accepts 0.114.0 through 0.118.x as a target until the service unit restarts. The check refuses nothing in these cases: - Without a live store, as in a copy of the vault on a fresh state root, the supervisor has no schema to compare. - After an export to files, no collection routes to the store. The exporting host sets its store aside, and every other host keeps its store until `exomem collections export --retire-local`. An earlier release is then the way back. - If the supervisor cannot read the store's schema, the target meets the store as any release would. A refusal would also block the release that repairs it. Started by hand on a migrated store, a release from 0.114.0 through 0.118.x cannot open the store. On a vault with a collection store fence, its writer gets no lease, so every write on that host refuses with `WRITER_LEASE_REQUIRED`. Roll forward to recover. From schema 12 on, the service also records a state compatibility ID for each store schema, such as `collections-store-schema-12`, in the vault's state manifest before it migrates the store. A failed migration removes the ID again. A release that does not support the ID then refuses at its own startup. Schema 11 has no ID, because every release since 0.114.0 treats an unknown ID the same way, so an ID would also refuse 0.119.0 and later. If an earlier release refuses at startup with `STATE_MIGRATION_MANIFEST_INVALID` or `STATE_COMPATIBILITY_UNSUPPORTED` after a rollback, the vault holds state that the release cannot use. Roll forward to a release that supports it. If you run 0.119.0 on a store that an earlier release wrote, 0.119.0 cannot back it up: its backup refuses with `COLLECTION_STORE_SCHEMA_PENDING`. In step 1, back up the store with the release that you ran before 0.119.0, then upgrade straight to 0.120.0 or later. To upgrade a vault with a collection store to a release that changes its schema: 1. With the release that you run now, back up the store: `exomem collections backup --vault --to `. 2. Upgrade the service. 3. Read a store collection, for example with `query_records`. The read answers with the collection. The backup in step 1 is a copy at the earlier schema. It does not make a rollback safe, because restoring it loses every store write after the upgrade. ### Export the vault to files before a downgrade A release from before the collection store cannot open a vault whose collections live in the store. Before a downgrade, the owner exports the vault: the export turns every store collection back into its file layout and then retires the store. Only the owner can export, with the service stopped on every host and the operator credential, `EXOMEM_LEASE_COORDINATOR_OPERATOR_TOKEN`, set. 1. Stop the Exomem service on every host that serves the vault. 2. Preview the export: `exomem collections export --vault --dry-run` 3. If the verdict is `blocked`, resolve each collection that the preview names, then preview again. 4. Note the `plan_id` value in the preview. 5. Apply it: `EXOMEM_LEASE_COORDINATOR_OPERATOR_TOKEN= exomem collections export --vault --plan-id --why ""` 6. If the apply stops before it reports `exported`, run step 5 again with the same `plan_id`. 7. If the apply reports `COLLECTION_STORE_EXPORT_STALE`, start again at step 2. 8. If the apply reports `COLLECTION_STORE_COORDINATOR_UPGRADE_REQUIRED`, upgrade the coordinator, then start again at step 2. 9. If a rerun reports `COLLECTION_STORE_EXPORT_VIEW_EDITED`, `COLLECTION_STORE_EXPORT_UNVERIFIED` or `COLLECTION_STORE_SYNC_PENDING`, follow "When an export cannot finish" below. 10. On every other host, after the exported vault reaches it, run `exomem collections export --vault --retire-local`. 11. Install the older release on every host, then start the Exomem services. The services then read every collection from its files. The `--retire-local` step needs only the writer-lease credential, because it changes the host that runs it and reads the coordinator's fence without moving it. The preview blocks the whole vault while any one collection cannot leave the store, and names the collection and the reason: - The store created the collection, so file mode never held it. This rule stands until the owner decides whether rollback must cover such collections. - The rendered files exceed a file-mode bound: 2,000 item files, 512 KB in one item, 8 MB in one collection or 2 MB in one log. - A value or a held correction has no file form. - A person changed a store view since the store published it. Restore the view's bytes, or wait for the release that classifies view edits. A collection that the store never wrote keeps its files byte for byte. The export renders a collection that the store wrote from its rows, and records one content-free checkpoint in the activity log, `log.md`, that names the store transitions it summarizes. Inspection then reports `acknowledged_gap` for that collection. After it verifies that file mode reads every collection as the store served it, the export records that result in its intent. Then it retires the coordinator's store fence through the operator-only `POST /v1/vaults//collection-store-fence/retire`. Then it renames `/_Collections/mode.json` (`Knowledge Base` by default) to `mode.json.exported-`, renames the replica and the live store to `collections.sqlite.exported-`, and removes the store's state compatibility descriptors. No host adopts the renamed files; they stay as evidence of the export. An apply that stops after the fence retire finishes these renames without reading the files again, so a file-mode edit made after the retire does not stop it. First it reads the coordinator's newest retired fence through `GET /v1/vaults//collection-store-fence/retired` and checks that the recorded head is this store's own. A retired fence alone never lets a rerun retire the marker: after a fence retire made by hand, the rerun first writes and verifies the files. The retire and the lease revocation are one coordinator transaction. The lease holder at that moment loses its fencing token, so a store writer that opened against the fence cannot renew or write. A repeated retire returns the same answer and does not revoke the lease again. After the retire, the hosts behave as follows: - A host whose vault copy routes no collection to the store acquires the lease as on a vault that was never enrolled. - A host whose vault copy still has the store marker does not serve the store. This release refuses its store requests with `COLLECTION_STORE_RETIRED` until the exported vault reaches that host. - A store-serving client from a release before the retire, such as 0.117, cannot read a retired fence. It fails closed with `WRITER_COORDINATOR_UNAVAILABLE` and does not serve the store. The cost is that the store collections stay unavailable on that host until its vault copy no longer routes to the store. Do not roll the coordinator back to a release from before the retire once a fence is retired. An older coordinator ignores the retired generations, so it enrols the vault at generation 1 again, and a stale store marker could then bind that enrolment. The coordinator keeps every retired generation. The fence read reports `enrolled: false` with the newest retired generation, or 0 for a vault that was never enrolled. A later enrolment, such as the owner's first `exomem collections create` or `exomem collections migrate --apply` on an exported vault, takes the next generation with a new store identity. Run that enrolment with a release that has the retire, because an older release cannot read a retired fence. An enrolment request based on an older generation gets HTTP 409. A vault that was migrated again after an export has fence generation 2 or later in its marker. Only the coordinator that holds that fence can serve it. A replaced coordinator, or one restored from a backup older than that migration, does not hold the fence. The hosts then refuse the store with `COLLECTION_STORE_MARKER_CONFLICT`, because only a marker at generation 1 may make the first fence cut on a new coordinator. Keep the coordinator database in the backups, and restore a copy taken after the latest migration. A running coordinator keeps recent grants and fences in its `-wal` file, so the database file alone is not current. Back it up with SQLite's online backup API, for example `sqlite3 writer-leases.sqlite ".backup copy.sqlite"`. A plain file copy is complete only after a clean stop; after a crash, copy its `-wal` file with it. #### When an export cannot finish An apply that stops leaves its intent in the live store. Once the apply has published the replica, the replica carries the intent too. The store then keeps every row and refuses writes. The services refuse its collections with `COLLECTION_STORE_EXPORT_PENDING` until the export finishes or the owner abandons it, on this host and on every host that adopts the replica. - `COLLECTION_STORE_EXPORT_VIEW_EDITED` names each view that changed after the export began. To export without the change, put back the bytes of each named file, for example from the vault's sync history, then run the apply again with the same `plan_id`. A rerun accepts a view that holds the bytes the store published or the bytes this export wrote. To keep the change for a later decision, abandon the export. - `COLLECTION_STORE_EXPORT_UNVERIFIED` means that file mode does not read a collection as the store served it. A rerun repeats the same check, so it cannot finish. Abandon the export, then file a bug report with the refusal text. - `COLLECTION_STORE_SYNC_PENDING` means that another host adopted the vault replica and wrote rows after the export began, and this host's store does not have them. Keep the services stopped. Keep both stores, `mode.json` and the coordinator's fence as they are, because the other host's store holds the newer rows. File a bug report with the refusal text. To abandon an export: 1. Keep the Exomem service stopped on every host that serves the vault. 2. Run `EXOMEM_LEASE_COORDINATOR_OPERATOR_TOKEN= exomem collections export --vault --abandon --plan-id --why ""`. 3. Start the Exomem services. The services serve every store collection again. The command reports `abandoned`. The vault stays routed to the store, and the store keeps every row that it served before the export. The abandon rewrites no vault file. Each file that the export already wrote keeps the export's bytes, and the store records those bytes as that file's view. A view that a person edited keeps the edit, and the store adopts no part of it. Before a later export, restore the bytes of each edited view, or wait for the release that classifies view edits. Then start again at step 2 of the export steps. The abandon refuses with `COLLECTION_STORE_RETIRED` when the coordinator's fence is already retired, because the store cannot serve again without a new enrolment. After a fence retire made by hand, run the apply again instead: it writes the files from the store, verifies them and finishes the renames. Do not retire the fence by hand. If the apply after a retire made by hand refuses, no step continues the export or abandons it. Keep the services stopped, keep `mode.json`, the live store and the fence as they are, and file a bug report with the refusal text. #### When the exporting host loses its live store after the fence retire The export verified every file before it retired the fence, so only renames remain. Do these steps only when the export itself retired the fence: 1. Keep the services stopped on every host. 2. Pick one time for both renames, in the form `YYYYMMDDTHHMMSSZ` (UTC). 3. Rename `/_Collections/mode.json` to `mode.json.exported-