# @devfn/testing Deterministic fixture manifests, temporary projects, and fake listeners for allocator, lifecycle, crash-recovery, and CLI contract tests. ## Networking qualification harness `@devfn/testing/networking` holds the versioned matrix (`devfn.networking-qualification/v1`) that DevFn's networking contract must pass on real hosts, and the harness that runs it. Each scenario names its deferred-matrix row, the prerequisites it needs, the assertions a live run must observe, the disposable resources it may create, and the local contract tests that cover the same behavior. Those tests are fixture evidence only; they never count as a live pass. The rows are: native and Compose endpoint wiring before start, two simultaneous worktrees, hostname/path isolation, internal TLS and registered domains (1); Google OAuth sign-in in two worktrees through one relay redirect URI (2); Cloudflare and ngrok reserved/ephemeral public journeys (3); startup failure, concurrent change, crash, one-worktree stop, proxy reload and Cloudflare, ngrok reserved and ngrok ephemeral tunnel recovery (4); and the opaque-owner endpoint interface (5). The DEV-2 networking outcome needs every scenario passed on both macOS and Linux at the exact head (`gate`). From a built checkout (`npx turbo run build --filter=@devfn/cli --filter=@devfn/testing`): ```sh node devfn/testing/dist/networking/cli.js plan [--approval approval.json] [--browser-operator] node devfn/testing/dist/networking/cli.js run --head --out macos.json [--scenario ]... [--row ]... [--approval approval.json] [--browser-operator] node devfn/testing/dist/networking/cli.js gate --head macos.json linux.json ``` `plan` only probes. `run` drives the built DevFn CLI against a new state directory under a temporary run root, using generic fixture repositories and worktrees. It exits 0 only when every selected scenario passed. `run` refuses to start unless `--head` is the HEAD of a clean checkout in which every tracked `devfn/*/src` file has a `dist` output at least as new, and `--devfn` names a file inside that same checkout; `git` is run only from an absolute `PATH` directory. The receipt records that provenance and `gate` requires it. Every DevFn command of the run, including the deliberately killed start, and the Caddy it starts get `XDG_DATA_HOME`, `XDG_CONFIG_HOME` and `XDG_STATE_HOME` under the run root, so the run's internal CA and Caddy configuration never touch the operator's. A DevFn `up` or `restart` is killed only after 15 minutes, above every fixture's summed health budgets plus an image pull, so a slow but healthy start never dies holding its lifecycle lock; other commands are killed after 3 minutes. - A scenario is `passed` only when its driver observed every planned assertion and none failed, every resource it created was removed and read back absent, and the operator's own DevFn state and the OS trust stores are unchanged. Each operator allocation, route, registered domain and tunnel binding is compared by name and by a digest of its material fields, so a retargeted route or a changed binding under the same name is a change; on macOS the user and admin trust settings are compared too, and on Linux every file under the trust locations, including nested directories and the certificates symbolic links name. An unreadable trust entry fails the protected read; it is never read as unchanged. Otherwise it is `failed`. `blocked` means a prerequisite or driver was unavailable (`EXTERNAL_ACCEPTANCE_GATE: …`), and `unsupported` means the host is not macOS or Linux. Neither is a pass, and Windows is never qualified. - Prerequisites are read-only probes that fail closed. They check Node.js 22+, the built CLI, git, Caddy (on Linux, also its existing permission to bind 80 and 443), and `lsof` and `ps` for process read-back. The DevFn proxy ports (2019, the probed platform's listener ports and 8444) must be free, because another runtime holding them is never touched. They also check Docker Compose against a local engine only: the endpoint from `DOCKER_HOST` or the current Docker context must be a local socket (both `DOCKER_HOST` and `DOCKER_CONTEXT` set is refused, since Docker lets `DOCKER_HOST` override the named context), and the run pins it for every DevFn and `docker` command (remote targets are a separate capability). They also check the approved resources; an approved Cloudflare tunnel must have no connected connector, read with the approved token from `GET cfd_tunnel/{id}/connections` (clients with live connections) and the tunnel status from `GET cfd_tunnel/{id}`; a healthy or degraded tunnel that lists no client is unreadable, never free, because Cloudflare spreads a tunnel's requests over every connected connector. An approved Cloudflare tunnel therefore serves one run at a time: the macOS and Linux runs of one approval run one after the other, and the operator's own DevFn connector for that tunnel is stopped first. Before and after each step that starts, swaps or recovers a Cloudflare connector, the run reads the connections again and stops the scenario when a connected connector is not one of its own. Its own are identified by the connector ID each cloudflared the run state records reports on its metrics `/ready`, never by when a connector appeared. The harness never installs tools, trust roots, port permissions or resolver entries. A registered domain must already resolve only to loopback. - Every resource is recorded in `/inventory.json` before it is created, and only inside the approved scope: the run root, an approved domain, or a run-prefixed binding or hostname. Cleanup runs even when a driver fails. Nothing is reaped before the read-back that would report it. A lifecycle is read back after `devfn down`, and every check runs even when `devfn status` answers `degraded` or `failed` (DevFn reports those with `ok: false`): a state other than `stopped`, a process that outlived it, a lease, a route of its owner or a Compose container or network of its owner fails the scenario. DevFn's Compose stop leaves the project network, so once no container of the owner remains the run removes that owner's networks before the read-back and records each one in the receipt limitations; the run's shared Caddy, OAuth relay and tunnel connectors are left to their own resources while they are recorded and still match their recorded birth identity. A process listing counts only when `lsof` completed it (exit 0, or 1 for a process the run cannot inspect). A killed, timed-out or truncated listing, or a checkout that cannot be read for any reason other than being absent, is a failed read-back, never absence. The relay and connector processes recorded before `oauth-relay stop` or `tunnel unregister` must be gone afterwards, and a record unreadable at that point leaves the resource unconfirmed, and a binding is absent only when its provider holds nothing carrying its marker. A public name's read-back counts only that name's own ngrok endpoints, plus any carrying the binding's marker without a readable URL (and, for an ngrok ephemeral name whose URL was not assigned yet, every endpoint carrying the marker), so the surviving route of a sibling worktree on the same binding is not reported as the withdrawn name's residue; the binding read-back still rejects every endpoint carrying its marker. A checkout that still runs processes is kept and fails its read-back. A public name is recorded before `devfn up` can publish it (for an ngrok URL assigned after start, the origin name DevFn derives for the route, with the URL attached once assigned). It is absent only when DevFn no longer routes it and a provider read-back (Cloudflare DNS, or the ngrok endpoints and tunnels at the name or its URL) holds nothing for it, and its URL gets a conclusive refusal: the provider edge's own error (a Cloudflare 1xxx error from Cloudflare's server, or an ngrok `ERR_NGROK` error), the DevFn origin's empty deny, or a system lookup and a direct DNS query that both answer the name does not exist. An unreadable provider, and a failed, timed-out or incomplete probe or any other answer, is never absence. A tunnel recovery likewise counts the killed connector's route as unserved only on the edge's own refusal. Run-level resources (the run's Caddy and any process left in the run root) are read back into the receipt, and leftovers, including Compose containers and networks of the run's own owners, are removed only after that. On SIGINT, SIGTERM or SIGHUP the run writes no receipt, and an interrupt that arrives while the receipt is written or finalized withdraws it (exit 130). A second signal exits at once without cleanup (exit 130), but first withdraws any receipt this run published (only its own; anything else is put back) and only then releases its claim; a receipt it cannot withdraw is reported. Its report, and any receipt or lock it could not remove, is appended to `/forced-exit.txt`. Signal messages on stderr are best-effort and never waited on: a closed, full or stopped stderr (a terminal paused with Ctrl-S included) loses them, never that withdrawal, the release, a later signal or exit 130. Each receipt write and withdrawal completes before a signal is handled, so none is left half done. Only SIGKILL or a crash can leave a published receipt, and then its lock stays too. It first kills in-flight DevFn commands and HTTP probes, refuses any further creation or probe and ends every wait, waits for the running scenario to return, then runs one cleanup and read-back of what the ledger lists and reports what remains. The run root is kept for inspection when anything is not confirmed absent. - Errors are never evidence: a timeout, refused connection, DNS failure or response cut before it was complete leaves an assertion failed or unobserved. Every HTTP probe settles within an overall deadline (`REQUEST_DEADLINE_MS`), however long its socket keeps sending, and a request Node.js refuses to build rejects at once without leaving its deadline armed. A not-ready or stopped claim needs a readable `devfn status` that names the checkout's own owner (exit 0, a lifecycle state, its instance ID); an unparseable, timed-out or killed command reads nothing. A provoked DevFn failure counts only as DevFn's own report of a shipped code (exit 1, `ok: false`): `DEVFN_START_FAILED` for the failing start, `DEVFN_ALREADY_RUNNING` or `DEVFN_REGISTRY_LOCK_TIMEOUT` for the losing concurrent start. Fault injection signals only a process that still matches its recorded birth identity: the connector DevFn recorded, and the crash target the run recorded while its lifecycle was ready. An exited, reused or unverifiable PID is never signalled, and its assertion fails. A Cloudflare connector ID counts as the run's own only when it is read from the metrics port of a connector record whose process still matches that identity. Each check passes only on the shipped component's own answer: the public origin's 403 with `Devfn-Public-Origin: deny` for a route's own name, its 421 for a forged Host and its marked `fallback` 404, the local proxy's empty 404 without that header (a marked public-origin 404 is not a local one), Caddy's 308 to HTTPS, the relay's 403 `DEVFN_OAUTH_RELAY_REJECTED: replayed`, and for a mismatched Host the provider edge's own refusal (a Cloudflare `error code: 1xxx` answer other than 1033, or a 4xx with an `ngrok-error-code` header). The exported `ANSWERS` predicates are tested against the failure answers. - A recovery scenario that claims an unaffected lifecycle is preserved (`sentinel-preserved`: startup failure, concurrent change, crash, one-worktree stop and proxy reload) watches every service of that lifecycle. Before the scenario's transitions it records the web and api processes with their birth identity, the `devfn url` URLs and the owner's proxy routes. While they run it probes the web route and the api path route every `WATCH_PROBE_INTERVAL_MS` for each service's own answer, then probes once more. The claim passes only when every probe answered, the status reads `ready`, and every process, URL and route is the one recorded. Routes are compared by every material field (hostname, path, match, prefix stripping, target and TLS, everything but their timestamps), and each probe also checks the path the service received (`/` for web, `/watched` for api behind its stripped `/api` route); a route that misses any probe round fails it even when it answers again before the end. Proxy reload's `reload-preserves` counts only probe rounds taken during its four route changes. - The receipt (`networking-qualification-contract`, mode 0600) records the host, provenance, prerequisites, each observation and cleanup read-back, run-level cleanup, the endpoint interface version and the limitations of every provider binding the run registered. Limitations are informational. Errors the run itself hit outside any scenario go to the always-present `runErrors` list, and any entry makes the receipt unusable for `gate`; `run` exits 0 only when that same receipt check finds no problem. Run errors include an unhandled rejection, a failed ledger write and a failed step such as the root removal. A reader sees a complete receipt or none, never a partial one. After the reap, the root removal and one macrotask turn, it is rewritten when a run error arrived after it was written. A receipt that cannot be brought up to date is removed. A failed ledger write never stops cleanup: every remaining resource is still removed and read back, and the last complete ledger stays on disk. Should the run end with an exception, such as a receipt that cannot be written, it still retries the cleanup of whatever is not confirmed absent and reaps the leftovers. Credential values from approved variables are redacted. `--out` is a receipt file name (such as `macos.json`), never a path: receipts are kept in `~/.devfn-networking-qualification`, which `run` creates with mode 0700 and otherwise accepts only as a real directory of the same user that nobody else can write. `run` prints the receipt's full path, which `gate` (or a copy) takes. A run attempt claims its name with an exclusive `.lock` and holds it until it exits, so a second run on the same name refuses to start and never withdraws or replaces the first run's receipt. A lock left by a killed run also refuses; remove it only once no run uses that name. An earlier receipt under the name is withdrawn as soon as `run` has validated `--head` and `--out`, before scenario selection, prerequisite probes, provenance, Docker resolution or the run root, so a rerun that fails any of them or is interrupted never leaves it; a receipt that cannot be withdrawn starts nothing, and `plan` and `gate` never touch the directory. Only a receipt is ever removed or replaced, decided on the entry the run took rather than on an earlier look at the name: a withdrawal first moves the entry to a private name and puts back anything that is not a receipt (another file, a directory, a symlink, or a FIFO or other special file, which is never opened), and a write links the new receipt in only where nothing is, so an entry that appears meanwhile is never overwritten. Rewrites and late withdrawals take only this run's own receipt. `gate` accepts a passed scenario only when the receipt records each prerequisite its matrix entry names as satisfied, exactly once. The approval file names nonproduction resources and the environment variables that hold credentials, never their values. Because it authorizes provider operations, it is read like a receipt: `--approval approval.json` names a regular file in `~/.devfn-networking-qualification`, which must already exist as that harness-owned directory (`plan` creates nothing). A path, a symlink or any other entry is refused (a FIFO or device at once, never waited on), and a refusal states a fixed reason, never the file's content. Unknown fields (listed by the fields allowed, never by the file's own keys), production, expired approvals and malformed names are refused: ```json { "version": 1, "nonproduction": true, "approvedBy": "owner", "expiresAt": "2026-12-31T00:00:00Z", "domain": "dev.example.test", "oauth": { "provider": "google", "clientIdEnv": "QUAL_GOOGLE_CLIENT_ID", "clientSecretEnv": "QUAL_GOOGLE_CLIENT_SECRET", "relayHost": "auth", "redirectUri": "https://auth.dev.example.test:8443/oauth/callback" }, "cloudflare": { "domain": "preview.example.test", "accountId": "<32 hex>", "zoneId": "<32 hex>", "tunnelId": "", "credentialsFile": "/abs/tunnel.json", "apiTokenEnv": "QUAL_CF_TOKEN" }, "ngrokReserved": { "domain": "ngrok.example.test", "authtokenEnv": "QUAL_NGROK_AUTHTOKEN", "apiKeyEnv": "QUAL_NGROK_API_KEY" }, "ngrokEphemeral": { "authtokenEnv": "QUAL_NGROK_AUTHTOKEN", "apiKeyEnv": "QUAL_NGROK_API_KEY" } } ``` The OAuth redirect URI must be the relay origin on this host plus the fixture callback path: `https://./oauth/callback`, with port 8443 on macOS and the default port on Linux. Both ngrok entries need an API key variable, which the run uses only to read endpoints back. With `--browser-operator`, the run prints each worktree's login URL and waits up to 10 minutes. A person or browser agent completes the Google sign-in, and opening the URLs needs an explicit trust decision in that browser for the DevFn internal CA. The fixture apps report outcomes and session hosts, never codes or tokens. A consumed callback URL stays in the app's memory until the harness replays it once, as soon as the app reports it and well inside the relay ticket lifetime; nothing writes it to disk or the receipt. The crash scenario holds the last node of its interrupted start before it listens (a `hold-` file), so the kill always lands before the start is ready, records every process that start launched with its birth identity before the kill, and requires each to be gone afterwards (exited, or its PID now another process's), and the startup-failure scenario fails the last node in plan order only after it reached the node DevFn started before it, so the rollback of a running sibling is observed. The crash scenario waits out the five-minute stale window of a killed start's lifecycle lock, and so does an interrupted run's cleanup when the interrupt killed a start.