# fcvm Development Log ## ALWAYS FIX FAILING TESTS. PERIOD. **If ANY test fails, FIX THE ROOT CAUSE.** No exceptions. No workarounds. No weakening assertions. - Never add flags like `--no-cache` to avoid failures - Never weaken assertions to accept broken behavior - Never skip, ignore, or comment out failing tests - Always fix the actual bug in the code This is non-negotiable. A test failure means the CODE is broken - fix the code, not the test. ## ZERO RACE CONDITIONS. ZERO. **Never write code that can race.** Not "unlikely to race." Not "only races under load." ZERO. - If two processes can touch the same file, use `flock()` or atomic rename - If two threads can touch the same state, use a mutex or atomic - If setup runs at shutdown, it WILL race with the shutdown sequence — handle it - If a container build runs in parallel tests, it WILL collide — serialize or lock it - "Works most of the time" means "broken" — races are bugs, period The stale `db.sql` disaster happened because setup "usually" cleaned up before shutdown recreated the file. "Usually" is not "always." One race condition broke every VM boot across all CI runners. **Before committing, ask:** "Can any other process/thread/test touch this state at the same time?" If yes, add synchronization or make it atomic. No exceptions. ## WRITE PLAINLY, ENGINEER TO ENGINEER Commit messages, PR descriptions, code comments, docs: all of it is one engineer telling another what changed and why. Say that and stop. - State the change, the reason, and the evidence. Nothing else belongs there. - No selling. Cut "worth having", "the real win", "makes it trustworthy", "elegant", "powerful", "seamless", "comprehensive", "robust". - No throat-clearing. First sentence names the change. Skip the scene-setting paragraph. - No rhetorical scaffolding: "Two things follow from that", "Four properties make this work", "Here's the thing". Make the point instead of announcing it. - No staccato triples ("No traceback. No crash. No exception."). One normal sentence. - No em dashes. Comma, parenthesis, or a new sentence. - Don't narrate the journey. "I first tried X, then found Y" belongs nowhere. Describe the end state. - Don't restate the diff in prose. If the code already says it, delete the sentence. - A claim needs a number or a command. "Faster" is not a claim; `2.4s -> 0.3s` is. - Read it back as if a colleague sent it to you. If it reads like marketing or an essay, rewrite it. ## SESSION VALUES STAY OUT OF THE REPOSITORY A value copied from a working session (a shell, an environment variable, a launch command or a transcript) is not useful to the repository and goes stale: profile names, account names, one-off paths, model names, local ports. Don't put one in a file, a comment, a commit message, or a pull request title, description or review comment. Use a variable or a made-up placeholder (`example-profile`, `example-name`). Where something specific has to be named, use its publicly documented identifier. Check the diff, the commit messages and the PR text before every push. ## SUSPENDING A VM DOES NOT DISTURB IT **`snapshot create` is a read of the source VM, not a reconfiguration of it.** A caller asked for a copy. They must not get their running VM altered to produce one, and nothing on the snapshot path may leave state the resume has to repair. What the boundary needs is narrow, and both halves are non-destructive: | need | mechanism | reversible? | |------|-----------|-------------| | no NEW socket may appear while the cookie dump runs | directional NEW-flow REJECT gate (iptables/ip6tables) | yes: its chains and hooks added, then removed | | packets already in receive processing must drain | AF_PACKET open/close, whose release runs `synchronize_net()` | nothing to reverse | | restore must destroy exactly the snapshot-time sockets | dump kernel socket **cookies** into a manifest | read-only | The dump has to happen at snapshot time and cannot move to restore: by the time a clone runs, the restored workload and an early published-port client can both have created sockets, so a restore-time dump cannot describe a fixed generation, and a four-tuple is not a socket identity (95ce328a). ### Never take the link down. Never purge kernel tables. `ip link set eth0 down` was once part of this sequence. It prevented sockets the gate already rejects, and paid for it by purging the device's **routes and neighbours**. Everything purged then had to be remembered and reinstated by hand, and that list was wrong twice: - **2026-08-10 (`ee3f5c30`)** the route table. *"A restored clone came up reachable at L2 and routed nowhere."* Fixed by capturing and reinstating routes. - **2026-08-15** the neighbour table, which the same link-down purged and nobody reinstated. After a snapshot the guest re-ARPed for the bridge address every health check and published connection arrives from. Both the bridge (which owns it) and pasta (which answers for the subnet it routes) reply, so the guest kept whichever landed first. When pasta won: ``` SYN br0-mac > guest-mac 10.0.2.1 > 10.0.2.100 [S] SYN-ACK guest-mac > 9a:55:9a:55:9a:55 10.0.2.100 > 10.0.2.1 [S.] <- pasta RST 9a:55:9a:55:9a:55 > guest-mac 10.0.2.1 > 10.0.2.100 [R] ``` The guest answered the wrong MAC and pasta correctly reset a connection it had never opened. A published port that accepts and then returns zero bytes, about 1 snapshot in 10, with **no drop counted at any layer** - because nothing was dropped. Latency was bimodal (421 requests under 0.1s, **zero** between 0.1s and 4.5s, 8 at the 5s deadline), which is what ruled out contention. Two bugs, one shape: *the link-down purged X and nobody put X back.* The fix was to delete the purge, not to grow the repair list. Nothing in the boundary now touches the link, so there are no routes and no neighbours to reinstate. **Rules:** 1. **Nothing on the snapshot path may down a link, flush a table, or otherwise destroy kernel state in the guest being snapshotted.** If a step needs quiescence, use a reversible gate. 2. **A repair list is a smell.** Needing to remember what a step destroyed means the step is wrong. The next table nobody thought of is the next outage. 3. **The bridge MAC is pinned** (`NAMESPACE_MAC`, `src/network/pasta.rs`) so a clone inheriting the source's neighbour table holds entries that are correct in its own namespace. fc-agent hardcodes the same constant; the pair is asserted by `namespace_neighbour_matches_host_constants`. 4. **A restored clone resolves none of its gateways.** fc-agent pins 10.0.2.1 and, in routed mode, the IPv6 gateway `fd00::1` to that MAC (permanent) right after flushing the inherited neighbours and before the IPv6 swap. A resolution of `fd00::1` in flight across the swap was sourced from the address the swap deletes, its answer was not delivered, and every reply waited a second for the retransmit (#1160, about 6 restores in 1000 with a client dialling during the restore). `test_routed_clone_answers_without_resolving_its_ipv6_gateway` holds that interleaving; `the_routed_gateway_is_pinned_before_the_ipv6_swap` pins the order. **Enforcement:** `snapshot_leaves_the_source_network_untouched` (`tests/test_snapshot_clone.rs`) diffs the guest's routes, neighbours and addresses across one snapshot and fails on any difference. The `ModelBarrier` and `ModelDiag` invariants in `fc-agent/src/snapshot_network.rs` fail if any boundary step takes the link down, on the source or the clone path. ### On restore, the gate's egress half opens before the stale sockets are retired The restored clone inherits the gate closed in both directions. Every daemon whose connection the cleanup retires reconnects at once, and a connection the gate answers with a TCP reset is a definitive refusal to its client, not a retry: ServiceRouter marks the host down, the configerator clients fall back to system configs. With the restore hook inside that window (4 to 35 s, where the window had been about 0.1 s) a www clone served 500s for as long as it lived (2026-10-14, `Sitevar FBLOGGER_CONFIG could not be loaded from Configerator` on every request, with a no-op hook as well as the real one). So the OUTPUT half of the gate is removed as soon as the receive barrier has drained, before the first socket is retired (a socket opened from then on is not in the manifest, so the cookie-bound retirement never touches it), the restore remounts NFS and waits for the exec server and the egress proxy, the hook runs last, and the INPUT half is removed after it: publication. `ModelDiag` fails if a socket is retired with egress closed, `ModelHook` if the hook runs with egress closed or after ingress opened; `the_restore_hook_runs_after_the_nfs_remount_and_the_proxy_waits` pins the source order of `handle_clone_restore`. The guest's clock step belongs on the same side of that line. The restore sets the guest's wall clock to the host's time, a jump of the snapshot's age, and every daemon whose lease or ticket ran out in the jump reconnects at once. Stepped before egress opened, those connects met the gate's reset: a www clone of a 9.5-hour-old snapshot was not healthy 10m38s in, when the restore script stopped waiting (falcon_proxy timed out on its SMC tiers, configerator fell back), and the same snapshot restored with its clock held back was healthy in 2m05s and 2m15s (2026-10-25). So the clock steps after the boundary opens egress and before publication; `the_clock_steps_after_the_egress_gate_opens_and_before_publication` pins it. chronyd can step the clock too (fc-agent configures `makestep 1 -1`), so the gate's OUTPUT chain also rejects the guest's NTP requests (UDP to port 123, and no other UDP), from the snapshot until egress opens; `the_egress_gate_rejects_ntp_and_no_other_udp` pins it. The rejected send fails with EPERM, so a source VM's chronyd logs at most one failed poll per snapshot, and both the source and a restored clone ask chronyd for a burst (`chrony_burst`) once egress is open again. ### Opening the gate costs no grace period on the restore path Each half of the gate opens with one nf_tables commit covering both families, sent by fc-agent over nfnetlink (`fc-agent/src/nftables.rs`), not with an `iptables-restore` per family. Releasing a NETLINK_NETFILTER socket after a commit that deleted rules waits one RCU grace period (`nft_rcv_nl_event` flushes the pending destruction, which starts with `synchronize_rcu()`) while holding the namespace's commit mutex; so does releasing any AF_PACKET socket (`packet_release()` calls `synchronize_net()`). Measured in restored clones, that wait was 26, 46 and 79 ms at 2, 4 and 8 vCPUs, and it was the whole 110-260 ms the gate used to cost (#1151). So fc-agent never releases those sockets: it keeps one nfnetlink socket and one send-only AF_PACKET socket for its whole life. Releasing them on another thread is not a fix: a process forked before the release holds a reference until its exec closes the CLOEXEC descriptor, so that process makes the final release and waits the grace period inside its own exec (the `iptables -S` after the ARP probe went from 2-3 ms to 27-75 ms). The receive barrier is the one release that must stay: its grace period is the barrier. `the_gate_socket_is_kept_and_never_released`, `frames_are_sent_through_the_socket_that_is_never_released` and `opening_each_half_is_one_transaction_for_both_families` pin it. ### The source's clock is stepped forward before its gate reopens On the arm64 `nested` (NV2) profile the Firecracker fork freezes the guest's counters while a VM is paused, so a resumed source was behind the host by the pause (1.56 s after a 1.6 s pause) until chronyd slewed or stepped it back. Every reopen after a pause now passes the host's clock (`resume_paused_source`, `src/commands/common.rs`) and the guest steps to it before the gate lets chronyd's polls out (`resume_source_network`), so chronyd never measures the lag. The step is forward only (`bootplan::advance_system_clock`): the host's instant is a lower bound on the present, and the `default` profile keeps the clock running through a pause. `test_source_clock_matches_the_host_after_snapshot_create` measures it; `every_reopen_after_a_pause_sets_the_sources_clock` pins the host side. ## MAXIMUM REUSE / CACHEABILITY **The whole goal of fcvm is to cache and reuse as much as humanly possible.** Kernels, rootfs images, initrds, container exports, and snapshots are content-addressed precisely so nothing is built or booted twice when it doesn't have to be. - **No snapshot-cache opt-outs unless absolutely required.** An invocation may skip the cache only when a cached artifact would BEHAVE differently for it and the difference cannot be reconciled at restore time. Every entry in `snapshot_cache_opt_out` (`src/commands/podman/mod.rs`) carries that justification next to it; an entry without one is a bug. - **Flags that change WHERE something binds are honored at restore time, not by bypassing the cache.** Example: `--vsock-dir` makes a cache hit place the clone's listener in the caller's directory (through the load's vsock override, or the restore mount redirect on Firecracker before 1.16.0); it does not force a cold boot. - **A value a restored VM can be given is set at restore time, not keyed.** Example: the snapshot key holds what a restore cannot change about the balloon, which is the device's existence (`--balloon`) and its free page reporting switch (`--free-page-reporting`), and not the target. A cache hit sets its own target on the loaded, paused VM before the guest resumes, so runs at different targets share the pre-start snapshot. The startup snapshot holds a workload that initialised under one target, which a restore cannot replay, so its name carries the target. - **Never add per-caller data to the content key when restore can reconcile it.** Fragmenting the key silently kills sharing between behaviorally identical artifacts. - Before adding an opt-out or a key ingredient, ask: can the restore path absorb this difference? If yes, that is the fix. - **A key names content, not a path, where the snapshot keeps its own copy.** A read-only `--disk-dir` is keyed on its tree's hash, taken in place so a hit copies nothing; a cold boot builds the image from a private copy with that hash (`src/storage/disk_dir.rs`). ## VERIFY PLAN AND RUN NEW TESTS LOCALLY **Before committing, ALWAYS:** 1. **Re-read the plan** — check every numbered step and verification item against what was implemented 2. **Run new tests locally** — don't just compile them, actually execute them and verify output 3. **Check for dead code** — search for old function/field names that should have been removed Writing tests without running them is pointless. Compilation does not equal correctness. A test that passes `--no-run` might fail at runtime due to logic bugs, missing `-T` flags, wrong paths, etc. **Anti-pattern:** "All 85 unit tests pass" + never ran the integration tests that actually exercise the new code paths. ## A DEFECT CLAIM IS CLOSED BY A RED TEST, NOT BY A FIX **Whenever anyone — a reviewer, a bot, a teammate, you — says "this is broken", the thing that closes it is a test that FAILS WITHOUT THE FIX.** Not the fix. Not "verified manually". Not a green suite after the change, which proves only that the suite never covered it. Applies to every source of a defect claim, not just review comments: a CI failure, a bug report, a hunch you had in the shower, a comment you wrote yourself. **The procedure, in order:** 1. Write the test. Run it against the **unfixed** tree. **Watch it fail.** 2. Apply the fix. Watch it pass. 3. Revert the fix once more and confirm it goes red again — if you skipped step 1, this is your last chance to learn the test was vacuous. 4. Only then resolve the thread / close the issue, citing the test by name. A test written after the fix, never observed failing, is indistinguishable from a test that cannot fail. This repo has repeatedly found checks that could never fire — a contention detector matching a truncated `comm`, a leak check whose pattern never matched, a `"VM exited"` branch logged 0 times across 137 runs, a `grep '^ *FAIL'` blind to nextest's `TRY 1 FAIL`. Every one of them was green for its whole life. Enforcement: `scripts/check-review-threads.sh ` fails while any review thread is unresolved, and while any finding carries no disposition, whether it arrived inline, in a PR-level review body from anyone but the PR author, or in a top-level PR comment. One vocabulary for all of them: A head that no reviewer has anything to say about is still reviewed: the gate counts a listed review bot's (`VERDICT_BOTS`) no-findings result as coverage of the head when the bot names the head. Codex names the commit it reviewed in two shapes, and both count: the "Didn't find any major issues" comment, and the review-summary comment it edits in place, whose table row must say Completed, name the head, and carry a datetime after the head arrived. That comment is created before any review has finished, so neither of its own timestamps dates a result, and a table holding any row that is not a finished review covers nothing. Every one of those datings is a comparison of parsed instants, not of strings: the row datetimes carry fractional seconds and the check-suite timestamps do not, and as text a fraction sorts before the "Z" it replaces. A timestamp the gate cannot parse dates nothing, so it covers nothing and answers nothing. CodeRabbit's "Full review finished." reply names nothing and never covers on its own; what names the head is the walkthrough comment CodeRabbit edits in place, whose recent-review block says which commit range it reviewed. That block must say no actionable comments were generated, the comment must carry no CodeRabbit notice (failed, skipped, paused, rate limited, in progress), the range must end at the head, and the edit must postdate the head's arrival at GitHub. Both of those signals live in a comment its bot edits in place, so the gate re-reads the PR comments after all its paging and blocks when one of them changed while it was reading: the head standing still does not mean a clean walkthrough is still clean. Both bots' no-findings replies need no disposition. Greptile is not a verdict bot, and neither its score nor its prose names a clean result; what does is the "Greptile Review" check run the greptile-apps app writes on the commit it reviewed. The latest such run on the head must have finished with success and a summary saying 0 comments added (success alone is not clean: a review that placed a finding still concludes success), after the head arrived, and it must read the same on the gate's second read of the head's check suites. "Latest" is by start time, so a run whose start time will not parse leaves the runs unordered and nothing covers. A check-suite or run list that does not account for every entry blocks, and once a Greptile suite is present both lists must carry a numeric count to account with. Greptile's summary comment then needs no disposition when its footer names the head and it carries no comments-outside-diff block. A PR with no review result of any kind is an unreviewed head, not a clear one. | `RED-VERIFIED: ` | a defect claim, closed by a test watched failing without the fix | | `NOT-A-DEFECT: ` | naming, docs, style | | `DISAGREE: ` | a defect claim you are rejecting, with reasoning | Three rules the gate enforces, each of them a hole it once had: - **A disposition must OPEN a reply of its own.** Matched anywhere in the text, "Please add `RED-VERIFIED: ` before resolving" counted the reviewer DEMANDING evidence as the person supplying it. - **An answer must be NEWER than the claim it answers.** Counting dispositions in aggregate meant one reply closed every finding posted after it, forever. - **Review bodies answer to the same vocabulary as inline findings.** A bare acknowledgement used to clear a PR-level defect claim that, posted inline, would have demanded a real disposition — the same finding adjudicated or waved through depending on where the reviewer happened to click. - **All three containers count.** A finding can arrive as an inline thread, a review body, or a plain top-level PR comment (`gh pr comment`). The last of those was invisible to the gate for a while, which meant a claim posted exactly the way the docs suggested could sit on a PR reporting CLEAR. - **But not everything is a claim.** An APPROVED review is GitHub saying the reviewer is not asking for changes. A listed notification bot's comment is not a finding, nor is a documented trigger command (`@codex review`, `@coderabbitai review`, `@greptileai`). Nor is a REVIEW BODY FROM THE PR AUTHOR: a review is the channel this gate names for answering, so counting the author's own review as a claim made every acknowledgement create the obligation it was posted to discharge, and the count grew by one each round (#874 was carrying 4 such bodies, #872 five). The author's TOP-LEVEL COMMENT is still a claim, because that is where a self-reported defect lands. None of this re-admits `REVIEW-ACK:` as a disposition; it answers nothing, it has simply stopped being a question. Treating every non-empty body as a finding blocked PRs that had none at all, `@codex review` included, which is the command the docs tell you to post. - **A reviewer saying it did not run is not a claim either.** Out of quota, Codex's bot posts "You have reached your Codex usage limits for code reviews" as a top-level comment; CodeRabbit answers a trigger it cannot serve with "Review rate limited." under an "Action not completed" fold, and until a review has run its summary comment holds only a "Review limit reached" or "Review skipped" notice. Both bots are listed reviewers, so none of that was a notification bot's comment, and the gate counted each one as a finding: on 2026-09-02 #898 carried three Codex quota notices reported UNANSWERED, aws #46 one more, and the author posted a `NOT-A-DEFECT:` review to answer text that claimed nothing. A listed bot's comment whose whole body is one of those notices, matched line for line, now needs no disposition and covers no head. The same words with anything added, or from any other account, stay claimable, and a summary comment that has grown a walkthrough is a walkthrough. Greptile is not a verdict bot, and its own notices get the same treatment: a top-level comment under `` saying the file limit or an error stopped the review, and, out of open-source review credits, a review whose whole body says reviews are paused. That pause notice is a non-empty review body on the head, and a non-empty review body covers its commit, so once answered it marked the head reviewed by a reviewer that had said it would not review. Its text covers nothing now, whoever posts it. `.greptile/config.json` keeps Greptile's findings in comments and threads the gate reads (not the PR description), posts its result as a check run, leaves auto-approval off, and reviews when a PR opens and on `@greptileai`, not on every push, because each review spends a free open-source credit. `greptile_config_keeps_its_results_where_the_gate_reads_them` pins it. - **The head commit must have been reviewed.** Answering every finding proves nothing if nobody reviewed the code you are merging. This gate shipped with five open findings that way: the branch was pushed, the prior round was answered, the gate went CLEAR, and it merged 36 seconds later while the reviewer was still working. Nothing was bypassed — the gate simply had no notion of review *coverage*. A rule was also **withdrawn**: requiring the disposition to be the last word in a thread. It could not distinguish a new defect claim from "Thanks, confirmed", so it blocked adjudicated threads. It traded a rare fail-open for a routine fail-closed, and a gate that cries wolf on ordinary conversation gets switched off — after which it catches nothing. Earlier versions also guessed which findings were defects from a `panic|crash|leak|...` regex. That failed BOTH ways: "this drops the final game from the schedule" matched nothing and closed on an assertion, while a *wrong* defect claim could never be closed at all. Do not guess; answer. The gate has its own tests, each written against the unfixed script and observed failing first. There are two harnesses and only one of them runs in CI, so run both before pushing a gate change: ```bash bash scripts/test-check-review-threads.sh # shell fixtures, not run by CI make test-unit FILTER="-E 'binary(test_log_scan)'" # tests/test_log_scan.rs, run by CI ``` `make test-unit FILTER=log_scan` runs 0 tests and exits 4: FILTER is passed to nextest as-is and no test NAME contains that substring, so the plain form selects nothing while looking like a pass. `scripts/probe-review-gate-pagination.sh` covers the live oversized-thread path that fixtures cannot reach. CI state cannot tell you whether a finding was answered. ### "LOAD RELATED" IS NOT A DIAGNOSIS. NEITHER IS "FLAKY", "TIMING", OR "CONTENTION" **These words describe the absence of an explanation, not an explanation.** They are what an investigation says when it has stopped. Every one of them has a measurement attached, and the measurement is cheap, so take it before saying the word: | the claim | what must be shown | |---|---| | "it was load" | `/proc/loadavg` AND per-process CPU **at the moment of failure** - captured by the failing run, not inferred afterwards | | "it was slow" | the operation completing at all, with its duration; "it appeared moments later" is worthless if the diagnostic itself provoked the appearance | | "contention" | a latency distribution that DEGRADES CONTINUOUSLY. Contention produces a spread; a broken path produces two clusters | | "flaky" | a failure rate, and a mechanism for the failures | Load produces a spectrum. A broken path produces a step function. That single test settles most of these arguments for free, and it is one `awk` away from data you already have. Observed 2026-08-15, three times in one investigation, all three wrong: - Claimed the published-port flake was contention. The distribution refuted it: **421 requests under 0.1s, ZERO between 0.1s and 4.5s, 8 at the 5s deadline.** Perfectly bimodal. The real cause was a purged neighbour entry, a binary state. - Claimed a box was "quiet" because I had stopped my own work. I had not stopped the other tenants (three `claude` sessions and two `codex` processes were running), and had never measured. Separately quoted `loadavg 6.43` as evidence of CPU load when the visible processes summed to well under one core of a 64-core box: loadavg counts uninterruptible I/O, so it was mostly btrfs. - Claimed a restored clone was "merely slow to announce", from a forensics dump taken AFTER the failure - a dump whose own probing can create the very entry it then reports. The actual error said `neighbour: ""` and `probe: (silence)` for a full 5 seconds. Nothing was slow. Something was broken, and a timeout increase built on that theory would have hidden it. **A timeout increase is the most dangerous form of this.** It converts a defect into latency and ships. Raise a timeout only when you can show the operation COMPLETING at a measured duration, and say the duration in the commit message. "It needs more time" without a number is a guess wearing a fix's clothes. ### A FUZZ CATCH IS A LEAD, NOT A REGRESSION TEST **When the chaos fuzz (or any stochastic harness) finds a defect, the fix is not done until a DETERMINISTIC test fails on it.** The fuzz found the bug by landing inside a timing window. It will pass ten, twenty, fifty times before it lands there again, so leaving it as the guard means the next regression ships and is discovered by a flake weeks later. The fuzz's job is DISCOVERY. Something else has to hold the line. Convert every catch into a test that fails **every run**: | the catch | what pins it | |---|---| | an ordering window (A must happen before B) | a source-level ordering assertion, or an event-log model like `ModelBarrier`/`ModelDiag` in `fc-agent/src/snapshot_network.rs` | | a state invariant (X must survive Y) | compare X directly across Y, as `snapshot_leaves_the_source_network_untouched` does | | a race that needs a precise interleaving | a named failpoint case in `test_lifecycle_interleave.rs` (crate `failpoint/`) | | a constant that two files must agree on | read the other file and assert, as `namespace_neighbour_matches_host_constants` does | Observed 2026-08-15, twice in one change: - CI's fuzz failed with `guest 10.0.2.100 never appeared at L2`, while the forensics dumped moments later showed that guest `REACHABLE`. The guest was not missing, it was LATE: removing the snapshot boundary's link cycle also removed the thing that made a restored clone announce itself, so the host's namespace had not learned its MAC before verification gave up. The fuzz hit that window; `the_clone_announces_itself_before_the_boundary_publishes` now fails on the ordering itself, every time. - The same run's seed had passed **20 consecutive local hunts** on a different seed. A stochastic pass count is not evidence about a code path; it is evidence about the schedule that seed happens to generate. **Corollary: do not certify a fix with the fuzz configuration you were already hunting with.** Run the configuration CI runs (`SEEDS=1 OPS=10` is the default), and run it enough times to say something. A green hunt on the seed you have been staring at proves the least interesting thing available. ### A GATE MUST FAIL CLOSED — check your dependencies before you trust your verdict A check that cannot run must **block**, never pass. Passing is a claim, and a tool that could not evaluate anything has no basis for making it. This bit immediately, in the gate written to enforce the rule above. `jq` is not in the CI container. Every `jq` call failed to stderr, the counts came back empty, and the script printed `verdict: CLEAR ... exit 0` — waving every PR through *precisely because it could not evaluate them*. Strictly worse than having no gate, because it looks like one: ``` review threads: total, unresolved verdict: CLEAR — every thread resolved, ... check-review-threads.sh: line 49: jq: command not found ``` Any script that renders a verdict must begin by proving it can: ```bash for tool in jq gh; do command -v "$tool" >/dev/null 2>&1 || { echo "BLOCKED: '$tool' missing" >&2; exit 2; } done ``` And its tests must run **where CI runs it**, not only on a dev box that happens to have the tooling. A green unit test on your laptop says nothing about the container. ### HOW TO GET CI LOGS — there is always a way; never report "no logs available" "The logs aren't available yet" is almost always a tooling mistake, not a fact. Two independent routes, in order of preference: **1. Per-job API — works even while the RUN is still in progress.** ```bash JOB=$(gh api repos/{o}/{r}/commits//check-runs \ --jq '.check_runs[] | select(.conclusion=="failure") | .id' | head -1) gh api --allow-escape-sequences repos/{o}/{r}/actions/jobs/$JOB/logs > job.log sed -i 's/\x1b\[[0-9;]*m//g' job.log ``` `--allow-escape-sequences` is **mandatory**: job logs contain ANSI, and without it `gh api` writes nothing to stdout and puts the reason **only on stderr** — so `> job.log` yields an empty file from a command that looks like it succeeded. That empty file is what makes people announce there are no logs. Note `gh run view --log` refuses while the RUN is in progress even for jobs that finished long ago. Do not wait for the run; use the per-job API above. **2. SSH to the self-hosted runner — it has the checkout AND the environment.** ```bash gh api repos/{o}/{r}/actions/runners --jq '.runners[] | "\(.name)\t\(.status)\t\(.busy)"' # names are runner-i-; get IPs read-only: aws ec2 describe-instances --instance-ids i-... \ --query 'Reservations[].Instances[].{id:InstanceId,pub:PublicIpAddress,arch:Architecture}' --output text ssh -i ~/.ssh/runner_key ubuntu@ # the CI tree, exactly as the job saw it (merge commit of PR head into base): cd /opt/actions-runner/_work/fcvm/fcvm/fcvm && sudo git log --oneline -1 ``` This is how a `cargo fmt --check` failure was diagnosed in one command after the API route had been (wrongly) given up on. `_diag/Worker_*.log` is runner METADATA — step output is not there; either use route 1 or re-run the failing command in that checkout. **Check `busy` first and leave a busy runner alone** — it is executing someone's job, and a build you start competes for its cargo cache and disk. ### "It is too big / slow / expensive to test" is almost always false That excuse is how the worst bugs stay uncovered, because expensive-to-reach paths are exactly where nobody looks. Find the cheap equivalent: - **Large files → sparse files.** The FUSE `remap_file_range` u32 truncation (`kernel/patches/0001-fuse-add-remap_file_range-support.patch`) only bites above 4 GiB, because `fuse_write_out.size` is a `u32` and the client saturates it — the destination inode records ~4 GiB and later guest reads come back short. Sounds like it needs a 4 GiB fixture. It does not: a sparse file costs no real blocks and on btrfs the reflink is O(1). `truncate -s 5G` + FICLONE reproduces it for free. - **Slow timeouts → inject the signal.** Use the failpoint harness (`make fuzz`, `FAILPOINT` specs) instead of waiting out a real timeout. - **Rare races → make the interleaving deterministic.** A seeded schedule beats hoping. - **Huge memory → test the arithmetic.** Feed the boundary value (`u32::MAX`, `pid_max`, a 9-digit `pid_start_time`) to the function directly rather than provisioning the machine that would produce it naturally. - **Multi-hour soak → assert the invariant, not the duration.** If a leak takes 6 hours to be visible, count the resource instead of watching the clock. If after genuinely trying you cannot make it cheap, say so **in the test file**, with what it would take — never only in a commit message, where the next reader will not find it. ## WHAT CI CAPTURES WHEN SOMETHING FAILS The rule behind all of it: **a failure must carry the evidence needed to attribute it.** Every unattributable failure this repo has chased ended the same way, with the state gone by the time anyone looked, and the argument settled by whoever sounded most confident. Host side, captured today: | What | Where | Why it is there | |---|---|---| | `sar -q` / `sar -u`, whole day | `host-load.log` artifact | sysstat already runs on every runner and nothing read it. 10-minute samples, so it answers "was the box busy", not "at this millisecond" | | `pidstat -u -h 30` during the suite | `pidstat.log` artifact | `sar` is system-wide and the post-step snapshot is taken after the suite ends, so neither can say WHICH process consumed the host at the moment of a timeout | | `/proc/loadavg` inline in the readiness error | the error text itself | so "the box was busy" is checkable in the message, without fetching anything | | filtered `dmesg` | `dmesg-filtered.log` artifact | host OOM, KVM, page-fault signatures | | D-state watchdog | live step log | prints nothing when healthy; an ephemeral runner dies with its artifacts, so the live log is the only channel that survives | Guest side, pushed by fc-agent (`fc-agent/src/vitals.rs`): | What | When | Why that field | |---|---|---| | `MemAvailable`, `Committed_AS` | every 10s, and in full at failure sites | a 1024 MiB guest can fail a fork or an mmap with nothing in dmesg | | `pty/nr` vs `pty/max` | same | conmon allocates a PTY per `-t` exec; this is the authoritative free count | | `statvfs` of `/run` | same | rootless podman's runroot is `/run/user//containers`; ENOSPC there means conmon could not create its socket or exit file and therefore wrote NOTHING, which is issue #841's exact signature | | `oom_kill` from `/proc/vmstat` | same | monotonic, cannot wrap, unlike the 128 KiB printk ring | | cgroup `pids.current` / `pids.max` | same | the pids controller rejects forks with no message the caller sees | | which threads are runnable or blocked, the vsock-named workers' state and stack, virtio interrupt totals, per-CPU ticks | once a second while runnable plus blocked threads reach three per CPU (six at least), every 10s after 30s of that | the exec stalls of #938 read 12 to 18 runnable for 20 s and the 10s line could not name them; exec cannot reach a guest in that state | **It is PUSHED, never pulled, and that is not a style choice.** The obvious design is to run `fcvm exec --pid P --vm -- sh -c '...'` from the host at the failure site. That cannot work for the cases it exists for: serving an exec makes fc-agent FORK, so a guest out of pids or threads fails the collection for the same reason it failed the operation. It also cannot reach a VM whose fcvm already exited, an L2 guest, or a test nextest killed. Everything in `vitals.rs` is `read`/`readdir`/`statvfs` from an already-running process, so it still answers when the guest can no longer create one. Routing is by prefix and it is load-bearing (`console_line_is_important` in `src/utils.rs` classifies console lines for both backends, and checks this prefix before anything else, because a pile-up line can name a thread called `fc-agent`): `[fcvm-vitals]` is DEBUG, so the 10s sampler lands in the per-VM file and NOT the job log, which is what lets it run for every VM without flooding a runner. `[fc-agent] VITALS` is INFO, so the on-failure block also reaches the test's captured output. A guest kernel crash headline (panic, Oops, BUG, a WARN, a stalled CPU, task or RCU, an OOM kill) is WARN under `fcvm::guest`, apart from the console's own `firecracker` and `cloud-hypervisor` targets, so it survives fcvm's default filter and RUST_LOG=fcvm=info: a guest that panics leaves nothing else that says why (on x86 Firecracker logs only `Unexpected exit reason on vcpu run: Shutdown`; on arm64 the guest resets through PSCI). Rules learned the hard way, each from a check that could not fire: - **Never conclude "no OOM" from a job log.** A job log keeps no guest console line but a crash headline: the console's `firecracker` target is outside `fcvm`, so a filter on fcvm's targets drops even fc-agent's INFO lines. Lines below the guest's console level (4, between early boot and the poweroff) never reach the console, console output can be lost or run into another line, and host `dmesg` only reaches an artifact. Measured: `grep -c "DEBUG firecracker"` on a 259,570-line job log returns 0. Render resource verdicts against `/tmp/fcvm-test-logs/*`, and fail closed when the artifact is missing. - **Prefer `/proc/vmstat`'s `oom_kill` to dmesg.** It is monotonic and cannot wrap. The guest's printk ring is 128 KiB (`CONFIG_LOG_BUF_SHIFT=17`) and a chatty boot does wrap it, so a dmesg-only OOM check intermittently cannot fire. - **Never `statvfs` a path you have not proved is local.** On a wedged FUSE mount it parks in uninterruptible sleep and no timeout can cancel it, turning a diagnostic into a second hang. Read `/proc/self/mountinfo` first and skip anything that is not tmpfs/ext4/btrfs/xfs. - **Never read `/proc//cmdline` in a wedge scan.** It faults the target's mm and hangs on exactly the processes you are trying to inspect. - **A collector that cannot run must say so.** Silence from a diagnostic is indistinguishable from a clean result, which is the same bug as a gate printing CLEAN because `jq` was missing. ## HOW TO INVESTIGATE A TIMEOUT A deadline miss says one thing only: the answer did not arrive in time. It does NOT say the peer was slow, and it never says the box was busy. Work the list in order; each step is a measurement, and any of them can end the investigation. 1. **Get the load, do not assume it.** Every runner runs sysstat, so the record already exists: `sar -q -f /var/log/sysstat/sa
-s HH:MM:00 -e HH:MM:00` for load average and run queue, `sar -u` for idle/steal. CI captures both (`Capture host load record`) and the readiness error embeds `/proc/loadavg` inline. Run 31906708922 was **95.1% idle, run queue 1, 0% steal** at the failing sample -- every "the box was busy" sentence written about it was false, and checkable in one command. 2. **Compare against the healthy distribution, not against the deadline.** Pull the same measurement from a passing run. Healthy readiness there was **84ms** against a 5s deadline. Two orders of magnitude is a different state, not a slow tail, and it rules out "just raise the timeout" without further argument. Log the duration on SUCCESS so this comparison is always available (`readiness_ms`). 3. **Prove the waiter actually ran.** Count its per-round log lines across the window. Zero rounds in 5s means the loop never iterated, so every field in its error message is a default, not an observation -- `neighbour: ""` meant "never queried", not "no entry". A check that reports on work it never did is the same fail-open bug as a gate that passes because `jq` was missing. 4. **Look for evidence that contradicts the error text.** The same forensics that said the guest never answered also held a TIME-WAIT socket to that guest and a REACHABLE neighbour entry: the guest had answered. When the error and the artifacts disagree, the error is wrong. 5. **Bound every attempt, not just the total.** One subprocess that never returns will eat an entire budget and produce a message blaming the peer. Wrap each attempt (`GUEST_PROBE_ATTEMPT_BUDGET`) and count abandonments in the error. Do not trust a subprocess to honour a budget you handed it -- the probe already ran `timeout(1) bash` and still hung the caller. 6. **Kill hypotheses with an experiment, cheaply.** "netns churn stalls nsenter" sounded right; 16 churners moved nsenter from 9ms to 18ms max, so it was wrong and cost 60s to find out. A two-arm measurement beats a paragraph of reasoning. Raising a timeout is a valid fix only after step 2 shows the healthy distribution genuinely crowds the deadline. Otherwise it hides the defect and buys a slower failure. ## STACKED PRs BY DEFAULT **All work goes in stacked PRs.** Each new PR should be based on the previous one, not main. ``` main → PR#55 → PR#56 → PR#57 (correct) main → PR#55, main → PR#56 (wrong - parallel branches) ``` Only branch directly from main when explicitly starting independent work. **When base PR merges:** Your branch's merge-base with main shifts automatically. The delta shown in your PR will only be your commits (base PR's commits are now in main). Merge conflicts can arise if main got other commits touching the same files. **CRITICAL: Verify base update before merging dependent PRs:** ```bash # After PR #1 merges, WAIT and verify PR #2's base changed to main gh pr view 2 --json baseRefName # Must show: {"baseRefName":"main"} # If it still shows the old branch name, DO NOT MERGE - wait or manually update ``` **Why this matters:** If you merge PR #2 while its base is still the old branch (not main), the commits go into the orphaned branch and never reach main. You'll lose your changes. **PR description:** Always note `**Stacked on:** (PR #N)` so reviewers understand the dependency. ### A STACKED PR MUST ACTUALLY RUN CI — "no failures" is not "it ran" **A `pull_request:` trigger's `branches:` filter matches the PR's BASE branch.** So `branches: [main]` silently skips the whole workflow for every stacked PR — the exact PRs this section tells you to open. GitHub then shows a check set containing whatever *other* workflows did fire (here: `safety-check`), with zero failures. It looks green. Nothing ran. Observed 2026-08-08: #752 (base `kernel-7.0.14`) was merged on a "no failures" reading. Its head sha had **zero** runs of `lint`, `packaging`, `host`, `host-root`, `container` — the only non-skipped check was `safety-check`. It carried three rustfmt violations that surfaced the moment the change reached a PR whose base *was* main. **Before treating any PR as green, prove the checks EXIST, then that they passed:** ```bash SHA=$(gh api repos/{o}/{r}/pulls/ --jq .head.sha) gh api --paginate repos/{o}/{r}/commits/$SHA/check-runs \ --jq '.check_runs[] | select(.conclusion != "skipped") | "\(.name)\t\(.conclusion)"' | sort -u # --paginate is not optional: the first page holds 30 runs, so on a busy commit a required # job can be absent from the output while present on the PR — the check-EXISTS assertion # below then fails for a job that is actually there, or worse, passes for one that is not. # An expected job missing from this list is a FAILURE to verify, not a pass. ``` Any monitor, gate, or script that renders a verdict from `statusCheckRollup` must assert the required jobs are **present**. Counting failures and finding none is the fail-open form of the same bug as `jq: command not found` printing `verdict: CLEAR`, and as a `CodeRabbit pass` that means the reviewer never started. Enforcement: `tests/test_ci_workflow_coverage.rs` fails if `ci.yml` reintroduces a `branches:`/`branches-ignore:` filter on `pull_request`, or if a gating job leaves that file. ### CI runs on the top of a stack, not on every layer A draft PR runs no tests, and neither does a PR that another open PR from this repository is stacked on (`scripts/ci-pr-scope.sh`). Their run ends in a check named **`Summary (deferred)`**, never `Summary`, so it can never be read as a pass. Push the stack as drafts, mark it ready, and the top PR's run tests every layer: its `changes` job classifies the whole stack's diff against `main`, not only the top PR's files, so a docs-only top cannot skip the matrix for code changed lower down. When the top's `Summary` passes, the `Stack coverage` job posts a **`Stack CI`** success status on each lower PR's head, naming the run, but only on heads that are ancestors of the tested head (`scripts/ci-stack-coverage.sh`). A lower PR is tested when it carries `Summary` success or `Stack CI` success on its current head; `Summary (deferred)` alone means nothing ran for it. A lower PR defers only to a child whose head already contains its new head. Push every updated layer of a stack in one command (`git push origin layer-1 layer-2 layer-3`): a lower layer pushed on its own starts its run before its child holds the new commit, and that run tests in full. To bisect a failure on the top, run one layer on its own: `gh workflow run ci.yml --ref ` (a dispatch run tests the full matrix). Also run it when the top PR is closed and a lower PR becomes the top, since nothing re-triggers its CI. Enforcement: `scripts/test-ci-stack-scope.py` (`make test-ci-scope`, run by the `actionlint` job) drives both scripts against a fake `gh` and evaluates every job's `if:` for a deferred run. ## Fresh Bare-Metal Box Quickstart Bring up a new ARM (or x86) KVM-capable box to building-and-benching in ~20 minutes. The ordering below matters; each step's trap is noted inline. ```bash # 1. KVM access (new login session required after; a live ssh master keeps old groups) sudo usermod -aG kvm $USER # 1b. Unprivileged user namespaces. Ubuntu 24.04 ships # kernel.apparmor_restrict_unprivileged_userns=1, which makes rootless # podman and skopeo fail: `unshare -U -r true` reports "write failed # /proc/self/uid_map: Operation not permitted" and the first image export # dies with "Error during unshare(...): Operation not permitted". Every # long-lived fcvm box gets this from terraform (dev-user-data.tf), so the # requirement was invisible to anyone following these steps by hand. # # THIS IS FOR A DEDICATED fcvm OR CI BOX, and host isolation is the # precondition for the whole quickstart. The two settings are host-wide: # they let ANY local user create a user namespace and hold root inside it, # which is what AppArmor's restriction exists to withhold, and they widen # the kernel surface an unprivileged local account can reach (every # CAP_SYS_ADMIN-gated interface a namespace makes available, and the # userns-reachable bug classes behind them). On a box other people log in # to, that is a boundary you are removing for them too. fcvm cannot run # rootless without it, so the answer for a shared host is a separate box, # not a narrower sysctl. sudo tee /etc/sysctl.d/99-fcvm.conf >/dev/null <<'SYSCTL' kernel.unprivileged_userns_clone=1 kernel.apparmor_restrict_unprivileged_userns=0 SYSCTL sudo sysctl -p /etc/sysctl.d/99-fcvm.conf # 1c. AllowOther mounts. Without user_allow_other, fuse-pipe falls back to # SessionACL::Owner (fuse-pipe/src/client/mount.rs) and # `make test-unit` fails on fuse-pipe::test_allow_other # test_allow_other_with_fuse_conf, "Test requires user_allow_other in # /etc/fuse.conf". CI and both test Containerfiles set it, so only a # hand-built box is missing it. # # This is also host-wide: it lets any local user mount a FUSE filesystem # other UIDs can read, which is the point here (pjdfstest switches uid # across a fuse-pipe mount) and is another reason step 1b's dedicated box # is the precondition. # # The guard matches the directive and nothing else, because that is what # fuse-pipe accepts (`l.trim() == "user_allow_other"`, mount.rs). A prefix # match calls `user_allow_other # comment` configured while every mount # still takes SessionACL::Owner and the test above keeps failing. grep -Eq '^[[:space:]]*user_allow_other[[:space:]]*$' /etc/fuse.conf || echo user_allow_other | sudo tee -a /etc/fuse.conf # 2. Packages (Ubuntu 24.04). btrfs-progs is needed by the very next step; # bc/bison/flex/libelf-dev are the kernel fallback build's dependencies — # setup builds a kernel locally whenever a profile's release artifact is # absent, and a fresh box hits that path. libseccomp-dev is the firecracker # fork's: setup builds it from source and the link fails with # "cannot find -lseccomp" without the headers. debhelper, libdw-dev, rsync, # kmod, cpio and patch are `make install-host-kernel`'s (bindeb-pkg); that # target checks them, and 40 GiB free in its build directory, before it # compiles anything (scripts/check-host-kernel-build.sh), and builds on # /mnt/fcvm-btrfs when /tmp is on the root disk. sudo apt-get install -y build-essential git jq qemu-utils busybox-static \ pkg-config libssl-dev clang make podman skopeo uidmap slirp4netns fuse3 passt \ btrfs-progs bc bison flex libelf-dev libseccomp-dev \ debhelper libdw-dev rsync kmod cpio patch # 3. Fast local storage. Instance-store NVMe is WIPED by a cloud stop/start — # put /mnt/fcvm-btrfs on it (rebuildable cache), keep sources on the durable # root. NEVER hardcode device names: the EBS/root disk is also an nvme # device and its number is NOT stable across boxes (this box's root is # nvme2n1). The guard below refuses to run mkfs rather than printing a # warning and formatting anyway, because the failure mode is a destroyed # operating system disk. ROOT_DISK=$(lsblk -no PKNAME "$(findmnt -no SOURCE /)") mapfile -t STORE_DEVS < <(lsblk -dno NAME,MODEL | awk '/Instance Storage/ {print "/dev/"$1}') printf 'root=/dev/%s store=%s\n' "$ROOT_DISK" "${STORE_DEVS[*]:-}" [ "${#STORE_DEVS[@]}" -gt 0 ] || { echo "no instance-store devices; do NOT format anything"; return 2>/dev/null || exit 1; } for dev in "${STORE_DEVS[@]}"; do [ "$dev" != "/dev/$ROOT_DISK" ] || { echo "REFUSING: $dev backs /"; return 2>/dev/null || exit 1; } done sudo mkfs.btrfs -f -d raid0 -m raid0 "${STORE_DEVS[@]}" sudo mkdir -p /mnt/fcvm-btrfs && sudo mount "${STORE_DEVS[0]}" /mnt/fcvm-btrfs sudo chown $USER:$USER /mnt/fcvm-btrfs # 4. Rootless UFFD (snapshot restores). The device is 0600 root by default, # so every rootless clone restore fails with "Error accessing # /dev/userfaultfd: Permission denied" until this runs (CI does the same): [ -e /dev/userfaultfd ] || sudo mknod /dev/userfaultfd c 10 126 sudo chmod 666 /dev/userfaultfd sudo sysctl -w vm.unprivileged_userfaultfd=1 # 5. Rust. If ~/.cargo is a dangling symlink left from a wiped NVMe (make # check-disk moves it there), recreate the target first: mkdir -p /mnt/fcvm-btrfs/cargo curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y # 6. Sources on the DURABLE disk. Cargo.lock pins the FUSE forks; sibling # checkouts are for local dependency development and container mounts. mkdir -p ~/src && cd ~/src git clone https://github.com/ejc3/fcvm.git git clone https://github.com/ejc3/fuse-backend-rs.git git clone https://github.com/ejc3/fuser.git # 7. Build + full setup (downloads released kernels, builds pasta + the # firecracker fork, creates the rootfs; ~10 min on 64 cores when the kernel # releases exist, plus a local kernel build when one does not) cd ~/src/fcvm && make build && make setup-fcvm ``` Run everything as the unprivileged user; `make` shells out to sudo exactly where root is needed. `sudo fcvm setup` also works — builds drop back to the invoking user and store entries are handed back to them (see src/setup/mod.rs) — but the rootless path is the default one CI exercises. Sanity check the result before benching: ```bash make test-root FILTER=sanity ``` ## Sending Email Use `aws ses send-email --region us-east-1` (recipient must be verified in SES sandbox). ## UNDERSTAND BRANCH CHAINS **ALWAYS fetch before investigating branches:** ```bash git fetch origin ``` Branches may already be merged on remote. Don't waste time on stale local state. **Run before starting work, committing, or opening PRs:** ```bash git log --oneline --graph --all --decorate | head -120 ``` Shows which branch you're on and what it's based on. **Don't confuse local vs remote:** After rebasing locally, `origin/` shows the old history until you force-push. They're the same branch at different points in time. ## fc-agent is a MUSL BINARY: check it against musl, not the host target `cargo check -p fc-agent` builds for the host's glibc target and proves nothing about the binary that actually ships in the guest, which `make build` cross-compiles for `*-unknown-linux-musl`. The two libcs disagree on real signatures: `ioctl`'s request argument is `i32` on musl and `c_ulong` on glibc, so an ioctl call that compiles cleanly on the host fails the guest build with a type error. A host-only check let exactly that reach CI, where it broke every job that builds the guest agent. Before believing fc-agent compiles: ```bash cargo check -p fc-agent --target x86_64-unknown-linux-musl cargo check -p fc-agent --target aarch64-unknown-linux-musl ``` Or just run `make build`, which does the real thing. When calling libc from fc-agent, prefer `as _` over a named integer type on any argument whose width or signedness the libc chooses. ## ALWAYS USE THE MAKEFILE **Never run raw cargo/podman commands. Use make targets.** ```bash # CORRECT make test-root FILTER=sanity make setup-fcvm make build # WRONG - bypasses setup, env vars, correct flags cargo test ... sudo cargo test ... ./target/release/fcvm setup ``` If the Makefile is missing a target or broken, **fix the Makefile** - don't work around it. ### Never share `CARGO_TARGET_DIR` across worktrees The Makefile owns Cargo target routing. It sets `CARGO_TARGET_DIR=target`, and `cargo-target-link` maps that path to a directory unique to the current worktree. The obsolete shell-profile export of `/mnt/fcvm-btrfs/cargo-target` was removed; do not restore it or export any other target directory shared by multiple worktrees. Two worktrees building the same package can produce the same test-binary path, so a shared target lets Cargo consider a sibling worktree's binary fresh and run code from the wrong checkout. Observed 2026-08-08 while verifying two stacked PRs: `cargo test --test test_ci_workflow_coverage` in worktree A printed a test that exists **only in worktree B** and silently omitted the one under test: ``` Running .../cargo-target/debug/deps/test_ci_workflow_coverage-113926a94ccc9fad test gh_existence_probes_do_not_discard_their_error ... ok <- not on this branch test result: ok. 4 passed <- summary_fails never ran ``` Red/green verification is worthless under these conditions: "it went red" and "it went green" may both be reports about code you are not editing. Use the appropriate Make target. Before believing a result, confirm `readlink -f target` resolves to this worktree's unique target directory and the `Running .../deps/` line points beneath it. ### Chromium bench harness is make-driven (bench/chromium) Drive every bench phase through make — the dependency graph lives in the Makefile, so half-set-up boxes fail at `make` time instead of mid-golden (2026-08-13: two goldens died at runtime, first on a missing fcvm binary, then on "Custom firecracker not found", because a hand-rolled runner called `reqbench.sh` directly): | Target | Deps | Knobs (command-line vars; make exports them) | |--------|------|----------------------------------------------| | `bench-chromium-request-build` | `build` | container image only | | `bench-chromium-request-golden` | `bench-chromium-request-build` + `setup-default` | `TAG=`, `HUGEPAGES=1`, `NETMODE=`, `CPU=`, `MEM=`, `GUEST_ENV=` (own `TAG=`) | | `bench-chromium-request-verify` | none (sealed bundle) | `TAG=` | | `bench-chromium-request-run` | none (sealed bundle) | `TAG=`, `BACKEND=`, `UFFD_MODE=`, `UFFD_PREFETCH=`, `REPS=`, `WARMUP=`, `ARMS=`, `RESOLVE_POLL_MS=`, `RESULTS=` | | `bench-chromium-request-diag` | none (sealed bundle) | `TAG=`, `BACKEND=`, `UFFD_MODE=`, `DIAG_URLS=`, `DIAG_REPS=` (default 3), `DIAG_EXPECT_IPS=`, `DIAG_MAX_LOAD_MS=`, `RESULTS=`. The serve always runs with `--uffd-prefetch off`, and `UFFD_PREFETCH=` set to anything but `off` is refused | | `bench-webkit-request-diag` | none (sealed bundle) | the diag knobs above except `DIAG_EXPECT_IPS=` (refused); the recipe sets `ENGINE=webkit`, default `TAG=cb-req-webkit` | | `bench-chromium-request-all` | `build` + `setup-default` | the build, golden, verify and run knobs above, one seal. It runs no diag phase, and it defaults `SETTLE_WAIT_SECS=120`, so the run's quiet gate re-samples for up to 120 s before it refuses | | `bench-chromium-hostcdp` | `bench-chromium-request-build` | host-container CDP baseline, no VM; `COMPARISON_LABEL=` (default `standalone`), `CPU_BUDGET=` (default `unlimited`), `CPUS=` (requires `CPU_BUDGET=vm-matched`), `BENCH_RESOLVE_ALL_TO=` | | `bench-chromium-fault` | `build` + `setup-default` | `FAULT_OUT=` (required), `FAULT_ARGS=` | - **verify/diag/run and bench-chromium-scale must never gain a `build` dependency.** reqbench.sh seals fcvm, fc-agent, the request harness's six sources and the scale harness's five into a hash-bound runtime bundle; the run, and the scale run through `reqbench.sh scale`, refuse a golden recorded under a different bundle hash. A rebuild, an edit to a sealed file, or a new commit between golden and run invalidates the chain: the run also refuses a golden whose recorded source_revision is not the current git HEAD. Regolden instead of working around the seal. Sealed set: reqbench.{sh,py}, reqanalyze.py, cdpdrive.py, render.py, wddrive.py, reqscale.py, reqscale_analyze.py, faulttrace.bt, guardexec.py, guardsupervise.py, fcvm, fc-agent. The Makefile and test files are not sealed: uncommitted edits to them do not affect a golden or a run in progress, but committing them before the run does. - Hugepage goldens are part of the snapshot identity: distinct tag, e.g. `make bench-chromium-request-golden TAG=cb-req-golden-huge HUGEPAGES=1` (the golden grows the pool to 2048x2MB pages and fails closed if the kernel cannot deliver them). - Quiet-box gates: `run` refuses when 1-min load exceeds 2.0 or stray fcvm/firecracker processes exist (hostcdp refuses at >= 1.0, faultbench above 1.0). Do not start builds — or ANY box-local work, including analysis scripts and background agents — while a measured run is in flight; the noop drift gate catches the contamination and voids the run (burned twice, 2026-08-13). - BACKEND=file is refused against a hugepage TAG (fcvm restores those via an implicit UFFD server, so the record would be mislabeled). The pool is grown to 4 x (guest MiB / 2) pages: at golden time from `MEM=`, and at verify, run and diag time from the snapshot's recorded `memory_mib` (falling back to `MEM=` only when the snapshot's config.json has no `memory_mib`). - The diag phase renders each URL `DIAG_REPS` times (default 3), each render on its own clone with cdpdrive's `--net-trace`. With `DIAG_EXPECT_IPS=` set, it refuses on any remote IP outside that list, on a trace in which no request names a remote address, and on any traced http(s) request that names no remote address and was not served from cache or a service worker. It also refuses on any name that did not resolve, any load event over `DIAG_MAX_LOAD_MS=`, any failed render (a record for another URL, not ok, written under a non-zero driver exit status, or without a load event timing), an unclean clone or serve teardown, or a sealed bundle that changed during the phase; `$RESULTS/diag/summary.json` holds the verdict and names the snapshot generation and config it diagnosed. WebKit renders carry no trace, so the webkit diag refuses `DIAG_EXPECT_IPS=`. `make bench-chromium-corpus` runs it after the golden's verify with the corpus URLs, `DIAG_EXPECT_IPS=10.0.2.2` (Chromium only) and `DIAG_MAX_LOAD_MS=15000` (overridable), and does not measure when it fails; `DIAG_ONLY=1` stops the campaign after golden, verify and diag. - Structural pin: `MakefileBenchGraph` in bench/chromium/test_reqbench.py (`make test-chromium-request`) — it fails if the dependency graph or the no-rebuild property of the measured phases regresses. ## NEVER ROUTE AROUND BUILD PROCESSES **If a build fails, FIX THE BUILD. Never manually copy files.** When a kernel, rootfs, or binary doesn't build correctly: 1. Fix the build script 2. Fix the source code 3. Fix the patches **NEVER:** - Manually copy files to work around naming issues - Run build scripts directly instead of through fcvm - Create symlinks to "fix" path mismatches If `fcvm setup` produces wrong output, the bug is in fcvm or build.sh. Fix it there. ## Test Helper Functions (tests/common/mod.rs) | Function | Purpose | |----------|---------| | `spawn_fcvm(&args)` | Spawn fcvm process, returns `(child, pid)` | | `spawn_fcvm_with_logs(&args, name)` | Same + debug log file | | `poll_health_by_pid(pid, timeout)` | Wait for VM healthy status | | `poll_health_status_by_pid(pid, expected, timeout)` | Wait for specific health status | | `poll_health(&child, timeout)` | Wait healthy (checks process exit) | | `kill_process(pid)` | SIGTERM then SIGKILL if needed | | `unique_names(prefix)` | Generate unique `(name, clone, snap, serve)` | | `find_fcvm_binary()` | Locate `./target/release/fcvm` | | `exec_in_vm(pid, &cmd)` | Run command in VM via exec | | `exec_in_container(pid, &cmd)` | Run command in container via exec | ## Nested Test Architecture Tests use `localhost/nested-test` container image built from `Containerfile.nested`. **Key files:** - `Containerfile.nested`: Container with fcvm, fc-agent, firecracker-nested, rsync - `tests/common/mod.rs`: `ensure_nested_image()` auto-builds via podman - `rootfs-config.toml`: VM rootfs packages (copied into container at `/etc/fcvm/`) **Package installation locations:** - Container packages: `Containerfile.nested` apt-get install - VM rootfs packages: `rootfs-config.toml` [packages] section Both need rsync for `--disk-dir` to work. ## NO HACKS **Fix the root cause, not the symptom.** When something fails: 1. Understand WHY it's failing 2. Fix the actual problem 3. Don't hide errors, disable tests, or add workarounds Examples of hacks to avoid: - Gating tests behind feature flags to skip failures - Adding sleeps or retries without understanding the race - Clearing caches instead of updating tools - Using `|| true` to ignore errors ## NEVER Parse JSON with Regex **Always use `jq` to parse JSON.** Never use grep, sed, awk, or string matching on JSON. ```bash # WRONG grep '"health_status":"healthy"' output.json # CORRECT jq -r '.[] | select(.health_status == "healthy")' output.json ``` ## Test Failure Investigation **Never say "likely" - always find the actual root cause.** When tests fail in CI or parallel runs: 1. Re-run in isolation to confirm the test itself is correct 2. Root-cause why it failed when run together 3. All tests must pass together 4. If it passes alone but fails in parallel, find and fix the race condition ## Debugging Test Hangs **When a test hangs, look at what it's ACTUALLY DOING - don't blame "stale processes".** ```bash # WRONG approach: blindly killing "old" processes ps aux | grep fcvm # "I see old processes, they must be blocking!" sudo pkill -9 fcvm # "Fixed it!" (No, you didn't debug anything) # CORRECT approach: understand what the test is doing ps aux | grep -E "fcvm|script|cat" # See: script -q -c ./target/release/fcvm exec --pid 1083915 -t -- cat # The test is running `cat` in TTY mode - it's waiting for input! # The bug is in the test, not "stale processes" ``` **Common causes of hanging tests:** - Command waiting for stdin (like `cat` without EOF signal) - Missing Ctrl+D (0x04) in TTY mode tests - Blocking reads without timeout - Deadlocks in async code **The process list tells you EXACTLY what's happening.** Read it. ## Overview fcvm is a hypervisor-agnostic VM manager for running Podman containers in lightweight microVMs. It drives two backends behind a pluggable `Hypervisor` trait — Firecracker (default) and Cloud Hypervisor — selected via `--hypervisor firecracker|cloud-hypervisor`. This document tracks implementation findings and decisions. ## Nested Virtualization fcvm supports running inside another fcvm VM using ARM64 FEAT_NV2. Recursive nesting (Host → L1 → L2 → ...) is enabled via the `arm64.nv2` kernel boot parameter. ### Requirements - **Hardware**: ARM64 with FEAT_NV2 (Graviton3+, c7g.metal) - **Host kernel**: 6.18+ with `kvm-arm.mode=nested` AND DSB patches - **Nested kernel**: Custom kernel with CONFIG_KVM=y (use `--kernel-profile nested`) ### Host Kernel with DSB Patches **CRITICAL**: Both host AND guest kernels need DSB patches for cache coherency under NV2. **Install host kernel**: `make install-host-kernel` (builds kernel, installs to /boot, updates GRUB). Patches from `kernel/patches/` are applied automatically during the build. **Current patches** (all apply to both host and guest kernels): - `nv2-vsock-cache-sync.patch`: DSB SY in `kvm_nested_sync_hwstate()` - `nv2-vsock-rx-barrier.patch`: DSB SY in `virtio_transport_rx_work()` - `nv2-targeted-shadow-s2-unmap.patch`: backport of upstream e6084aecaeef and its series, so an MMU notifier invalidation unmaps only the shadow stage-2 mappings of the invalidated range (#1274) - `mmfr4-override.vm.patch`: ID register override for recursive nesting (guest only) **VM Graceful Shutdown (PSCI)**: - fc-agent uses `poweroff -f` to trigger PSCI SYSTEM_OFF (function ID 0x84000008) - KVM forwards this to Firecracker via KVM_EXIT_SYSTEM_EVENT - NOTE: `halt -f` does NOT trigger PSCI - it just enters a WFI loop without calling PSCI ### How It Works 1. Set `FCVM_NV2=1` environment variable (auto-set when `--kernel-profile nested` is used) 2. fcvm passes `--enable-nv2` to Firecracker, which enables `HAS_EL2` vCPU feature 3. vCPU boots at EL2h in VHE mode (E2H=1) so guest kernel sees HYP mode available 4. EL2 registers are initialized: HCR_EL2, VMPIDR_EL2, VPIDR_EL2 5. Guest kernel initializes KVM: "VHE mode initialized successfully" 6. `arm64.nv2` boot param overrides MMFR4 to advertise NV2 support 7. L1 KVM reports `KVM_CAP_ARM_EL2=1`, enabling recursive L2+ VMs ### Running Nested VMs ```bash # Build nested kernel (first time only, ~10-20 min) fcvm setup --kernel-profile nested --build-kernels # Run outer VM with nested kernel profile sudo fcvm podman run \ --name outer \ --network bridged \ --kernel-profile nested \ --privileged \ --map /mnt/fcvm-btrfs:/mnt/fcvm-btrfs \ nginx:alpine # Inside outer VM, run inner fcvm fcvm podman run --name inner --network bridged alpine:latest ``` ### Key Firecracker Changes Firecracker fork with NV2 support (configured in kernel profile) - `HAS_EL2` (bit 7): Enables virtual EL2 for guest in VHE mode - Boot at EL2h: Guest kernel must see CurrentEL=EL2 on boot - VHE mode (E2H=1): Required for NV2 support in guest (nVHE mode doesn't support NV2) - VMPIDR_EL2/VPIDR_EL2: Proper processor IDs for nested guests ### Tests ```bash make test-root FILTER=kvm ``` - `test_kvm_available_in_vm`: Verifies /dev/kvm works in guest - `test_nested_run_fcvm_inside_vm`: Full nested virtualization test ### Recursive Nesting: The ID Register Problem (Solved) **Problem**: L1's KVM initially reported `KVM_CAP_ARM_EL2=0`, blocking L2+ VMs. **Root cause**: ARM architecture provides no mechanism to virtualize ID registers for virtual EL2. 1. Host KVM stores correct emulated ID values in `kvm->arch.id_regs[]` 2. `HCR_EL2.TID3` controls trapping of ID register reads - but only for **EL1 reads** 3. When guest runs at virtual EL2 (with NV2), ID register reads are EL2-level accesses 4. EL2-level accesses don't trap via TID3 - they read hardware directly 5. Guest sees `MMFR4=0` (hardware), not `MMFR4=NV2_ONLY` (emulated) **Solution**: Use kernel's ID register override mechanism with `arm64.nv2` boot parameter. 1. Added `arm64.nv2` alias for `id_aa64mmfr4.nv_frac=2` (NV2_ONLY) 2. Changed `FTR_LOWER_SAFE` to `FTR_HIGHER_SAFE` for MMFR4 to allow upward overrides 3. Kernel patch: `kernel/patches/mmfr4-override.patch` **Why it's safe**: The host KVM *does* provide NV2 emulation - we're just fixing the guest's view of this capability. We're not faking a feature, we're correcting a visibility issue. **Verification**: ``` $ dmesg | grep mmfr4 CPU features: SYS_ID_AA64MMFR4_EL1[23:20]: forced to 2 $ check_kvm_caps KVM_CAP_ARM_EL2 (cap 240) = 1 -> Nested virtualization IS supported by KVM (VHE mode) ``` ### Known NV2 Architectural Limitations ARM's FEAT_NV2 has fundamental architectural issues acknowledged by Linux kernel maintainers. These affect memory visibility, register access, and timer emulation under nested virtualization. **Kernel source citations** (from `torvalds/linux` master branch): From [`arch/arm64/kvm/nested.c`](https://github.com/torvalds/linux/blob/master/arch/arm64/kvm/nested.c): > "In yet another example where FEAT_NV2 is fscking broken, accesses to MDSCR_EL1 are redirected to the VNCR despite having an effect at EL2." > "One of the many architectural bugs in FEAT_NV2 is that the guest hypervisor can write to HCR_EL2 behind our back" From [`arch/arm64/kvm/arch_timer.c`](https://github.com/torvalds/linux/blob/master/arch/arm64/kvm/arch_timer.c): > "Paper over NV2 brokenness by publishing the interrupt status bit. This still results in a poor quality of emulation" > "NV2 badly breaks the timer semantics by redirecting accesses to the EL1 timer state to memory" **Impact on fcvm**: Under L2 (nested) VMs, vsock packet fragmentation can trigger memory visibility issues due to double Stage 2 translation (L2 GPA → L1 S2 → L1 HPA → L0 S2 → physical). Large writes that fragment into multiple vsock packets may see stale/zero data instead of actual content. **Fix**: The DSB SY kernel patch in `kernel/patches/nv2-vsock-cache-sync.patch` fixes this issue. The patch adds a full system data synchronization barrier in `kvm_nested_sync_hwstate()` to ensure L2's writes are visible to L1's reads before returning from the nested guest exit handler. With the patch applied, FUSE max_write can be unbounded (default). Without the patch, set `FCVM_FUSE_MAX_WRITE=32768` to limit writes to 32KB as a workaround. ### L2 Cache Coherency Fix (2026-01) **Problem**: L2 FUSE-over-FUSE corrupted with unbounded max_write (~1MB). After ~3-10MB transferred, L1 reads all zeros where L2's data should be. **Error pattern**: ``` STREAM CORRUPTION: zero-length message at count=67 after 10489619 bytes peek_bytes=128 hex=00 00 00 00 00 00 00 ... (128 bytes of zeros) ``` **Data path**: 1. L2 app writes to FUSE → L2 fc-agent multiplexer → L2 vsock → virtio ring 2. L2 kicks virtio (trap to L1 KVM) 3. L1 Firecracker reads from virtio ring (mmap of guest memory) 4. L1 VolumeServer writes to L1 FUSE → Host FS **Investigation**: - Raw vsock works fine (2MB packets, 4480/4480 tests pass) - Only FUSE-over-FUSE path triggers corruption (many small requests/responses) - Corruption happens when L1 reads virtio ring and sees stale/zero data **Root cause**: Under double Stage 2 translation, L2's writes to the virtio ring weren't visible to L1's mmap reads due to missing cache synchronization at nested guest exit. **Solution**: Add `dsb(sy)` in `kvm_nested_sync_hwstate()` - a full system data synchronization barrier that ensures all L2 writes complete and are visible before returning to L1. ```c // In arch/arm64/kvm/nested.c dsb(sy); // Full system barrier - ensures L2 writes visible to L1 ``` **Why it works**: The DSB SY barrier forces cache coherency across the entire system, including the mmap'd guest memory that Firecracker reads. ISH (inner-shareable) barriers weren't sufficient because the double S2 translation creates a cross-domain cache coherency issue. **Test results**: With the DSB SY patch, 100MB file copies through FUSE-over-FUSE complete successfully with unbounded max_write (~1MB packets). Test: `make test-root FILTER=nested_l2_with_large` ### NV2 Snapshot Lifecycle: Timer Coherence + One Restore Path (#630, 2026-06) The "nested tests are too slow/flaky" disables (Jan 2026) traced to TWO snapshot-lifecycle bugs, both fixed. Neither was an inherent NV2 limitation: **1. Timer-domain skew on restore (firecracker fork fix).** Restore replayed CNTVCT/CNTPCT (KVM adjusts per-timer offsets, which for HAS_EL2 guests live in CNTVOFF_EL2 storage), then replayed CNTVOFF_EL2 — clobbering the adjustment. The guest's monotonic clock jumped forward by the host time elapsed since the snapshot, every armed timer CVAL landed in the past, and KVM's emulated EL1 timers (NV2 traps these) re-fired in a storm (~14k kvm_timer_emulate/s measured) that starved the vCPUs 25-300x. Fix: firecracker owns the VM-wide counter offset via `KVM_ARM_SET_COUNTER_OFFSET` — set at boot (zero-based counters), set coherently before the register replay on restore, and advanced by the pause duration on resume so the guest clock FREEZES while a VM is paused (fcvm's pre-start snapshot pauses VMs for many seconds). **2. Event-loop starvation after pause→save→resume (fcvm design fix).** Resuming the VM that just produced a snapshot intermittently left Firecracker's single device event-loop thread spinning (94 CPU-seconds measured), starving every virtio queue: `NETDEV WATCHDOG: transmit queue 0 timed out` in the guest, stalled FUSE-over-vsock, apparent 100x guest slowdowns that self-heal minutes later. Restored VMs never storm (fresh process, fresh event loop): create+resume stormed 3/3 under load; restore was clean 12/12. Fix: the snapshot-miss path CONVERGES on the restore path — create the pre-start snapshot, tear the throwaway VM down, and relaunch by restoring it. Snapshot hit and miss now run the exact same flow. **Copy-on-write on the File backend (2026-09).** In CI's arm64 SnapshotEnabled lane `test_nested_run_fcvm_inside_vm` took 470 to 708 s, against 22 s in the SnapshotDisabled lane on the same heads. The only difference is that the SnapshotEnabled L1 is restored from its pre-start snapshot, and that restore used Firecracker's File backend. File maps memory.bin MAP_PRIVATE, so L1's first write to each page is a host copy-on-write. On arm64 the MMU-notifier invalidate for that copy reaches `kvm_unmap_gfn_range`, which calls `kvm_nested_s2_unmap` and unmaps every nested stage-2 MMU over its whole range, so each first write by L1 discards all of L2's translations. NV2 restores therefore page in through an implicit UFFD copy server (`direct_restore_memory` in `src/commands/snapshot.rs`), which fills anonymous memory that later writes do not copy. The chain is from reading arch/arm64/kvm/mmu.c and nested.c; it has not been traced on hardware. **Copy-on-write after a Full snapshot of the source (#1264).** The same invalidation reaches a cold-booted NV2 source through `snapshot create`. A Full snapshot that reads every guest page maps each page the guest never wrote to the shared zero page (a 1 GiB L1: 0 such pages before the dump, 144,696 after), so after the resume each first write to one is a copy-on-write that unmaps every shadow stage-2: ~600k exits/s on one IPA, guest soft lockups of 74 s, and `--resume-snapshot-network` unacknowledged for 40 s or more, in 12 of 337 pre-start snapshots on a pbox. The NV2 fork's Full dump leaves those pages as holes without reading them (pagemap and mincore say they have no page table entry), so the source keeps them unmapped and its first write is an ordinary anonymous fault. It still reads every page of a restore through a userfaultfd, of file-backed memory, and of a memory file it overwrites in place. `test_snapshot_create_leaves_nv2_source_pages_unmapped` (tests/test_kvm.rs) counts the zero-page pages across one snapshot. **Targeted shadow stage-2 unmap (#1274).** `kernel/patches-arm64/nv2-targeted-shadow-s2-unmap.patch` backports upstream e6084aecaeef and its series to the 6.18 host and nested kernels: KVM records each shadow mapping's canonical and nested IPA ranges, and an MMU-notifier invalidation unmaps only the shadow mappings that overlap the invalidated canonical range. With the host compacting memory every second, `test_nested_l2_with_large_files` passed in 32 s on the patched host (206,243 invalidations, none through `kvm_nested_s2_unmap`); on the unpatched host its first try failed after 849 s. The hand port's differences from upstream are in the patch description; `nv2_invalidation_unmaps_only_the_shadow_mappings_of_the_invalidated_range` (tests/test_kernel_patch_invariants.rs) runs the patched 6.18 functions in a C harness. CI runners get the patched host kernel only from a runner AMI built after the patch reaches main. **Debugging these**: guest `dd if=/dev/zero of=/dev/null` per-CPU (healthy ≈ 19 GB/s on Graviton3; storms read 0.07-0.3 GB/s), host `ftrace` on `kvm:kvm_timer_emulate`, per-thread CPU of the firecracker process (`/proc//task/*/stat` field 14+15), and the fork's env-gated `FC_DEBUG_REG_DUMP` / `FC_DEBUG_REG_DIFF` register instrumentation. **Historical note — "L2 single vCPU requirement"**: multi-vCPU L2 VMs hitting NETDEV WATCHDOG (TX queue not serviced) was previously attributed to NV2 cross-vCPU interrupt issues. That symptom matches bug 2 (host event loop starvation, which always reports the watchdog on the queue's CPU). Nested launches still use `--cpu 1` as a conservative default; re-evaluating multi-vCPU L2 is tracked in #632. ## FUSE Performance Tracing Enable per-operation tracing to diagnose FUSE latency issues (especially in nested VMs). ### Enabling Tracing Set `FCVM_FUSE_TRACE_RATE=N` to trace every Nth FUSE operation: ```bash # Trace every 100th request (recommended for benchmarks) FCVM_FUSE_TRACE_RATE=100 fcvm podman run --name test nginx:alpine # Trace every request (high overhead, use for debugging specific issues) FCVM_FUSE_TRACE_RATE=1 fcvm podman run ... ``` The env var is automatically passed to the guest via kernel boot parameters (`fuse_trace_rate=N`). ### Trace Output Format ``` [TRACE lookup] total=8940µs srv=159µs | fs=149 | to_srv=33 to_cli=1974 [TRACE fsync] total=70000µs srv=3000µs | fs=2900 | to_srv=? to_cli=? ``` | Field | Meaning | |-------|---------| | `total` | End-to-end client round-trip time | | `srv` | Server-side processing (reliable) | | `fs` | Filesystem operation time (subset of srv) | | `to_srv` | Network: client → server (may show `?` if clocks differ) | | `to_cli` | Network: server → client (may show `?` if clocks differ) | ### L2 Performance Expectations Based on FUSE-over-FUSE architecture: | Operation | Expected L2/L1 Ratio | Notes | |-----------|---------------------|-------| | `stat`/metadata | ~2x | One extra FUSE layer | | Async writes | ~3x | Data transfer overhead | | Sync writes (fsync) | ~8-10x | fsync propagates synchronously through layers | The fsync amplification occurs because each L2 fsync must wait for L1's fsync to complete, which itself waits for the host disk sync. This is fundamental to FUSE-over-FUSE durability. ### Related Configuration ```bash # Reduce FUSE readers for nested VMs (saves memory) FCVM_FUSE_READERS=8 fcvm podman run ... # Default: 64 readers × 8MB stack = 512MB ``` ## Quick Reference ### Shell Scripts to /tmp **Write complex shell logic to /tmp instead of fighting escaping issues:** ```bash # BAD - escaping nightmare for dir in ...; do count=$(grep ... | wc -l); done # GOOD - write to file, execute cat > /tmp/script.sh << 'EOF' for dir in */; do count=$(grep -c pattern "$dir"/*.rs) echo "$dir: $count" done EOF chmod +x /tmp/script.sh && /tmp/script.sh ``` ### Streaming Test Output **Use `STREAM=1` to see test output in real-time:** ```bash make test-root FILTER=sanity STREAM=1 # Host tests with streaming make container-test-root FILTER=sanity STREAM=1 # Container tests with streaming ``` Without `STREAM=1`, nextest captures output and only shows it after tests complete (better for parallel runs). **Log levels:** Tests run with `fcvm=debug` by default (FUSE spam suppressed). Override with: ```bash RUST_LOG=debug make test-root # Full debug (slow, 18x more output) ``` ### Debug Logs **All tests automatically capture debug-level logs to files.** How it works: - `spawn_fcvm()` and `spawn_fcvm_with_logs()` always create a log file - fcvm runs with `RUST_LOG=debug` for full debug output - Console shows INFO/WARN/ERROR only (DEBUG filtered out) - Log file has everything including DEBUG/TRACE - Path printed at end: `📋 Debug log: /tmp/fcvm-test-logs/{name}-{timestamp}.log` - CI uploads `/tmp/fcvm-test-logs/` as artifacts (7 day retention) - Tests add `--setup` flag automatically, so missing initrd auto-creates - Firecracker's own log (`firecracker.log` in the VM's data directory, copied to `/tmp/fcvm-firecracker-.log` at teardown) is written at `Info`. `FCVM_FIRECRACKER_LOG_LEVEL=Debug` raises it, and then the vsock muxer logs one line per packet: 311 MiB for a 75 s FUSE workload in a small VM, and 9.99 GiB in the first 84 minutes of a 128 GiB guest whose volumes are FUSE over vsock ### Common Commands ```bash # Build make build # Build fcvm + fc-agent make build-fc-mock # Build fc-mock (Firecracker mock for container mode) make test # Run fuse-pipe tests make setup-fcvm # Download kernel and create rootfs # Run a VM (requires setup first, or use --setup flag) sudo fcvm podman run --name my-vm --network bridged nginx:alpine # With custom command (docker-style trailing args) sudo fcvm podman run --name my-vm --network bridged alpine:latest echo "hello" # Or using --cmd flag sudo fcvm podman run --name my-vm --network bridged --cmd "echo hello" alpine:latest # Or run with auto-setup (first run takes 5-10 minutes) sudo fcvm podman run --name my-vm --network bridged --setup nginx:alpine # With extra root disk space (default: 10G free) sudo fcvm podman run --name my-vm --rootfs-size 50G nginx:alpine # Snapshot workflow fcvm snapshot create --pid --tag my-snapshot fcvm snapshot serve my-snapshot # Start UFFD server (prints serve PID) fcvm snapshot run --pid --name clone1 ``` ### Local Test Containers **Build test logic into a container, run with fcvm.** No weird feature flags or binary copying. ```bash # Build with localhost/ prefix podman build -t localhost/mytest -f Containerfile.mytest . # Run with fcvm (exports via skopeo automatically) sudo fcvm podman run --name test --network bridged \ --map /mnt/fcvm-btrfs/test-data:/data \ localhost/mytest ``` See `Containerfile.libfuse-remap` and `Containerfile.pjdfstest` for examples. ### Manual E2E Testing with Claude Code **CRITICAL: VM commands BLOCK the terminal.** You MUST use Claude's `run_in_background: true` feature. **PREFER NON-ROOT TESTING**: Run tests without sudo when possible. Rootless networking mode (`--network rootless`, the default) doesn't require sudo. Only use `sudo` for: - `--network bridged` tests - Operations that explicitly need root (iptables, privileged containers) The ubuntu user has KVM access (`kvm` group), so `fcvm podman run` works without sudo in rootless mode. ```bash # PREFERRED - Rootless mode (no sudo needed, use run_in_background: true) ./target/release/fcvm podman run --name test alpine:latest 2>&1 | tee /tmp/vm.log # Defaults to --network rootless # Get PID from state and use exec: ls -t /mnt/fcvm-btrfs/state/*.json | head -1 | xargs cat | jq -r '.pid' ./target/release/fcvm exec --pid -- hostname # ONLY WHEN NEEDED - Bridged mode (requires sudo) sudo ./target/release/fcvm podman run --name test --network bridged nginx:alpine 2>&1 | tee /tmp/vm.log # Then sleep and check logs: sleep 30 grep healthy /tmp/vm.log # Get PID from state and use exec: sudo ls -t /mnt/fcvm-btrfs/state/*.json | head -1 | xargs sudo cat | jq -r '.pid' sudo ./target/release/fcvm exec --pid -- curl -s ifconfig.me ``` **Testing egress connectivity:** ```bash # VM-level egress (runs in guest OS) fcvm exec --pid -- curl -s --max-time 10 ifconfig.me # Container-level egress (runs inside the container) fcvm exec --pid -c -- wget -q -O - --timeout=10 http://ifconfig.me ``` ### Debugging Network Issues with Quick VMs When debugging network issues (connectivity, DNS, routing), spawn quick one-off VMs with inline commands: ```bash # Test basic connectivity ./target/release/fcvm podman run --name net-test-$(date +%s) --privileged alpine:latest sh -c " echo '=== Network config ===' ip addr show eth0 ip route cat /etc/resolv.conf echo '' echo '=== Test gateway ping ===' ping -c 2 -W 3 10.0.2.2 || echo 'gateway ping failed' echo '' echo '=== Test DNS ===' nslookup example.com || echo 'DNS failed' echo '' echo '=== Test external ===' wget -q -O - --timeout=10 http://ifconfig.me || echo 'external failed' " 2>&1 & sleep 60 # Wait for VM to boot and run commands ``` **Debugging technique:** 1. Run background (`&`) so terminal stays available 2. Sleep to let VM boot (~30-60s) 3. Check stdout for results 4. Use `--privileged` for ping (requires raw sockets) 5. Test incrementally: gateway → DNS → external **Inspecting namespace state for running VM:** ```bash # Get holder PID from state file HOLDER_PID=$(cat /mnt/fcvm-btrfs/state/*.json | jq -r '.holder_pid') # Check namespace network config sudo nsenter --net=/proc/$HOLDER_PID/ns/net ip addr sudo nsenter --net=/proc/$HOLDER_PID/ns/net bridge link # Show bridge ports sudo nsenter --net=/proc/$HOLDER_PID/ns/net ip route ``` ### Code Philosophy **NO LEGACY/BACKWARD COMPATIBILITY.** This applies to everything: code, Makefile, documentation. - When we change an API, we update all callers - No deprecated functions, no compatibility shims, no `_old` suffixes - No legacy Makefile targets or aliases - No "keep this for backwards compatibility" comments - Clean breaks only - delete the old thing entirely Exception: For **forked libraries** (like fuse-backend-rs), we maintain compatibility with upstream to enable merging upstream changes. ### File Operations **Always use `git mv` when renaming files.** This preserves git history. ```bash # CORRECT - preserves history git mv old_name.rs new_name.rs # WRONG - loses history mv old_name.rs new_name.rs ``` ### Development Workflow (PR-Based) **IMPORTANT: Use `/pr-workflow` skill for ALL PR operations.** This includes: - Creating, checking, or merging PRs (`gh pr create`, `gh pr checks`, `gh pr merge`) - Checking CI status - Fixing lint/clippy/format errors - Running `cargo fmt` or `cargo clippy` **NEVER APPLY SKIP CONDITIONS FROM AUTO-FIX PRs.** When CI creates auto-fix PRs: - If the fix adds `#[ignore]`, early returns, or weakened assertions → CLOSE IT - Find and apply the ACTUAL FIX that makes tests pass - Tests must PASS, not be SKIPPED **Main branch is protected. All changes MUST go through pull requests.** #### Creating a PR **TEST LOCALLY BEFORE PUSHING.** CI is for validation, not discovery. #### Quick Reference | Action | Command | |--------|---------| | Create branch | `git checkout -b branch-name` | | **Test locally first** | `make lint && make test-root FILTER=` | | Push & create PR | `git push -u origin branch-name && gh pr create --fill` | | Check CI | `gh pr checks ` | | Merge PR | `gh pr merge --merge --delete-branch` | | List my PRs | `gh pr list --author @me` | **Stacking PRs:** create a branch chain (not parallel branches): ```bash git checkout -b feature-a && git push -u origin feature-a gh pr create --base main git checkout -b feature-b && git push -u origin feature-b gh pr create --base feature-a gh pr list --json number,headRefName,baseRefName ``` Merge in order (#1 first, then #2). **Never use `--delete-branch` on the base PR**: it closes dependent PRs. Merge without delete, run `gh pr edit --base main`, then delete branch. For the full merge procedure and safety checks, see `.claude/skills/pr-workflow/SKILL.md`. **CRITICAL: Maintain Stack Coherence.** PR #2's branch must actually be based on PR #1's branch, not just have GitHub base set. ```bash git log --oneline origin/main..feature-b ``` If PR #2's branch is based on main instead of feature-a, tests will fail because PR #2 won't have PR #1's changes. Fix with: ```bash git checkout -B feature-b origin/feature-a git cherry-pick git push origin feature-b --force ``` **One PR per concern:** Unrelated changes get separate PRs. ### THREE BROAD REVIEW PASSES, NOT AN OPEN-ENDED REVIEW LOOP Three broad passes are a ceiling, not a checklist. Every PR has the same minimum bar: focused regression coverage for each accepted defect class, one run of the relevant suite, self-review of the complete delta, required CI, and exact-head review coverage. Set any additional review from the downstream impact and demonstrated failure mechanisms, not from where the code runs. Production runtime, security, isolation, persistent data, migrations, recovery, public compatibility, and any CI, release, or review gate that can fail open may need the architecture, candidate, and closure passes below. For benchmarks, non-gating test harnesses, internal developer tools, and documentation whose failure cannot approve or ship production code, the minimum bar is normally sufficient. Do not add broad review passes, soak tests, fuzzing, platform matrices, or historical research unless the user asks for them or the demonstrated failure mechanism needs them. Write the PR's contract, downstream impact, and minimum evidence in the PR description. A race in benchmark output can invalidate the benchmark and must be fixed, but it does not turn benchmark tooling into a production safety system. Once the stated evidence is green, stop. A review campaign has at most three broad passes: 1. **Architecture pass, before implementation.** Write the invariants, ownership boundaries, failure states, publication boundary, and test plan. For process or file lifecycle work, draw the complete create/use/cleanup sequence. Review this model before patching individual call sites. 2. **Candidate pass, on one frozen tree.** Finish focused tests, freeze one commit, and run all independent reviewers in parallel against that exact commit. Wait for every reviewer, then deduplicate their findings into one report and fix the batch together. Reviewers must return one complete report, not drip findings while the tree moves. 3. **Closure pass, on the integrated exact head.** Run the full relevant suite, freeze the resulting commit, and review that exact head in parallel. Wait for every reviewer and fix the consolidated batch. Every accepted defect claim needs the required test observed red without the fix. A process, architecture, or configuration contract can be pinned by a fixture or source-level assertion when no runtime test fits. Naming, documentation, and style findings that are not defect claims use `NOT-A-DEFECT` and verification of the corrected text. Never substitute prose evidence for a red test on an accepted defect. Review the repair deltas, then run the full relevant suite and CI on the repaired exact head and verify exact-head coverage. Targeted checks supplement that validation; they do not start another broad campaign. Never send reviewers a moving worktree, add new broad reviewers halfway through a pass, or run the full suite after each individual finding. Batch findings, make one repair, and run the suite once per candidate. **The second instance of one defect class is an architecture finding.** Stop fixing sites one at a time. Extract a shared primitive or invariant, build a test matrix over every caller, and then resume the current pass. If the closure pass finds a new systemic defect class, stop and report that the architecture model was incomplete. Re-plan the abstraction once instead of silently starting broad review passes four, five, and six. This does not waive a concrete defect: fix it and prove it red, using a fixture or source-level assertion for a non-runtime contract. Keep the follow-up review targeted to that repair and the final exact head. There is one closure repair batch. If another blocking finding appears after that batch, reduce or split the PR to its last proven contract before adding more machinery. If the contract cannot be preserved by reduction, report the blocker and ask the user whether to continue. Do not silently begin another repair and review cycle. When a user-controlled remote is authorized, push each signed, focused-green frozen checkpoint before beginning the next pass. A remote checkpoint is recovery and visibility; it is not merge authorization and its old checks do not cover later local commits. **Land the smallest correct PR, then continue on the next stacked PR.** The architecture pass must partition the work into a sequence of independently useful, mergeable contracts. Once one PR satisfies its stated contract, focused and full tests, CI, and exact-head review, report it ready. Merge it only when the user has separately authorized that merge. Push or checkpoint authority is not merge authority. Do not hold a ready PR open to add adjacent features, speculative hardening, or cleanup that can be expressed safely in a follow-up. Every review finding must be classified before work starts: - **Blocks this PR:** the diff causes it, makes it worse, violates the PR's stated contract, makes its tests/evidence invalid, or would make the merged state unsafe. - **Stacked follow-up:** it is pre-existing or adjacent work that the PR does not rely on and can remain safely after this PR merges. File it or put it in the next stacked branch, with its dependency stated. Review scope is the PR's contract and delta, not an invitation to redesign the whole system. Never use “follow-up” to waive a known defect in changed code, but never keep a correct PR hostage to unrelated improvements either. A review thread classified as a follow-up still needs the gate's existing disposition vocabulary. Reply with `DISAGREE: this PR does not cause or worsen the reported behavior; tracked in #N` and resolve it only when the diff and tests substantiate that statement. If they do not, the finding blocks this PR. ### REMOVE GOLD PLATING BEFORE CLOSURE Every changed file, abstraction, test, and evidence artifact must do at least one of these: - implement the PR's stated contract; - close an accepted blocking finding caused or worsened by the PR; - provide evidence that is not already supplied by a cheaper test or summary. Remove or split everything else. Prefer deleting optional behavior or reverting to the last proven implementation over expanding a low-risk PR to defend it. Use one deterministic test per defect class, not several tests of the same invariant. Do not generalize a one-caller fix without a second required caller. Do not commit raw benchmark output when a small summary and the command that produced it carry the required evidence. ### Claude Review Workflow PRs trigger an automated Claude review via GitHub Actions. After pushing: ```bash # Wait for review check to complete gh pr checks # Look for: review pass 4m13s ... # Read review comments gh pr view --json comments --jq '.comments[] | .body' ``` If review finds critical issues, it may auto-create a fix PR. Cherry-pick the fix: ```bash git fetch origin git cherry-pick git push gh pr close # Close the auto-generated PR ``` **MANDATORY before merging any PR:** Read all review comments first — and note that `--json comments` returns **only issue comments**. Inline review comments, which is where CodeRabbit and Codex put their actual findings, live on a different endpoint and are invisible to that query: ```bash gh pr view --json comments --jq '.comments[] | .body' # top-level only gh api --paginate repos/{owner}/{repo}/pulls//comments \ --jq '.[] | "=== \(.user.login) \(.path):\(.line // .original_line)\n\(.body)\n"' ``` Two things that look like details and are not: - **`--paginate` is mandatory.** Without it you get the first page only, so a PR that has accumulated findings over several review rounds silently reports a subset — and the audit that was supposed to catch hidden findings becomes one. - **Print `.body` whole.** A `[0:200]` preview drops the scenario, the evidence, and the suggested fix — the parts you need in order to decide. A truncated finding is not a finding you have read. On 2026-08-08 a PR carried **four unread inline findings, two of them Major**, while the check rendered `CodeRabbit pass`. One was a slice index that would panic inside the fault-handler task and hang the guest. Reading only the top-level comments would have merged all four. **A green `CodeRabbit` check does NOT mean CodeRabbit reviewed anything.** When it hits its rate limit it posts *"Review limit reached ... we couldn't start this review"* and the check still renders as `CodeRabbit pass`. "The reviewer never ran" is indistinguishable from "the reviewer approved" — the same class of bug as a contention detector that can never fire, or a leak check whose pattern can never match. Prove the review happened before counting it: ```bash gh pr view --json comments \ --jq '.comments[] | select(.author.login=="coderabbitai") | .body' | head -5 # "Review limit reached" / "next review in NN minutes" => NOT reviewed. Re-run before merging. ``` **GitHub re-anchors still-open review comments onto the current head.** After you push a fix, an *old* comment's `commit_id` and `line` both change to match the new HEAD — so a finding you already fixed reappears looking brand new, at a shifted line number. Observed live: four comments created at `09:01:36Z` against `23558456` re-anchored onto the fix commit, with `prefetch.rs:179 → 196` and `server.rs:1975 → 1991`, while the comment count never changed. **`isResolved` is the ONLY field that means resolved.** Age does not: ```bash gh api graphql -f query='{repository(owner:"O",name:"R"){pullRequest(number:N){ reviewThreads(first:100){nodes{isResolved comments(first:100){nodes{author{login} path line body}}}}}}}' \ --jq '.data.repository.pullRequest.reviewThreads.nodes[] | select(.isResolved==false)' ``` An earlier version of this file said "`created_at` BEFORE your fix commit ⇒ already addressed", and that rule is **unsound** — it was wrong here for months of nothing and then wrong in practice within a day. When one commit fixes SOME of several findings, every older comment still predates it, **including the ones nobody fixed**, so the rule silently reclassifies unfixed blockers as handled. `original_commit_id` + `created_at` can tell you a comment is OLD or RE-ANCHORED; neither can tell you its concern was addressed. Only a human reading the thread, or `isResolved`, can. Use age for one thing only: deciding whether a finding needs re-reading after a push, never whether it needs fixing. **A CLOSED PR reports CI results for a commit you are no longer on.** GitHub does not advance a closed PR's head, and `pull_request` events do not fire for one — so `git push` updates `refs/heads/` and creates **zero check-runs**, while `gh pr checks` keeps serving the *previous* commit's results. Same shape as the above: "CI never ran" is indistinguishable from "CI passed", and it survives a fetch, a re-push, and a `--force`. Anyone can close a PR out from under you (a dedupe, a bot, a stale-branch sweep), so check PR state before trusting its checks, and bind results to a SHA rather than to the PR: ```bash gh pr view --json state,headRefOid --jq '"\(.state) head=\(.headRefOid)"' git rev-parse HEAD # must equal headRefOid; if it does not, the checks are for other code gh api --paginate repos/OWNER/REPO/commits/$(git rev-parse HEAD)/check-runs \ --jq '.check_runs[].id' | wc -l # --slurp is rejected alongside --jq; count the stream # 0 => nothing ran for this commit. `gh pr reopen ` resyncs the head and triggers CI. ``` **Reopening is not guaranteed to start CI.** It fires a `reopened` event, which a workflow receives only if its `pull_request` trigger omits `types` (or lists `reopened`) — but even then `paths`/`paths-ignore` still applies, so a PR whose every changed file matches an ignored pattern gets a `reopened` event and **still runs nothing**. Docs-only and bench-only PRs land in exactly that hole. Check, and fall back to an explicit dispatch: ```bash gh api --paginate repos/OWNER/REPO/commits/$(git rev-parse HEAD)/check-runs \ --jq '.check_runs[].id' | wc -l # --slurp is rejected alongside --jq; count the stream # still 0 after reopen => the workflow filtered this PR out, not a sync problem gh workflow run ci.yml --ref "$(git branch --show-current)" ``` Do not read "0 checks" as "CI passed"; it means nothing ran, which is the same green-by-absence trap as a reviewer that never started. ### GitHub Actions Workflow Security (claude.yml) Jobs run with secrets, so editing `.github/workflows/claude.yml` is security-critical. Rules (Anthropic `claude-code-action/docs/security.md` + GitHub "pwn requests"); re-run local codex until SAFE: - `pull_request` from forks has no secrets; `issue_comment`/`pull_request_target`/`workflow_run` do. - Never check out/build untrusted PR head in a secret job (postinstall/build.rs → exfil). - Gate on PR-author `author_association`, not the commenter (commit email is spoofable). - `workflow_run`: require `head_repository.full_name == github.repository`, not branch name. - Never interpolate `${{ github.event.* }}` into `run:` (injection) — pass via `env:`. - `issue_comment` fires for issues too; gate on `github.event.issue.pull_request`. - Centralize into one `eligible` allowlist every job gates on. ### PR Descriptions: Show, Don't Tell **CRITICAL: Review commits in THIS branch before writing PR description.** For stacked PRs (branches of branches), only describe commits in YOUR branch: ```bash # First: identify your base branch gh pr view --json baseRefName # Shows what branch this PR targets # Then: review only YOUR commits (not the whole stack) git log --oneline origin/..HEAD # Commits in THIS branch only git log --oneline origin/main..HEAD # Only if PR targets main directly ``` **Anti-pattern:** On stacked PRs, reviewing `main..HEAD` includes parent commits and causes incorrect claims. **Include test evidence.** Actual output, not "tested and works." Simple PR: ```markdown ## Fix cargo fmt scope Changed to only check workspace packages. Tested: cargo fmt -p fcvm -p fuse-pipe --check # passes ``` Complex PR (kernel patches, workarounds, architectural changes): ```markdown One-line description of what this enables. ## The Problem - What was broken - Root cause analysis ## The Solution What changed and why this approach over alternatives. ## Test Results $ actual-command-run actual output ``` ### Commit Messages **Include what changed, why, and test evidence.** ``` Remove obsolete require_non_root guard function The function was a no-op kept for "API compatibility" - exactly what our NO LEGACY policy prohibits. Removed function and all 12 call sites. Tested: make test-root FILTER=sanity (both rootless and bridged pass) ``` ### JSON Parsing **NEVER parse JSON with string matching.** Always use proper deserialization. ```rust // BAD - Fragile, breaks with formatting changes if stdout.contains("\"health_status\":\"healthy\"") { ... } // GOOD - Use serde #[derive(Deserialize)] struct VmState { health_status: String } let vms: Vec = serde_json::from_str(&stdout)?; if vms.first().map(|v| v.health_status == "healthy").unwrap_or(false) { ... } ``` ### Test Failure Philosophy Test failures are bugs. Do not dismiss them as "resource contention", "timing issues", "flaky tests", or "works on my machine". For failures under load/parallel: use logs and timestamps, find the race/concurrency bug, fix the code, and add regression coverage. ### POSIX Compliance Testing **fuse-pipe must pass pjdfstest** - the POSIX filesystem test suite. When a POSIX test fails: 1. **Understand the POSIX requirement** - What behavior does the spec require? 2. **Check kernel vs userspace** - FUSE operations go through the kernel, which handles inode lifecycle. Unit tests calling PassthroughFs directly bypass this. 3. **Use integration tests for complex behavior** - Hardlinks, permissions, and refcounting require the full FUSE stack (kernel manages inodes). 4. **Unit tests for simple operations** - Single file create/read/write can be tested directly. **Key FUSE concepts:** - Kernel maintains `nlookup` (lookup count) for inodes - `release()` closes file handles, does NOT decrement nlookup - `forget()` decrements nlookup; inode removed when count reaches zero - Hardlinks work because kernel resolves paths to inodes before calling LINK **If a unit test works locally but fails in CI:** Add diagnostics to understand the exact failure. Don't assume - investigate filesystem type, inode tracking, and timing. ### Race Condition Debugging Protocol **Show, don't tell. We have extensive logs - it's NEVER a guess.** 1. **NEVER "fix" with timing changes** (timeouts, sleeps, reducing parallelism) 2. **ALWAYS find the smoking gun in logs** - compare failing vs passing timestamps 3. **Real example**: Firecracker crashed in parallel tests. Logs showed: failing test took 122s to export image (lock contention), then VM crashed 24ms after spawn. Passing test took 103s. **Root cause:** thundering herd after podman lock. **Fix:** content-addressable image cache. 4. **The mantra:** What do timestamps show? What's different between failing and passing? The logs ALWAYS have the answer. 5. **Real example (cross-VM cache race, #677):** De-serialized nested tests (max-threads>1) failed. The lazy conclusion was "nested KVM can't take concurrency (#660)." It was not — the smoking gun was in the per-VM logs (`/tmp/nested-l2-*.log`), and the technique that caught it is reusable: - **Read the per-VM debug logs, not the nextest summary.** Two concurrent chains showed, at the same second: chain A `resize2fs: Inode checksum does not match inode` and chain B `renaming storage image to final path: No such file or directory (os error 2)` — both on the SAME content-addressed path `image-cache/.storage-v2.img.tmp`. Same file, two writers = race, full stop. - **`e2fsck -fn` the *cached artifact* to prove the race POISONED the cache.** The already-cached `.storage-v2.img` failed checksum ("Filesystem still has errors"). A race that persists a corrupt content-addressed file silently breaks every *later* run too (including serialized ones) — so the symptom and the cause can be in different test runs. - **Know when a single host CAN'T reproduce it.** The per-digest `flock` works fine within one kernel, so host-only concurrent processes serialize and never collide. The race only appears across VM boundaries: nested tests `--map` the host `/mnt/fcvm-btrfs` into every L1, and `flock()` over a fuse-pipe mount that negotiates no `FUSE_FLOCK_LOCKS` is granted LOCALLY per guest kernel — it never reaches the shared host backing store. Faithful repro = clear the digest's cache entry (cold cache) + run two nested tests concurrently. - **Fix at the right altitude:** don't try to make `flock` work over FUSE — make the writes not need a lock. Unique-per-builder temp names (`uuid`, NOT pid — separate PID namespaces reuse numbers) + atomic rename to the content-addressed final (identical bytes → the rename race is idempotent). See `src/commands/podman/{image.rs,mod.rs}`. **Generalizable rule:** any file under a `--map`'d / FUSE-shared directory written by more than one VM must be made safe WITHOUT relying on `flock`/`fcntl` locks — those don't cross the fuse-pipe boundary. Use content-addressing + unique temp + atomic rename. ### NO TEST HEDGES **Test assertions must be DEFINITIVE.** A test either PASSES or FAILS - no middle ground. **NEVER write hedges like:** - "NOTE: this may not work (known limitation)" - "We log the result but don't fail the test for now" - "skip this assertion for now" - "this is expected to fail sometimes" **If a feature should work:** - Write an assertion that FAILS if it doesn't work - Fix the bug so the assertion passes - If you can't fix it, file an issue and mark the test `#[ignore]` with a link **Example of UNACCEPTABLE test code:** ```rust // BAD - This hides bugs! if !localhost_works { println!("NOTE: localhost port forwarding not working (known limitation)"); } // BAD - Test "passes" even when feature is broken ``` **Example of CORRECT test code:** ```rust // GOOD - This catches bugs! assert!(localhost_works, "Localhost port forwarding should work (requires route_localnet)"); // GOOD - Test fails if feature is broken ``` ### Parallel Test Isolation **Tests MUST work when run in parallel.** Resource conflicts are bugs, not excuses. **Test feature flags:** - `#[cfg(feature = "privileged-tests")]`: Tests requiring sudo (iptables, root podman storage) - No feature flag: Unprivileged tests run by default - Features are compile-time gates - tests won't exist unless the feature is enabled - Use `FILTER=` to further filter by name pattern: `make test-root FILTER=exec` - For multiple tests or regex: `make test-root FILTER="-E 'test(/pattern1|pattern2/)'" STREAM=1` - FILTER is a nextest substring match on test function name, NOT file name. Use `-E` for expressions. **Common parallel test pitfalls and fixes:** 1. **Unique resource names**: Use `common::unique_names()` helper to generate timestamp+counter-based names ```rust let (baseline, clone, snapshot, serve) = common::unique_names("mytest"); // Returns: mytest-base-12345-0, mytest-clone-12345-0, etc. ``` 2. **Port forwarding**: Both networking modes use unique IPs, so same port works ```rust // BRIDGED: DNAT scoped to veth IP (172.30.x.y) - same port works across VMs "--publish", "8080:80" // Test curls veth's host_ip:8080 // ROOTLESS: each VM gets unique loopback IP (127.x.y.z) - same port works "--publish", "8080:80" // Test curls loopback_ip:8080 ``` - Tests must curl the VM's assigned IP (veth host_ip or loopback_ip), not localhost - Get the IP from VM state: `config.network.host_ip` (bridged) or `config.network.loopback_ip` (rootless) 3. **Disk cleanup**: VM data directories are cleaned up on exit - `podman.rs` and `snapshot.rs` both delete `data_dir` on VM exit - Prevents disk from filling up with leftover VM directories 4. **State file cleanup**: State files are deleted when VMs exit - Prevents stale state from affecting IP allocation 5. **Unique ports/directories**: Tests must not share ports or temp directories - Use `std::process::id() % 1000` offset for ports - Use test name suffix for directories (e.g., `/tmp/scripts-{test_name}/`) - Test owns lifetime of any services it starts (kill at end) **If tests fail in parallel but pass alone:** - It's a resource isolation bug - FIX IT - Check for shared state (files, ports, IPs, network namespaces) - Add unique naming or proper cleanup ### Build and Test Rules Do not run `sudo cargo`. Use Makefile targets so cargo runs as your user and test binaries run via `CARGO_TARGET_*_RUNNER='sudo -E'`; otherwise `target/` becomes root-owned and breaks builds. ### PROCESS TEARDOWN IS PER-HOP. EVERY HOP. **`PR_SET_PDEATHSIG` only kills a child whose OWN pdeathsig is set. One unprotected hop orphans the entire subtree below it.** The chain that must hold, end to end: ``` cargo-nextest (uid 1000) → sudo (ruid 1000) → test binary (uid 0) → fcvm ─┬→ firecracker ^ setpriv --pdeathsig ^ set_test_pdeathsig ├→ holder └→ pasta (rootless) ^ see per-hop list below ``` - `scripts/root-test-runner.sh` (`ROOT_TEST_RUNNER` in the Makefile) covers the **sudo hop**. Never reduce the privileged runner back to a bare `sudo -E env PATH=...`. - `tests/common/mod.rs::set_test_pdeathsig{,_std}` covers **every long-lived fcvm spawn** in tests. `spawn_fcvm*` applies it; a hand-rolled `Command::new(&fcvm_path)…spawn()` must too. - `src/utils.rs::install_namespace_pre_exec` covers **every VMM spawn** (Firecracker, Cloud Hypervisor, and the Layer-2 setup VM in `src/setup/rootfs.rs`). It must stay the LAST `pre_exec` — `setns(CLONE_NEWUSER)` zeroes `pdeath_signal`, so setting it earlier is lost. - `src/commands/common.rs::spawn_namespace_holder` covers the **holder hop** (rootless mode). The holder's pdeathsig is armed in its `pre_exec`, before UID/GID mappings are written. - `src/network/pasta.rs` (in `start_pasta`) covers the **pasta hop** (rootless mode). Must be the last `pre_exec`; includes a `getppid` re-check to close the fork/exec race window. Regression tests: `test_sigkill_reaps_rootless_vm_tree` asserts all three children (firecracker, holder, pasta) die when fcvm is SIGKILL'd. `test_root_test_runner_reaps_vm_when_sudo_is_killed` covers the sudo hop specifically. **Why a privilege boundary is special:** the kernel refuses a signal from uid 1000 to a uid-0 process, and `killpg` still returns success when it managed to signal *any* member — so nextest's slow-timeout kill fails **silently** against a `sudo`'d test binary. SIGKILL cannot be caught, so `sudo` never forwards it either. On 2026-08-06 that left ~498 `firecracker` processes alive on two ARM runners (load average 523, 103 tasks in D-state, 420 zombies) for 21 hours, and the AWS lease logic kept renewing the dead runners because GitHub still reported them busy. Regression test: `test_root_test_runner_reaps_vm_when_sudo_is_killed` (`tests/test_signal_cleanup.rs`). Defense in depth (NOT a substitute): `timeout-minutes` on every self-hosted job plus the `scripts/ci-stray-vm-guard.sh` pre/post steps in `ci.yml`. **Prefer kernel-enforced reaping over cleanup code.** A `Drop` impl, a signal handler, or an `always()` cleanup step does not run when the process is SIGKILLed — which is exactly the case that leaked. If teardown matters, it must survive SIGKILL. **Host state the kernel does not reap is recorded before it is made.** A VM's runtime directory, the mountpoint a restore makes over a gone source's directory, and image-cache temporaries are recorded with an owner (`src/reclaim.rs`) whose flock the kernel releases on SIGKILL, and every launch first reclaims what dead owners recorded. A new resource that outlives fcvm unless teardown removes it gets a `reclaim::Resource` and a SIGKILL case in `tests/test_reclaim.rs`. ### Container Build Rules **Container builds work naturally with layer caching.** No workarounds needed. - Podman caches layers based on Containerfile content - When you modify a line, that layer and all subsequent layers rebuild automatically - Just run `make container-build-root` and let caching work - NEVER use `--no-cache` or add dummy comments to invalidate cache **Symlinks for sudo access**: The Containerfile creates symlinks in `/usr/local/bin/` so that `sudo cargo` works (sudo uses secure_path which includes `/usr/local/bin`). This matches how the host is configured. Cargo.lock pins the Git revision of `fuse-backend-rs` for normal builds and CI. Host builds do not need a sibling checkout. Container recipes retain the `FUSE_BACKEND_RS` and `FUSER` source mounts, so those directories must exist; an explicit backend override replaces that mount's source directory. The local override is Make-only. To compile uncommitted changes in a local checkout, opt in explicitly: ```bash make build FUSE_BACKEND_RS_OVERRIDE=../fuse-backend-rs make container-test FUSE_BACKEND_RS_OVERRIDE=../fuse-backend-rs make clippy FUSE_BACKEND_RS_OVERRIDE=../fuse-backend-rs ``` The override uses a Cargo path patch, including local manifest dependency changes. Container targets mount that checkout and rewrite the path for the inner build. An override changes Cargo.lock to the local source; do not commit that local-only lockfile change. After removing the override, `make dependency-metadata` resolves the Git dependency again; review its lockfile diff before committing. `make lint` audits the default Git dependency graph and rejects a local override: the pinned cargo-deny version cannot receive the local Cargo patch configuration. Use `make clippy` to lint local dependency development without that audit. The standalone `scripts/run_fuse_pipe_tests.sh` uses the default Git dependency and rejects a local override. For local FUSE tests, use `make test-root FILTER='-p fuse-pipe' FUSE_BACKEND_RS_OVERRIDE=../fuse-backend-rs`. ### Container KVM Access (Rootless Podman) `--device /dev/kvm` fails silently in rootless podman (ignores group membership). Use `-v` bind mount with `--group-add keep-groups` instead. See Makefile `CONTAINER_RUN` and [podman#16701](https://github.com/containers/podman/issues/16701). ### Monitoring Long-Running Tests **Max 30 second sleeps** when waiting for results. Provide play-by-play updates as tests run. ### Preserving Logs from Failed Tests Always include branch name in tee log filenames to avoid overwrites across branches/worktrees. ```bash BRANCH=$(git branch --show-current) # Standard run make test-root 2>&1 | tee /tmp/test-${BRANCH}-root.log # Filtered run make test-root FILTER= 2>&1 | tee /tmp/test-${BRANCH}-root-.log ``` If a run fails, archive the log with test name + timestamp before re-running: ```bash cp /tmp/test-${BRANCH}-root.log /tmp/fcvm-failed-${BRANCH}-root-test_exec_rootless-$(date +%Y%m%d-%H%M%S).log # Continue with a fresh log file make test-root 2>&1 | tee /tmp/test-${BRANCH}-root-run2.log ``` Optional automation: ```bash if grep -q "FAIL\|TIMEOUT" /tmp/test-${BRANCH}-root.log; then cp /tmp/test-${BRANCH}-root.log /tmp/fcvm-failed-${BRANCH}-root-$(date +%Y%m%d-%H%M%S).log fi ``` ### Debugging fuse-pipe Tests **ALWAYS run tests with debug logging enabled when debugging issues:** ```bash # Full debug RUST_LOG=debug make test-root FILTER=permission STREAM=1 # Component-focused RUST_LOG="passthrough=debug,fuse_pipe=debug" make test-root FILTER=permission STREAM=1 RUST_LOG="fuse_backend_rs=debug" make test-root FILTER=permission STREAM=1 ``` **Tracing targets:** - `passthrough` - fuse-pipe passthrough operations - `fuse_pipe` - fuse-pipe client/server - `fuse_backend_rs` - fuse-backend-rs internals (uses `log` crate, bridged via tracing-log) ### Debugging Protocol Issues (ftruncate example) When a FUSE operation fails unexpectedly, trace the full path from kernel to fuse-backend-rs: 1. **Add debug logging to passthrough handler** to see what parameters arrive: ```rust debug!(target: "passthrough", "setattr inode={} handle={:?} valid={:?}", inode, handle, valid); ``` 2. **Run test with logging** to see the actual values: ```bash RUST_LOG='passthrough=debug' make test-root FILTER=permission STREAM=1 ``` 3. **Check if kernel sends parameter but protocol drops it** - e.g., `handle=None` when it should be `Some(1)` means the protocol layer isn't passing it through. 4. **Trace the path**: kernel → fuser → fuse-pipe client (`_fh` unused?) → protocol message → handler → passthrough → fuse-backend-rs This pattern found the ftruncate bug: kernel sends `FATTR_FH` with file handle, but fuse-pipe's `VolumeRequest::Setattr` didn't have an `fh` field. ### Kernel Tracing (Ftrace) Use `common::Ftrace` for KVM debugging: ```rust let tracer = common::Ftrace::new()?; tracer.enable_events(common::Ftrace::EVENTS_PSCI)?; tracer.start()?; // ... run VM ... tracer.stop()?; println!("{}", tracer.read_grep("kvm_exit", 50)?); ``` **Event sets:** `EVENTS_PSCI` (low noise), `EVENTS_INTERRUPTS`, `EVENTS_DETAILED` (noisy) ## CI and Testing **See README.md for test categories, CI summary, and Makefile targets.** Run `make help` for full list. Key points for development: - CI runs on every PR: Host (bare metal) + Container (privileged) - Manual trigger: `gh workflow run ci.yml --ref ` - Get in-progress logs: `gh api repos/OWNER/REPO/actions/runs/RUN_ID/jobs` ## PID-Based Process Management **Core Principle:** All fcvm processes store their own PID (via `std::process::id()`), not child process PIDs. ### Process Types 1. **VM processes** (`fcvm podman run`) - `process_type`: "vm", health check: HTTP to guest 2. **Serve processes** (`fcvm snapshot serve`) - `process_type`: "serve", health check: process existence 3. **Clone processes** (`fcvm snapshot run`) - `process_type`: "clone", references parent via `serve_pid` ### State Management ```rust pub struct VmConfig { pub snapshot_name: Option, // Which snapshot pub process_type: Option, // "vm" | "serve" | "clone" pub serve_pid: Option, // For clones: parent serve PID } pub struct VmState { pub pid: Option, // fcvm process PID (from std::process::id()) pub pid_start_time: Option, // Process start time (clock ticks since boot) } ``` **Concurrency model** — state files use per-VM flock for all mutations: - `save_state()`: Overwrites entire file. Used for **initial creation** only (single writer). Records `pid_start_time` automatically from `/proc//stat` field 22. - `update_state(vm_id, |state| { ... })`: **Locked read-modify-write.** Loads current on-disk state, applies the closure, writes back. Used by all post-startup writers (health monitor, snapshot name recording) so concurrent updates are not clobbered. Returns `Ok(None)` if the state file was deleted (no-op, cannot resurrect). - `delete_state()`: Acquires the per-VM flock, deletes the state file, then removes the lock file. A concurrent `update_state` either completes before deletion or finds no state file afterwards and becomes a no-op. - `update_health_status()`: Thin wrapper around `update_state` that sets `health_status` and `exit_code`. ### Cleanup Architecture On serve process exit (SIGTERM/SIGINT): 1. Query state manager for all VMs where `serve_pid == my_pid` 2. Kill each clone process: `kill -TERM ` 3. Remove socket file: `/mnt/fcvm-btrfs/uffd-{snapshot}-{pid}-{pid_start_time}.sock` 4. Delete serve state from state manager ### Stale State File Handling **Problem**: State files persist when VMs crash (SIGKILL, test abort). When the OS reuses a PID, the old state file causes collisions when querying by PID. **Solution — two layers of defense:** 1. **PID start-time identity** (`pid_start_time` field): `save_state()` records the process's start time from `/proc//stat` (field 22, clock ticks since boot). A (pid, start_time) pair uniquely identifies a process even after PID reuse. `load_state_by_pid()` and `cleanup_stale_state()` verify this — if the process at the recorded PID has a different start time, the state file is stale. 2. **PID collision cleanup in `save_state()`**: Before saving, checks if any OTHER state file claims the same PID. If found, that file is stale (process is dead, PID was reused). Deletes the stale file with a warning log, then saves. **State file layout**: Individual files per VM, keyed by `vm_id` (UUID): ``` /mnt/fcvm-btrfs/state/ ├── vm-abc123.json # { vm_id: "vm-abc123", pid: 5000, pid_start_time: 12345, ... } ├── vm-def456.json # { vm_id: "vm-def456", pid: 5001, pid_start_time: 67890, ... } └── loopback-ip.lock # Global lock for IP allocation ``` No master state file - `list_vms()` globs all `.json` files. ## Architecture ### Project Structure ``` src/ ├── lib.rs # Module exports (public API) ├── main.rs # CLI dispatcher ├── paths.rs # Path utilities for btrfs layout ├── health.rs # Health monitoring ├── kvm_trace.rs # KVM ftrace helper for debugging snapshot restore ├── utils.rs # Process/system utility functions ├── cli/ # Command-line parsing │ └── args.rs # Clap structures ├── commands/ # Command implementations ├── state/ # VM state management ├── hypervisor/ # Pluggable hypervisor (VMM) abstraction (Firecracker + Cloud Hypervisor) ├── firecracker/ # Low-level Firecracker API client (Firecracker backend impl) ├── network/ # Networking layer (bridged + pasta + routed + egress proxy) ├── storage/ # Disk/snapshot management ├── uffd/ # UFFD memory sharing ├── volume/ # FUSE volume handling └── setup/ # Setup subcommands tests/ ├── common/mod.rs # Shared test utilities (VmFixture, poll_health_by_pid) ├── test_sanity.rs # End-to-end VM sanity tests (rootless + bridged) ├── test_state_manager.rs # State manager unit tests ├── test_health_monitor.rs # Health monitoring tests ├── test_fuse_in_vm_matrix.rs # In-VM pjdfstest (17 categories, parallel via nextest) ├── test_localhost_image.rs # Local image tests ├── test_snapshot_clone.rs # Snapshot/clone workflow tests ├── test_egress.rs # Egress proxy tests (rootless + bridged, fresh + clone) ├── test_egress_stress.rs # Egress proxy IPv4/IPv6 stress tests └── test_egress_proxy_bench.rs # 8000-connection concurrent benchmark fuse-pipe/tests/ ├── integration.rs # Basic FUSE operations (no root) ├── integration_root.rs # FUSE operations requiring root ├── test_permission_edge_cases.rs # Permission/setattr edge cases ├── test_mount_stress.rs # Mount/unmount stress tests ├── test_allow_other.rs # AllowOther flag tests ├── test_unmount_race.rs # Unmount race condition tests ├── pjdfstest_matrix_root.rs # Host-side pjdfstest (17 categories, parallel) └── pjdfstest_common.rs # Shared pjdfstest utilities fuse-pipe/benches/ ├── throughput.rs # I/O throughput benchmarks ├── operations.rs # FUSE operation latency benchmarks └── protocol.rs # Wire protocol benchmarks ``` ### Design Principles - **Library + Binary pattern**: src/lib.rs exports all modules, src/main.rs is thin dispatcher - **One file per command**: Easy to find, easy to test - **Single binary**: `fcvm` with subcommands (guest agent `fc-agent` is separate) ## Implementation Status ### ✅ Completed 1. **Core Implementation** (2025-11-09) - Pluggable `Hypervisor` trait (`src/hypervisor/mod.rs`) selecting the VMM behind the same orchestration, with two backends: **Firecracker (default)** and **Cloud Hypervisor**, chosen via `--hypervisor firecracker|cloud-hypervisor` (#632) - Firecracker backend: API client using hyper + hyperlocal (Unix sockets); boot plan via MMDS - Cloud Hypervisor backend: cold boot + container run, snapshot/restore/clone via `--restore` + in-process UFFD; boot plan via vsock (no MMDS) - Dual networking modes: bridged (iptables) + rootless (pasta) - Storage layer with btrfs CoW disk management - VM state persistence - Guest agent (fc-agent) receives its boot plan via MMDS (Firecracker) or over vsock (Cloud Hypervisor) 2. **Snapshot/Clone Workflow** (2025-11-11, verified 2025-11-12) - Pause VM → Create Firecracker snapshot → Resume VM - UFFD memory server serves pages on-demand via Unix socket - Clone disk uses btrfs reflink (~3ms instant CoW copy) - Clone memory load time: ~2.3ms - UFFD clones populate lazily (faulted pages are per-VM copies); File-backend clones share clean pages via the page cache (measured in #632) - **Memory per clone**: no committed record measures an idle UFFD clone. `bench/chromium/results/20260808-corrected` measured 2 GiB Chromium guests, each sampled while it idled after one render, on a binary without working-set replay. As the slope of cgroup `memory.current` over 1 to 16 concurrent clones, it found 257.8 MiB per clone in copy mode, 132.5 MiB in minor mode and 143.5 MiB file-backed. Its 2 MiB hugepage cell does not count the guest's hugepages and is not a per-clone cost (`bench/chromium/REVIEW.md`). With prefetch on (the default), a copy-mode clone also holds a private copy of every page in the recorded set. 3. **True Rootless Networking** (2025-11-25) - `--network rootless` (default): pasta, no root required - `--network bridged`: Network namespace + iptables, requires root - `--network routed`: veth + IPv6 kernel routing, requires root + IPv6 host - User namespace via `unshare --user --net` with external UID/GID mappings - Health checks use unique loopback IPs (127.x.y.z) per VM 4. **Hierarchical Logging** (2025-11-15) - Target tags showing process nesting - Smart color handling: TTY gets colors, pipes don't - Strips Firecracker timestamps and `[anonymous-instance:*]` prefixes 5. **Container Lifecycle Management** (2025-12-08) - Container exit code forwarding via vsock status channel (port 4999) - `--privileged` mode for containers requiring device access and mknod - Health monitoring detects stopped containers (`HealthStatus::Stopped`) - `fcvm podman run` returns non-zero exit code when container fails - State tracking includes `exit_code` field in `VmState` 6. **Supplementary Groups Forwarding** (2025-12-08) - fuse-pipe forwards supplementary groups through wire protocol - Enables proper permission checks for remote filesystems - Uses raw `SYS_setgroups` syscall for per-thread credential switching - Critical for vsock-based FUSE where server can't read /proc 7. **Resource Limits** (2025-12-08) - RLIMIT_NOFILE raised to 65536 on startup (both fc-agent and fcvm) - Prevents EMFILE errors during parallel test execution - Required for large-scale POSIX compliance test suites ## Technical Reference ### Backend Requirements fcvm runs two microVM backends behind a pluggable `Hypervisor` trait (`src/hypervisor/`), selected with `--hypervisor firecracker|cloud-hypervisor` (default: firecracker). Both cold-boot fcvm's kernel + initrd + rootfs and run a Podman container; fc-agent is injected at boot via initrd in both cases. Rootfs is Ubuntu 24.04 with systemd, Podman, and iproute2 for both. **Firecracker** (default) - **Kernel**: vmlinux or bzImage, boot args: `console=ttyS0 reboot=t panic=1 pci=off` on x86_64 (`reboot=k` on arm64) - **Rootfs**: ext4 or btrfs (via `--rootfs-type`) **Cloud Hypervisor** (ARM64) - **Kernel**: ARM64 `Image` (PE) format (not vmlinux/bzImage), boot args: `console=hvc0 reboot=k panic=1` (virtio console, no `pci=off`) - **Rootfs**: declared with `image_type=Raw` (CH does not take an ext4/btrfs filesystem-type flag) ### Network Modes | Mode | Flag | Requires Root | Performance | Port Forwarding | |------|------|---------------|-------------|-----------------| | Rootless (default) | `--network rootless` | No | Good | pasta CLI flags (-t/-u) | | Bridged | `--network bridged` | Yes | Better | iptables DNAT | | Routed | `--network routed` | Yes (+ IPv6 host) | Best (kernel line rate) | TCP proxy on the mapping's HOSTIP, else loopback IP | **Rootless Architecture:** - Holder process starts with `unshare --user --net`, UID/GID mappings written externally - Linux bridge (br0) connects pasta0 and tap-fc for L2 forwarding - Bridge preserves MAC addresses for proper pasta ARP/NDP learning - Namespace IP (10.0.2.1) on bridge enables health checks via nsenter - Guest uses pasta network (10.0.2.100) - Port forwarding via pasta CLI flags (-t/-u) - IPv6 supported with `--enable-ipv6` and native DNS proxying **Routed Architecture** (`src/network/routed.rs`): - veth pair connects VM namespace to host with native IPv6 kernel routing (no userspace proxy) - Each VM gets a unique IPv6 derived from host's /64 subnet via hash of vm_id - Network namespace with bridge (br0) connecting TAP and veth for L2 forwarding - Proxy NDP on default interface makes VM IPv6 routable from network fabric - ip6tables MASQUERADE for AWS VPC source/dest checks (skipped when `--ipv6-prefix` is set) - Port forwarding via built-in TCP proxy (setns + tokio relay). A mapping with a HOSTIP (`--publish '[::]:80:80'`) listens on that address, and `[::]` accepts IPv4 and IPv6 clients. A mapping without one listens on the VM's unique loopback IP (same allocation as rootless), which is allocated only when a mapping needs it - IPv4 stays internal to namespace (health checks only); all external traffic uses IPv6 - Egress proxy is NOT used — IPv6 goes natively through the kernel stack - `--ipv6-prefix` is also read from `FCVM_IPV6_PREFIX`. On a host whose own addresses are not a routable /64, set it to the host's delegated subnet: the routed tests pass no prefix, so without it they stop at "routed mode preflight check failed" before a VM boots **Loopback IP Allocation** (`src/state/manager.rs`): - Sequential allocation: 127.0.0.2, 127.0.0.3, ..., 127.0.0.254, then 127.0.1.2, etc. - Lock-protected with persistence to avoid conflicts ### btrfs CoW Reflinks **Performance: ~1.5ms disk copy (560x faster than standard copy)** **Architecture:** - All data under `/mnt/fcvm-btrfs/` — native btrfs used directly if host is btrfs, otherwise a loopback image is created (size from `paths.btrfs_size` in rootfs-config.toml, default 60G) - Base rootfs: `/mnt/fcvm-btrfs/rootfs/layer2-{sha}.raw` (~10GB raw disk with Ubuntu 24.04 + Podman) - VM disks: `/mnt/fcvm-btrfs/vm-disks/{vm_id}/disks/rootfs.raw` (sparse, expanded per `--rootfs-size`) - A snapshot keeps its disks (`disk.raw` and the extra disks) in `snapshots/{tag}/disk/`, a btrfs subvolume, and a clone's `disks/` directory is one `btrfs subvolume snapshot` of it with the disk renamed to `rootfs.raw`. A reflink copy of the disk is O(extents): 9 to 11 s per clone on a 154 GB www root disk with 860,177 extents, 11 of a 25 s restore ACK (2026-10-15); the subvolume snapshot costs one transaction commit whatever the extent count: 20 to 60 ms on a quiet host, 0.2 to 1.0 s on a shared devserver at load 14 to 46. The disks sit in their own subvolume because taking a snapshot flushes the source subvolume's dirty data first, and the memory image of a fresh snapshot can hold tens of GiB of it, while unlinking a shared file inside a snapshot costs one reference per extent (2.57 s for a 128 GiB memory image of 2,969,599 extents). Where the store cannot make a subvolume (not btrfs, or btrfs behind a FUSE map in a nested guest) `disk/` is an ordinary directory and the clone reflinks the disk as before, and so does an unprivileged clone of a snapshot another user made (a snapshot keeps every inode's owner, so it would get a disk it cannot write or remove). `rm -rf` removes subvolumes, so teardown is unchanged, but btrfs reclaims a removed subvolume's space on its cleaner thread after the rmdir returns, where an unlinked reflink freed its space at once; `df` right after many clone teardowns can read high for a while. - Initrd: `/mnt/fcvm-btrfs/initrd/fc-agent-{sha}.initrd` (injects fc-agent at boot) **Per-VM Disk Sizing (`--rootfs-size`):** - Default: `10G` — ensures at least 10G free space on root filesystem - After btrfs reflink copy, `ensure_free_space()` auto-detects filesystem type: ext4 uses `dumpe2fs` + `truncate` + `resize2fs`; btrfs uses `btrfs dump-super` + `truncate` (guest resizes btrfs at boot) - Sparse files: only written blocks use real disk space (50G file with 2G content uses ~2G) - Included in `FirecrackerConfig.rootfs_size` → affects snapshot cache key (different sizes = different snapshots) - Clones inherit parent's disk size naturally (reflink copies the already-resized file) - Implementation: `src/storage/disk.rs::ensure_free_space()`, called from `src/commands/podman/vm_config.rs` **Layer System:** The rootfs is named after the SHA of a combined script that includes: - Init script (embeds install script + setup script) - Kernel URL - Download script (packages + Ubuntu codename) This ensures automatic cache invalidation when: - The init logic, install script, or setup script changes - The kernel URL changes (different kernel version) - The package list or target Ubuntu version changes **Package Download:** Packages are downloaded using `podman run ubuntu:{codename}` with `apt-get install --download-only`. This ensures packages match the target Ubuntu version (Noble/24.04), not the host OS. The `codename` is specified in `rootfs-config.toml`. **Setup Verification:** Layer 2 setup writes a marker file `/etc/fcvm-setup-complete` and prints `FCVM_SETUP_COMPLETE` to serial console on successful completion. After the setup VM exits, fcvm checks the serial console output for the completion marker. If missing, setup fails with a clear error. The initrd contains a statically-linked busybox and fc-agent binary, injected at boot before systemd. **Setup**: Run `make setup-fcvm` before tests (called automatically by `make test-root` or `make container-test-root`). **Content-Addressed Caching** All assets are content-addressed - changing the input automatically creates new output: - **Kernel**: Cached by URL hash. Different URL = new kernel. - **Rootfs**: Cached by setup script SHA. Change script = new rootfs. - **Initrd**: Cached by fc-agent binary SHA. Rebuild fc-agent = new initrd. **NEVER manually delete cached assets.** Just rebuild and run `make setup-fcvm`: ```bash # Change fc-agent code, then: cargo build --release -p fc-agent make setup-fcvm # Creates new initrd with new SHA # Change rootfs-config.toml, then: make setup-fcvm # Creates new rootfs with new SHA ``` **Custom Kernel (Nested Virtualization)** Use `--kernel-profile` flag for named kernel configurations: ```bash # Build nested kernel with CONFIG_KVM=y fcvm setup --kernel-profile nested --build-kernels # Run VM with nested kernel profile sudo fcvm podman run --name my-vm --network bridged \ --kernel-profile nested \ nginx:alpine ``` **Kernel Build Architecture:** - **Config is source of truth**: All kernel versions and build settings flow from `rootfs-config.toml` - **No hardcoded versions**: Version numbers like `6.18.3` are ONLY in config, never in Rust code - **Dynamic build scripts**: Rust generates build scripts on-the-fly (no `build.sh` or `build-host.sh` in source) - **Config sync**: `make build` automatically syncs embedded config to `~/.config/fcvm/` via `fcvm setup --generate-config --force` - **Content-addressed**: Kernel SHA computed from `build_inputs` patterns (config + patches) Key config fields in `[kernel_profiles.nested.arm64]`: ```toml kernel_version = "6.18.3" # Version to download/build kernel_repo = "ejc3/fcvm" # GitHub repo for releases build_inputs = ["kernel/nested.conf", "kernel/patches/*.patch"] # Files for SHA kernel_config = "kernel/nested.conf" # Kernel .config patches_dir = "kernel/patches" # Directory with patches ``` **Creating/Editing Kernel Patches:** ```bash make kernel-patch-create PROFILE=nested NAME=0004-my-fix FILE=fs/fuse/dir.c make kernel-patch-edit PROFILE=nested PATCH=0002 make kernel-patch-validate PROFILE=nested ``` NEVER hand-write patches - the hunk counts will be wrong. Always use the helper script which generates proper `git format-patch` output. When a patch change doesn't fix the issue, the bug is incomplete root cause analysis - not "needs a workaround". Adding workarounds (env vars, flags) masks bugs. Find and fix ALL causes. ### NEVER Assume - Always Investigate **Disabling tests is NEVER acceptable.** When a test fails: 1. **Don't assume** the test is wrong or the limitation is fundamental 2. **Don't assume** someone else's workaround (like #[ignore]) was correct 3. **Investigate** the actual code path - read the library source 4. **Find the root cause** - there's usually a missing initialization or config **Example anti-pattern (O_WRONLY + writeback cache):** ``` ❌ WRONG: "O_WRONLY is fundamentally incompatible with writeback cache" → Added #[ignore] to test → Assumed the limitation was in FUSE kernel design ✅ CORRECT: Read fuse-backend-rs source code → Found get_writeback_open_flags() exists and promotes O_WRONLY → O_RDWR → But init() wasn't being called to enable the writeback flag → Fixed by calling inner.init(FsOptions::WRITEBACK_CACHE) → Test passes, no workaround needed ``` **The fix is almost always in the code, not in disabling tests.** NEVER manually edit rootfs files. The setup script in `rootfs-config.toml` and `src/setup/rootfs.rs` control what gets installed. ### Memory Sharing (UFFD) **Workflow:** ```bash # 1. Start baseline VM fcvm podman run --name baseline --network bridged nginx:alpine # 2. Create snapshot from running VM fcvm snapshot create --pid --tag my-snapshot # 3. Start memory server (serves pages via UFFD) fcvm snapshot serve my-snapshot # Creates /mnt/fcvm-btrfs/uffd-my-snapshot--.sock # 4. Spawn clones from the memory server fcvm snapshot run --pid --name clone1 ``` **How it works:** - Two modes, chosen by `--uffd-mode` / `FCVM_UFFD_MODE` (default `copy`; an NV2 snapshot is always served in copy mode): - `copy`: the server maps the snapshot file read-only, so the page cache holds one copy. Guest RAM is anonymous memory, and each fault is filled with `UFFDIO_COPY`, a private per-clone copy of the page. Once it has bound its socket, the server maps the file's holes (`SEEK_HOLE`) on a thread of its own and publishes the map when it is whole (`published the hole map` with `hole_mib` and `build_ms`). Until then every fault is served from the file, which reads a hole as zeros. A memory file on FUSE is flushed (`fdatasync`) first, because fuse-pipe with the writeback cache in a nested L1 answers `SEEK_HOLE` from what it has written back. A local filesystem answers it from the page cache and is not flushed. Once the map is published, a fault on a hole, or on a page the clone's balloon gave back, is answered with zeros and the file is not read. A read fault gets `UFFDIO_ZEROPAGE`: the kernel's shared zero page, read-only, so it costs the clone no memory until the guest writes it, and that write is a copy-on-write fault the server never sees. A write fault gets a private copy of zeros, mapped writable, so the write does not fault again. So does every fault of an NV2 clone, because on arm64 that copy-on-write drops every nested stage-2 mapping (see "Copy-on-write on the File backend"), and every fault on hugetlb memory, which refuses `UFFDIO_ZEROPAGE`. - `minor`: at startup the server copies the snapshot into one memfd (all-zero 4 KiB pages stay holes), seals it, and passes a read-only descriptor to every clone's Firecracker over the handshake socket. Guest RAM maps it `MAP_PRIVATE`, and each minor fault is resolved with `UFFDIO_CONTINUE`, which maps the shared page read-only. Clean pages have one physical copy across all clones; a page a clone writes becomes private to it. - Server uses tokio AsyncFd to handle UFFD events non-blocking - tokio::select! multiplexes: accept new VMs + monitor VM exits - Each VM gets dedicated async task (JoinSet) for page faults - All clone tasks share one `Arc`: the copy-mode mapping or the minor-mode memfd. - The serve runs until SIGTERM, or SIGINT when none of its clones is running (with clones running, a second SIGINT within 3 s), or until the server task fails. Shutdown stops this serve's clones first (SIGTERM, then SIGKILL after a timeout) and then cancels the server. The server stops accepting and keeps serving any still-connected clone until its Firecracker exit is seen through its pidfd. **Memory efficiency:** - UFFD path, copy mode: every page a clone faults, and with prefetch on (the default) every page of the replayed recorded set, is a private copy in that clone. Minor mode: faulted and replayed clean pages map the one shared memfd, and only pages a clone writes become private to it. - File-backend restores (`snapshot run --snapshot`): clones genuinely share clean pages via the page cache (MAP_PRIVATE) — measured 3x 1GiB clones ≈ 230MiB total PSS, with or without dirty tracking (#632). Hugepage and NV2 snapshots never take this path: `snapshot run --snapshot` restores them through an implicit UFFD server (`direct_restore_memory`) ### Working-Set Replay (`--uffd-prefetch`, default on) Demand paging is the UFFD path's latency tax. Commit 86a05b9a (not on main) reports, from a 2026-08-08 Chromium run that is not committed, ~56,300 faults per clone and 316 ms more request latency than a file-backed restore of the same image. Its ~5.6 us per fault equals 316 ms divided by 56,300. The same message gives a pairwise page-set Jaccard median of 0.927 across 8 clones, 82.2% of the union faulted by all 8, and 8.6% of faults on the page after the previous fault. The only committed fault record is `bench/chromium/results/faultbench-0813-073507-690998`. It used 2 GiB guests with prefetch at its unrecorded default, and its trace holds demand faults only, so replayed pages are absent from it. It gives a mean pairwise Jaccard of 0.69 in copy mode and 0.65 in minor mode (lowest pair 0.54), and 49% and 42% of the union faulted by every restore. It also shows 1.9 to 5.0% of faults on the next page, and 45 to 59% of faults per request in runs of 4 or more consecutive pages. Clones fault an overlapping set of pages in a different order each time, so the server records the SET and replays it. On the synthetic page, replay cut the request median from 762.6 to 477.9 ms on one golden (`results/reqbench-134b408e1a3d42feb866bb4c5fed774c` off, `results/reqbench-9c23e9d7da424351a390b5c1ddfd8e1d` on; commit e78350c5 names which run had prefetch on). Fault-around (below) covers neighbouring pages and is measured separately. - **Record**: every demand fault within the clone's recording window marks its snapshot file offset in a 4 KiB-granular bitmap (32 KiB per GiB of guest RAM). A fault that is on a page the clone's balloon gave back when it is first read is the exception: it is answered with zeros and not recorded. A fault on a hole in the memory file (copy mode) is answered with zeros and recorded like any other. The window starts at the clone's UFFD handshake and defaults to 300 s (`--uffd-prefetch-record-window` / `FCVM_UFFD_PREFETCH_RECORD_WINDOW`; 0 records nothing; issue #858). Later faults are served but not recorded. On handler exit the bitmap is unioned into the serve process's in-memory set and scheduled onto one bounded, coalescing background writer. The writer publishes `.working-set` beside the snapshot under an `flock` + atomic rename, and only writes when the union actually grew — so the steady state writes nothing, and a clone that was killed mid-restore gets completed by the next one instead of baking in a truncated set. Persistence is a performance hint: sidecar lock contention skips that publication attempt without delaying clone teardown, admission slots, or server shutdown; the in-memory union remains available for the next request. Publication also takes the snapshot generation lock shared from the final image-identity check through the atomic rename. Snapshot replacement takes that lock exclusively, so it either finishes before the check (the old clone declines to publish) or waits until an active publication completes; it cannot land in between. - **Replay**: at handshake the recorded set is coalesced into runs, mapped into that clone's regions, aligned to its page size, and populated in 2 MiB `UFFDIO_COPY`/`UFFDIO_CONTINUE` chunks. Replay starts during the snapshot load, when Firecracker hands the server its userfaultfd (fcvm loads with `resume_vm: false`), but nothing waits for it: fcvm patches the root drive and resumes the guest as soon as the load returns, and replay continues while the guest runs. The handler drains pending demand faults before every populate call it makes itself, so a demand fault waits behind at most one of the handler's calls. Replay yields to the runtime once per batch (one 2 MiB chunk, or 32 populate calls when the runs are small), because a yield after every small copy cost more than the copy. In copy mode the copies of plain runs go to worker threads (`--uffd-replay-threads`, `FCVM_UFFD_REPLAY_THREADS`, default 8 or the host's core count if lower): a large guest's set is millions of runs a few pages long (one 128 GiB guest: 2.36 M pages in 1.00 M runs, and merging runs across gaps of up to 4 pages would have cost 39 percent more pages), so replay is bound by `UFFDIO_COPY` calls, about 15 us each, and one thread took 15.3 s for that set while the guest demand-faulted 479,129 pages replay had not reached. The handler still plans the segments, steps over given-back pages and holes, copies zeros itself, and keeps serving demand faults, also while it waits for room in the workers' queue and for their last outcomes; the workers' copies run beside the handler's fault service, never ahead of it in the queue. A job is a batch of up to 64 runs, or one 2 MiB chunk of bytes, through a bounded queue: one queue slot, one wake of the handler and one outcome per batch, not per run. Each worker has its own handle on the clone's userfaultfd. A copy the kernel refuses while a REMOVE is queued and unread is retried a millisecond apart until it lands or the generation moves; if the retries do not clear the refusal within 2 s, the rest of the job falls back to demand paging. A clone on 2 MiB hugetlb pages copies one page per piece, since `UFFDIO_COPY` takes nothing smaller than the registered page. A balloon REMOVE the handler reads lets the kernel zap the range and accept copies into it again, so a run planned before it must not be copied after it: every job carries the generation it was planned under, the handler moves the generation when it records a REMOVE, a worker checks it before each copy, and a stale job goes back to the handler, which plans it again against the given-back pages, as it does with a job it still held when the generation moved. The check and the copy are kept apart from the read by a spin gate (`ReplayGate`): the handler holds it across each event read and waits for the copies in flight first, a worker holds it from its check through one copy (at most 64 KiB on 4 KiB pages, one page on 2 MiB hugetlb pages), and a worker that finds the handler reading waits for the read. The handler takes the gate only when the queue is readable (one `poll` before each chunk it plans), because a read-write lock taken for every read, the empty ones included, made eight threads slower than one (replay 16.5 s against 14.3 s on that set). The spin itself costs about 1.5 s of replay against no exclusion (6.4 to 6.9 s against 5.1 s); beginning the read and planning runs while the copies in flight finished measured the same (6.3 to 6.6 s), so the handler spins. The runs of a planned range come out of a cursor (`PlainRuns`) one at a time, so a dense 128 GiB set, 2 M runs of 64 KiB, costs the fault server no list. A pool that cannot start, or whose threads have all exited, means the handler copies alone, as it does in minor mode. The runs of a job that a REMOVE read while the workers are finished turned stale are left to demand paging, as a refused copy's are. The replay line carries `replay_threads`, `worker_jobs`, `worker_runs`, `worker_populated_mib` and `worker_jobs_replanned`. Every recorded run is planned. The plan is an iterator over the recorded bitmap, so planning costs no memory, its work is bounded by the image size, and no cap drops part of a large guest's set when it fragments into millions of runs. A recorded page the clone's balloon has given back by the time replay reaches it is stepped over. In copy mode a run of recorded pages that are holes in the memory file is mapped to the kernel's shared zero page with one `UFFDIO_ZEROPAGE`, as a read fault on a hole is, so a replaying clone holds no memory for zeros it only reads and a later write is the kernel's own copy-on-write. An NV2 clone, and hugetlb memory, get private copies of zeros instead. `hole_zero_filled_pages` on the replay line counts them (also in `prefetched_pages`), `hole_map_at_start` says whether the hole map was published when replay started (before it is, a recorded hole is replayed from the file), and the faults on holes are `hole_zero_filled` on the `VM exited` line. In minor mode on shmem the backing memfd leaves the snapshot's all-zero pages unwritten (a hugetlb backing is written in full), and each stays a hole until a clone touches it, which fills it in the memfd with no fault reaching the server. `UFFDIO_CONTINUE` refuses a hole, and a refusal would give up the rest of the run, so for a recorded run of those pages replay asks the memfd (`mincore`) which are filled now, maps them, and steps over the rest. A later clone's MINOR fault on a filled hole is recorded like any other fault. - **Page cache warm-up (copy mode)**: replay and demand faults both read the image through the serve's mapping, so with a cold page cache every source page is a synchronous major fault in the serve, ahead of the guest. One detached thread (`fcvm-ws-warm`, `src/uffd/warmup.rs`) calls `posix_fadvise(POSIX_FADV_WILLNEED)` on the recorded runs in ascending file offset, which is the order replay plans them in. A warm-up starts when the serve starts, and again when a clone is admitted while none is running, because the kernel evicts the image as clones grow under a long-lived serve. Measured on a 128 GiB guest (41.4 GiB recorded in 3.39M runs, memory file evicted before every run): replay took 94.5 s and 87.7 s on main, 32.3 s and 32.9 s with the warm-up when the clone connected as soon as the serve was ready, and 22.3 s when it connected after the warm-up had logged. Demand faults during replay fell from 992,535 to 562,321. The warm-up at serve start issued 3.49M requests in 18.4 s, and the one at clone admission, over an already cached set, took 0.8 s. A request for a page that is still cached measured 0.1 to 0.7 us, so walking a cached set of 3M runs costs that one thread between 0.3 and two seconds. Runs are never merged across a gap, so the warm-up requests exactly the recorded runs. What the kernel reads for a request is up to the filesystem: on btrfs with `compress-force=zstd`, a request for one 4 KiB page inside a compressed extent left 27 pages resident, the 128 KiB extent it sits in, and exactly 1 with incompressible data. A read fault on that page, with read-around off, left the same 27 and 1, so replay's own reads bring in the same extents. A run longer than 128 KiB takes several calls, because the kernel truncates one readahead request to `max(io_pages, ra_pages)` and still returns 0: one call over a 64 MiB range left 1,024 of its 16,384 pages resident on btrfs with a 4 MiB window, and the same range in 128 KiB calls left all of them. A request returns when its read is queued, not when it completes, so whether a clone that connects during a warm-up finds its pages cached is a race, not a guarantee. Nothing waits for the thread: the ready record, clone admission, fault service and shutdown do not depend on it, and a refused request is counted with only the first logged. Cancelling the serve sets a flag the thread reads before every request, so at most one further request starts after the stop returns. Every warm-up logs one line: `asked the kernel to read the recorded working set into the page cache` with `trigger`, `runs`, `pages`, `failures` and `elapsed_ms`, or, for a fresh snapshot, that no working set is recorded yet. Minor mode reads the whole image into its memfd at startup, so it has nothing to warm, and `--uffd-prefetch off` starts no warm-up. Known limits, with numbers, are in the module doc: the warm-up is not paced, so on a cold cache another running clone's major fault waits behind its readahead, it can outrun replay when free page cache is smaller than the recorded set, and a minor-mode server starting on the same image drops what it loaded. - **Idle snapshots leave the page cache when a server starts** (`src/uffd/release.rs`, #1066): a server that starts asks the kernel to drop the memory file of every other snapshot that nothing is reading (`posix_fadvise(POSIX_FADV_DONTNEED)`, on a detached thread). Without it, a restore that follows restores of another snapshot reads its recorded set several times over: the kernel keeps the other snapshot's recently used pages and evicts the ones the new restore has just read. Measured on a 236 GB host with two snapshots of a 128 GiB guest and recorded sets of 40 to 42 GiB: 180 s to guest ACK, 7m20s to healthy and 465 GiB read with the other snapshot's file cached, against 65 s, 2m02s and 112 GiB with it dropped first. The reverse switch took 58 s, 4m20s and 254 GiB against 59 s, 1m51s and 87 GiB. A dropped memory file costs its own next restore about 40 s to guest ACK (60 s cold against 24 s warm). In use is a shared `flock` on the memory file, held by a memory server from before it maps the file until it is dropped. A pass takes the lock exclusively, without waiting, around each drop, so two servers that start together cannot drop each other's file, and the lock goes when its holder dies. fcvm's state is not asked: a server writes its state after it has started reading, and a running VM also names the snapshots it was created from. A directory a create is still writing (`.creating`) or has just replaced (`.old`) is never touched. A `snapshot run --snapshot` restore through Firecracker's File backend starts no server, releases nothing and holds no mark, so a pass can drop the cached pages of its snapshot that its VMM has not touched yet. - **Invalidation**: keyed by the exact `config.json` digest plus the memory image's (`len`, nanosecond `mtime`) identity, not a memory-image content hash — SHA-256 of a 2 GiB image measures 1.4 s at 1.5 GB/s here, which costs more than the mis-prefetch it would prevent. The config digest makes atomically installed generations distinct even under timestamp granularity. The inode and device are not in the key, so a store restored from a tar archive (`--format=posix`, which keeps nanosecond mtimes) keeps its sets. Safe because a working set only says WHICH offsets to copy; the BYTES always come from the file being served, so a stale set wastes a copy and can never corrupt a guest. - **Inheritance (`snapshot create --inherit-working-set`)**: a new snapshot starts with no set, so its first restore faults in on demand every page the guest touches, and a recording pass has to run before restores are fast. With the flag, a Firecracker memory snapshot whose VM has a parent (the snapshot it was restored from or last snapshotted to) gets the parent's recorded set as its own sidecar. A snapshot of a restored clone is the parent's image with the clone's writes merged at the same offsets, so the parent's offsets name the same guest pages. The parent's set is read while the create still pins the parent's generation, and written into the new snapshot's sidecar before the create gives back that snapshot's generation lock, so a serve waiting on the lock loads the set when it starts. The write takes only the sidecar's own lock, because the create already holds the generation lock exclusively. A parent that is the target itself, a different image length, or any read or write failure logs a warning and leaves the snapshot without a set. The inherited set is a hint like any other: serves of the new snapshot keep recording on top of it, and since sets only grow, a long chain of inherited sets grows with it (#955). Measured on a 128 GiB guest, one per-diff snapshot: a clone of it with the inherited set (13.7 M pages) was ready in 317 s, the same as with a set recorded for it in a 485 s pass (6.6 M pages, 315 s) and against 606 s with no set, but held 52.7 GiB at healthy against 25.4 GiB. Inheriting skips the recording pass at the price of every clone's memory, until a recording that replaces the set (rather than adding to it) takes its place. - **Isolation**: replay touches pages the guest never asked for, so it must stay private. `UFFDIO_COPY` writes into the clone's own anonymous memory, and `UFFDIO_CONTINUE` installs a read-only PTE that copies on write. Proven by `prefetched_pages_are_private_to_each_clone` (mechanism level, milliseconds) and `test_snapshot_clone_working_set_replay` (two clones, cross-write invisibility, `memory.bin` byte-identical). - **Measuring it**: the memory server logs `replayed recorded working set` with `prefetched_pages`, and `VM exited` with `fault_count`. Compare a recording clone against a replaying one; `--uffd-prefetch off` (or `FCVM_UFFD_PREFETCH=off`) gives an inert baseline arm — no recording, no replay, no files. In minor mode the replay line's `filled_holes_mapped` counts the pages replay mapped at holes earlier clones filled. - **Replay trades memory for latency.** A replaying clone materialises the recorded set at restore instead of faulting it in as it runs. Commit 86a05b9a reports, from a Chromium run that is not committed, that 98% of faults arrived within 750 ms. The set is a union over clones' recording windows and only grows. In copy mode every replayed page is a private copy, so a clone also holds pages it would never have touched, including a clone that is created and destroyed at once. On one 128 GiB guest the set grew from 3.7M pages after one clone to 9.4M after six (issue #955). PR #956 measured two clones on that guest at first healthy. They held 23.0 and 23.2 GiB of private memory when replay planned 3.7M of the 9.4M recorded pages (main's run cap at the time), and 35.8 and 35.9 GiB when it planned all 9.4M. Turn replay off for workloads that spawn many clones which never run. #955 tracks bounding the set. **End of clone (`PeerVmm`)**: a userfaultfd reports nothing when the process that created it dies — measured on this kernel, `poll` returns 0/revents=0 forever and `read` returns EAGAIN — because the server still holds its own reference. The server therefore obtains the connecting Firecracker's pidfd atomically from the accepted socket with `SO_PEERPIDFD`, before handshake or admission, and selects that exact handle alongside UFFD readiness. PIDs are never re-resolved, so process reuse cannot redirect observation or the fail-closed SIGKILL. The pidfd edge ends a normal handler and schedules the working-set union for best-effort background publication. Handshake and page-fault service failures follow the existing fail-closed path and terminate the pinned VMM rather than leaving its guest wedged on an unserved fault; hint load, speculative-population, and persistence failures degrade to ordinary demand paging because they do not stop fault service. ### Fault-Around (`--uffd-fault-around `, default off, experimental) Replay covers the pages every clone touches. It cannot cover a clone that does fresh work right after restore, because the memory that work allocates lands on different pages in every clone. Measured on a 128 GiB guest before this option existed: the first real page render in a fresh clone took 541 s while the clone demand-faulted 3.3 million pages (12.4 GiB), one userfaultfd round trip each. The same render took 305 s in a clone whose recorded set already held those pages (110 MiB faulted). Fault-around trades memory for those round trips. The one measurement so far is mixed, which is why the option is experimental. One snapshot of a 128 GiB guest, 64 KiB granules, a host whose page cache cannot hold the memory file, and a fresh clone restored through a recording of 4.07 million pages that covers the path to healthy and nothing else: | | off | `--uffd-fault-around 65536` | |---|---|---| | restore to healthy | 2m23s | 4m07s | | first real page, last byte | 518.4 s | 336.2 s | | pages installed beyond the demanded ones | none | 17,545,557, 14.5 per fault | | clone private memory during the first page | 14.5 to 27.0 GiB (earlier run) | 52.2 to 75.8 GiB | The first page got 35 percent shorter, and the restore itself got slower and much larger. Almost every granule held one demanded page. A locality projection made earlier, from a clone that had run for a long time, put the cost at 2.1 times the memory for 7.6 times fewer faults at 64 KiB. Touched pages are dense in such a clone and sparse right after a restore, so that projection does not describe this period. The extra copies also sit in the fault loop while the guest is recovering. On this workload a recorded set that already covered the first page did better on both time and memory (304.7 s). Not measured: granules under 64 KiB, fault-around held back until replay has finished, and a host whose page cache holds the memory file. One run with 2 MiB granules on the same host cost the restore more: 114.5 s to guest ACK and 4m44s to healthy, against 27.6 s and 2m23s with the option off. The cost to the restore grows with the granule. - **What it does**: copy mode only. On a demand fault the server copies the faulting page and wakes its vCPU exactly as it does without the option, so that fault's latency is unchanged. It then populates the rest of the granule that holds the page: the pages after it first, then the pages before it. The granule is `BYTES` aligned in snapshot file offsets and clipped to the memory region that took the fault. Pages that are already present are stepped over, and so are pages the clone's balloon gave back. - **The trade is density**: every fault privately materialises the rest of its granule, bar pages the balloon gave back, whether or not the guest touches it. At 64 KiB that measured 14.5 extra pages per fault right after a restore. Idle clones are unaffected, because a clone that does not fault populates nothing. - **When to use it**: as an experiment, on large guests whose clones do real work right after restore, where the time goes to round trips for pages no earlier clone touched and the memory is available. A recorded working set that covers the work is the better fix when one can exist. Leave it off for hosts packed for density. - **Values**: 0 (off), or a power of two above the host page size through 2097152 (2 MiB). One host page is refused: it is the page a demand fault already serves, so the option would be on and do nothing. Anything else is a usage error, and so is a non-zero value with `--uffd-mode minor`, where it is not implemented: minor faults are resolved with `UFFDIO_CONTINUE` onto shared pages and have no populate step to extend. A hugepage clone's own page is already 2 MiB, so the option does nothing for it and the server logs that. `FCVM_UFFD_FAULT_AROUND` carries the same value to the implicit server inside `snapshot run`. - **It is speculation, under replay's rule**: it goes through the chunk population replay uses (`prefetch::populate_chunk_counted`), and a refused copy, which includes the zero-progress `EAGAIN` while `mmap_changing` is raised, abandons that granule to demand paging. It never fails a clone. A vCPU that faults inside the granule meanwhile is woken by the populate itself, because `UFFDIO_COPY` wakes every sleeper in the range it installs. It stands down in three cases. While any fault is parked it serves demanded pages only: a parked fault is retried between batches and fails the clone if it is still refused 2 s after it was parked or after the last REMOVE event read, so nothing optional may lengthen a batch. After a demand copy that found its page already present it populates nothing, because another populator owns that range. After a fault on a page the clone's balloon gave back, which is answered with zeros, it populates nothing either. A fault on a hole in the memory file is answered with zeros and gets fault-around like any other. A page of the granule that the balloon gave back, or that is a hole, is stepped over. - **Limitation: recording under the option is thin, and replay does not make up for it**: only the demanded page is recorded, and a page that fault-around installed is never recorded, because the guest never faults on it. So with the option on the recorded working set converges one demanded page per granule per clone: clone 1 faults on A and records A, clone 2 replays A alone, faults on B and records B, and so on. Replay restores the recorded pages and nothing more, so it never restores what fault-around would have installed. That is a reason to record with the option off: a set recorded that way holds every page the guest faulted on, not one per granule. - **Why replay does not expand recorded runs to their granules**: it would hide the limitation above, at a cost the measurement argues against. Right after a restore the demanded pages are sparse: at 64 KiB, 14.5 of the 16 pages of a granule were installed beyond demand. Expanding every recorded run would multiply the replayed memory by up to the granule factor, in a restore the option already slows. Not expanding costs one extra fault per granule per clone until the set converges. `replay_populates_only_the_recorded_pages_with_fault_around_on` pins it. - **What the option leaves behind**: the sidecar stays a record of demand, so it is still valid with the option off or with a smaller granule. It is thinner than a record made without the option, and each page missing from it costs one demand fault, once, after which it is recorded. - **Measuring it**: at handler exit the serve logs `fault-around populated pages beyond the demanded ones` with `granules`, `extra_pages`, `extra_mib` and `refused_granules`, next to the existing `VM exited` line and its `fault_count`. Compare `fault_count` between an arm with the option and an arm without it, and read the memory cost from `extra_mib`. ### FUSE Parallelism (fuse-pipe) **Kernel clone fd model (`FUSE_DEV_IOC_CLONE`):** Each cloned fd is a complete request-response pipeline. A request dequeued from clone_fd_A is pinned to that fd — the response **must** be written back to clone_fd_A (`fuse_request_find()` searches only per-fd `fpq->processing`, not a global list). | What | Serialized | Parallel | |------|-----------|----------| | Dequeue from `fiq->pending` | Yes — brief `fiq->lock` spinlock shared across ALL fds | N/A (one request dequeued at a time) | | Copy request to userspace | No lock held | Yes — each fd copies independently | | Process request | N/A | Yes — each thread processes its own request | | Write response | Per-fd `fpq->lock` (no cross-fd contention) | Yes — each fd writes independently | | FORGET/BATCH_FORGET | Fire-and-forget, no response written | Dequeued from shared `fiq->forget_list` | Without clone fd, all threads share one fd and contend on both `fiq->lock` and the single `fpq->lock`. **fuse-pipe layers:** | Layer | Serialized | Parallel | |-------|-----------|----------| | Client reader threads (`mount.rs`) | Dequeue briefly serialized (kernel `fiq->lock`) | N threads with N cloned fds read/write independently | | Multiplexer (socket) | Socket write serialized by channel | Request submission lock-free; responses routed by unique ID | | Server socket reader (`pipelined.rs`) | Reads requests sequentially from socket | — | | Server handler dispatch | — | Each request dispatched via `tokio::spawn` + `spawn_blocking`; concurrent on tokio blocking pool | **Implication:** `FilesystemHandler` implementations (including `RemapFs`) must be thread-safe. Shared state requires atomic operations or lock-free data structures (e.g., `DashMap::entry()` for atomic check-and-insert). ### FUSE Passthrough Performance (fuse-pipe) **Benchmark**: 256 workers, 1024 files × 4KB #### Parallel Reads | Readers | Time (ms) | vs Host | Speedup vs 1 Reader | |---------|-----------|---------|---------------------| | Host FS | 10.7 | 1.0x | - | | 1 | 490.6 | 45.8x slower | 1.0x | | 16 | 63.7 | 5.9x slower | 7.70x | | **256** | **57.0** | **5.3x slower** | **8.61x** | #### Parallel Writes (with sync_all) | Readers | Time (s) | vs Host | |---------|----------|---------| | Host FS | 0.862 | 1.0x | | 16 | 2.435 | 2.8x slower | | **256** | **2.765** | **3.2x slower** | **Recommendation**: Use 256 readers for mixed workloads. ## Build Instructions Run `make help` for all targets. See README.md for details. ### How Setup Works **Setup is explicit, not automatic.** VMs require kernel, rootfs, and initrd to exist before running. **Two ways to set up:** 1. **`fcvm setup`** (explicit, works for all modes): - Downloads kernel and creates rootfs - Required before running VMs with bridged networking (root) 2. **`fcvm podman run --setup`** (rootless only): - Adds `--setup` flag to opt-in to auto-setup - Only works for rootless mode (no root) - Disallowed when running as root - use `fcvm setup` instead **Without setup**, fcvm fails immediately if assets are missing: ``` ERROR fcvm: Error: setting up rootfs: Rootfs not found. Run 'fcvm setup' first, or use --setup flag. ``` **What `fcvm setup` does:** 1. Downloads the released default fcvm kernel selected by the explicit architecture profile (content-addressed by version, arch, and manifest SHA) 2. Downloads packages using `podman run ubuntu:noble` with `apt-get install --download-only` - Packages specified in `rootfs-config.toml` (podman, crun, fuse-overlayfs, skopeo, fuse3, haveged, chrony, strace) - Uses target Ubuntu version (noble/24.04) to get correct package versions 3. Creates Layer 2 rootfs (~10GB): - Downloads Ubuntu cloud image - Boots VM with packages embedded in initrd - Runs install script (dpkg) + setup script (config files, services) - Verifies setup completed by checking for `/etc/fcvm-setup-complete` marker file 4. Creates fc-agent initrd (embeds statically-linked fc-agent binary) **Kernel source**: fcvm builds the pinned Linux release from an immutable Firecracker base config. Default arm64 and amd64 fragments require FUSE plus `CONFIG_INET_DIAG`, `CONFIG_INET_DIAG_DESTROY`, and `CONFIG_PACKET` for safe snapshot socket cleanup. ### Data Layout Paths are configured in `rootfs-config.toml` under `[paths]`: - `assets_dir`: Content-addressed files (shared across nesting levels) - `data_dir`: Mutable per-instance data (separate per nesting level) ``` assets_dir (default: /mnt/fcvm-btrfs) ├── kernels/vmlinux-{profile}-{version}-{arch}-{sha}.bin # Released source kernel ├── rootfs/layer2-{sha}.raw # Base image (~10GB, SHA of setup script) ├── initrd/fc-agent-{sha}.initrd # fc-agent injection (SHA of binary) ├── image-cache/sha256:{digest}/ # Container image layers └── cache/ # Downloaded cloud images data_dir (default: /mnt/fcvm-btrfs, override per nesting level) ├── vm-disks/{vm_id}/disks/ # CoW reflink copies per VM ├── state/{vm_id}.json # VM state files ├── reclaim/{euid}/{id}.json # what a live fcvm made on the host; the next launch reclaims a dead one's (src/reclaim.rs) ├── pmem/{sha}.img # --pmem copies, one per image version, read-only └── snapshots/{name}/ # Firecracker snapshots └── disk/ # the snapshot's disks, a btrfs subvolume (clones snapshot it) ``` ## Key Learnings ### Serial Console - Problem: VM booted but no output after init - Fix: Kernel boot args include `console=ttyS0` (done automatically) ### Clone Network Configuration - Problem: Guest retains original static IP after snapshot restore - Root cause: Firecracker's network override only changes TAP device name, not guest IP - Fix: Configure TAP devices on SAME subnet as guest's original IP ```bash # Wrong: TAP on different subnet than guest ip addr add 172.16.201.1/24 dev tap-vm-c93e8 # Guest thinks it's 172.16.29.2 # Correct: TAP on same subnet as guest ip addr add 172.16.29.1/24 dev tap-vm-c93e8 # Guest is 172.16.29.2 ``` - Reference: https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/network-for-clones.md ### KVM Requirements - Firecracker REQUIRES `/dev/kvm` - On AWS: c6g.metal (ARM64) or c5.metal (x86_64) work; c5.large does NOT - On other clouds: use bare-metal or hosts with nested virtualization ### DNS Resolution in VMs - VMs use host's DNS servers directly (read from `/etc/resolv.conf`) - For systemd-resolved hosts, falls back to `/run/systemd/resolve/resolv.conf` - Traffic flows: Guest → NAT → Host's DNS servers - No dnsmasq required ### Container Resource Limits (EAGAIN Debugging) **Symptom:** Tests fail with "Resource temporarily unavailable (os error 11)" or "fork/exec: resource temporarily unavailable" **Debugging steps:** 1. Check dmesg for cgroup rejections: ```bash sudo dmesg | grep -i "fork rejected" # Look for: "cgroup: fork rejected by pids controller in /machine.slice/libpod-..." ``` 2. Check actual process/thread counts (usually much lower than limits): ```bash ps aux | wc -l # Process count ps -eLf | wc -l # Thread count ps -eo user,nlwp,comm --sort=-nlwp | head -20 # Top by threads ``` 3. Check container pids limit (NOT ulimit - cgroup is separate!): ```bash sudo podman run --rm alpine cat /sys/fs/cgroup/pids.max # Default: 2048 (way too low for parallel VM tests) ``` **Root cause:** Podman sets cgroup pids limit to 2048 by default. This is NOT the same as `ulimit -u` (nproc). The cgroup pids controller limits total processes/threads in the container. **Fix:** Use `--pids-limit=65536` in container run command (already in Makefile). ### Pipe Buffer Deadlock in Tests (CRITICAL) **Problem:** Tests hang indefinitely when spawning fcvm with `Stdio::piped()` but not reading the pipes. **Root cause:** - Linux pipe buffer is 64KB - fcvm outputs 100+ lines of Firecracker serial console logs - When buffer fills, child process blocks on `write()` syscall - This prevents ALL async tasks in the child (including health monitor) from running - Result: VM never becomes "healthy", test times out **Symptoms:** - Test works manually with `| tee /tmp/log` (because tee consumes output) - Test hangs when run via `cargo test` - State file timestamp never updates (health monitor blocked) - VM is actually running fine, just not being monitored **Fix:** NEVER use `Stdio::piped()` unless you actively consume the output. Use the `spawn_fcvm()` helper which uses `Stdio::inherit()`: ```rust // WRONG - will deadlock! let child = tokio::process::Command::new(&fcvm_path) .args([...]) .stdout(Stdio::piped()) // Never read = deadlock .stderr(Stdio::piped()) // Never read = deadlock .spawn()?; // CORRECT - use the helper let (mut child, pid) = common::spawn_fcvm(&["podman", "run", "--name", &vm_name, ...]).await?; ``` **The helper enforces:** - `Stdio::inherit()` for stdout/stderr - output goes to parent (visible with `--nocapture`) - No deadlock because parent's stdout/stderr handle the data - Consistent error handling and PID extraction ## Exec Command Flags `fcvm exec` uses `-i` and `-t` separately, matching podman/docker: - `-t`: allocate PTY (for colors/formatting) - `-i`: forward stdin - `-it`: both (interactive shell) - neither: plain exec ### `fcvm exec` LANDS IN THE CONTAINER BY DEFAULT. The guest needs `--vm`. ``` -c, --container Execute inside container (default) --vm Execute in the VM instead of inside the container ``` A bare `fcvm exec --pid -- ` runs in the **container**, not the guest OS. They are different machines with different tooling, so a diagnostic aimed at the wrong one comes back EMPTY rather than wrong, which reads like a finding: | | container (`nginx:alpine`) | guest (Ubuntu 24.04, `--vm`) | |---|---|---| | `iptables` | absent | `/usr/sbin/iptables` | | `bash` / `ss` / `curl` | absent | `/usr/bin/{bash,ss,curl}` | | `sh`, `wget`, `netstat` | busybox | present | On 2026-08-15 that cost several rounds of a flake hunt. Netfilter probes returned nothing, and the conclusion written down was "the guest rootfs carries no iptables and no nft, so guest netfilter state is unreadable" - a statement about alpine, presented as a fact about Ubuntu, and committed as a code comment. The rules were readable the whole time, and reading them is what exonerated the loopback containment DROP (packet counter still 0 at failure) and moved the search to the virtio-net RX path. Two rules follow: 1. **An empty probe result is not evidence. It is a broken probe until proven otherwise.** Run every diagnostic against a HEALTHY VM first and confirm it returns real output. Both of these looked like findings and were defects in the instrument: `exit=127` from `curl` in a guest image that has no curl, and `exit=127` from passing a whole command line as one argv - `fcvm exec -- 'ss -ltn | head'` execs that string as a program name, so it needs `sh -c` as separate arguments. 2. **Use absolute paths.** exec hands over a minimal PATH: `iptables` fails where `/usr/sbin/iptables` works. The same split exists in the test helpers: `exec_in_vm(pid, &cmd)` versus `exec_in_container(pid, &cmd)` (`tests/common/mod.rs`). **NO backward compatibility wrappers.** When the API changed from `run_tty_mode(stream)` to `run_tty_mode(stream, interactive)`, all callers were updated directly - no deprecated functions or compatibility shims. ## References - Main documentation: `README.md` - Performance guide: `PERFORMANCE.md` - Design specification: `DESIGN.md` - Firecracker docs: https://github.com/firecracker-microvm/firecracker/blob/main/docs/getting-started.md