# Test Plan Milestone 27a authority qualification retains the selected deployment manifest, strict-intent and legacy compatibility results, gateway refusal/audit evidence, host execution crash/restart evidence, and the distinct gateway microbenchmark specified by [BENCHMARKS.md](BENCHMARKS.md). Host simulations cannot accept root policy, dedupe or debug-memory behavior; exercise those paths on the selected QEMU/Pi target. The normal five-stage gates still apply. Use `COHSH_BASE_MANIFEST` and `COHSH_GATED_MANIFEST` for explicitly provisioned regression sources, with `COH_AUTH_TOKEN_REF` selecting the same Queen key. References fail closed; the runners do not fall back to bootstrap secrets. The generated Python profile and retained public verification key must match the selected image. Refresh embedded operator snippets with coh-rtc's explicit `--embed-userland-doc docs/USERLAND_AND_CLI.md` option after canonical generation. For literal capability tickets in the checked-in TCP regressions, the runner invokes `coh-rtc-regression-tickets`. It verifies the original development MAC, then signs the exact same claim bytes with the selected manifest's issuer. Expiry, role, subject, mounts, scopes, quotas and expected responses remain unchanged. An invalid original MAC or any claim-byte change fails before the client runs. Retained private script copies and hash-only binding records are included in each transport result's evidence inventory; originals are never edited. This fixture provisioning does not relax live placeholder refusal or qualify a production profile through compatibility scripts. This is the execution and acceptance guide for the complete Cohesix stack. Use the [action catalog](#canonical-action-catalog) for executable checks and the [26e enforcement map](#milestone-26e-executable-enforcement) for the stage, collector, and promotion gates that own milestone closure. Test bodies and evidence validators own detailed assertions; do not maintain a parallel prose inventory of individual tests. Current implementation status belongs in [STATUS.md](STATUS.md); task scope, candidate chronology, and qualified result history belong in [BUILD_PLAN.md](BUILD_PLAN.md). Do not append another run diary here. ## Coverage without duplicate execution For the [27b–27g release sequence](https://lukeb-aidev.github.io/cohesix/docs/BUILD_PLAN.html#post-26e-investment-constrained-delivery-sequence), each component milestone owns focused acceptance; 27g owns exact assembled CLI/Python/CI/native-app journeys and the complete applicable staged release gates. Foundation conformance does not require downstream recipes or UI. Historical milestone IDs in catalog entries, generated evidence and retained records resolve through the [roadmap mapping](BUILD_PLAN.md#roadmap-id-mapping); this planning change does not rename those artifacts or alter any evidence threshold. M27b's scoped host matrix is `configs/provider_conformance.toml`, executed by `scripts/ci/provider_conformance_run.sh --matrix configs/provider_conformance.toml --state-dir `. Select a single `--evidence-only`, `--executors-only`, `--observability-only`, `--packaging-only`, `--identity-only` or `--registry-only` group for the contract being changed; do not rerun unaffected groups. `--validate-only` emits NOT_RUN case rows and cannot stand in for execution. The bounded runner retains exact commands, source inventory, generated graph/profile requirements, output hashes, result classes and elapsed time. It stops on the first failed selected case. These are host contracts, with no implicit native/provider/Worker/target promotion. Native discovery uses the separate `--native-providers --live-reference` lane; observations remain INCOMPLETE until the corresponding execution and evidence obligations are independently met. Historical failed attempts remain retained. Recoverable CUDA recipes use `cargo test --locked -p coh --test recipe --test workflow --test evidence_case`, the `coh-rtc` `recipe::tests` contract check and the focused Python playbook wrapper tests. These prove deterministic journal, reuse, cancellation/accounting, corruption and diagnostic contracts. A fresh recipe on the selected native CUDA lane must separately prove actual output, lost-ACK/controller recovery and confirmed termination using the existing QEMU admission and independent enrolled evidence. Reuse unchanged 27b native executor refusal/cancel/restart evidence at its original component identity; controller fixtures do not replace it. Compile affected host implementations and run generated/Test Plan consistency. Whole-release staged gates remain 27g. Private LoRA release uses the focused `coh` `peft_release` integration tests, existing recipe/workflow guards, shared action and Worker receipt contract tests, affected compiler profile tests and Python wrapper/native-input tests. Run native attestation input tests in the qualified environment with all required packages; an optional-dependency skip elsewhere is not that evidence. These checks own comparison, phase intent, uncertainty, fresh recovery authority, confinement and forgery contracts. Actual import/training, native evaluation, serving promotion, regression refusal and failed-canary verified rollback require fresh admitted execution on the selected pinned CUDA host, with exact WorkerLora receipts and independently enrolled native evidence. Host journal fault tests cannot substitute for those outcomes. Compile exact affected host/target profiles and run generated and Test Plan consistency; the full staged release campaign remains owned by 27g. The selected M28b lifecycle adds `m28b-peft-live` and `m28b-serving-live` to the provider matrix and canonical action catalog. Run each with a fresh private reference file and state directory against the exact-source KVM Queen and Linux AArch64 NVIDIA host. The PEFT case needs separate admitted train, independent import and checkpoint-resume operations. Train and resume references use `expected_adapter_sha256 = "observed"` because their weights are produced during execution; the live runner binds the digest from both native scan and stage. Import and promote references pin the supplied adapter digest before admission. The serving case needs a fixed base/incumbent/candidate comparison, pre-load rejection, a separate streaming application request after promotion, and interruption followed by an observed incumbent request at the frozen generation. Preserve every negative result and original identity. Record approved staging transfers and a live host/network observation or enforced egress boundary for the private reference; an offline environment variable alone cannot establish no payload egress. The focused Rust, compiler, Python and UI projection tests cover deterministic contracts; they cannot replace any of those live outcomes. Run `python3 -m pytest -q tools/cohesix-py/tests/test_model_release.py tools/cohesix-py/tests/test_playbooks.py` for the Python operator path. Require wrong-operation and wrong-request reports, contradictory pending/terminal states, and changed deployment requests to refuse; lost submission response must use read-only recovery with the original operation. Only the shared CLI verifier's successful requested outcome may set the Python verified flag. MCP and A2A projections inherit these underlying identity and evidence boundaries when their separate 28d/28e tasks are activated; M28b Python tests are not protocol evidence. This component lane does not qualify Pi hardware or the assembled Release B. Installation/adoption/CI uses focused signed-package refusal, explicit Python archive/install, service-rendering and journey identity/outcome/retry tests. Install selected packages into fresh directories, build/install the SDK in a clean interpreter, and exercise both shipped lifecycle paths and signed outcome verification without source-checkout imports. Retain exact package/source digests and first-use friction. Unchanged native execution evidence from 27b–27d retains its original identity; package or CI tests cannot promote it to a new physical target or integrated release. Compile affected native profiles and run generated/Test Plan consistency; 27g owns the complete assembled campaign. Select tests by the contract at risk, its real execution surface, and the claim being made. Record the affected surfaces, catalog actions, expected positive and negative outcomes, exact profiles, and any evidence still missing. A successful command or test count is not a coverage argument. M28's protocol-control foundation uses `cargo test --locked -p coh-rtc --test agent_protocol_controls`, `cargo test --locked -p hive-gateway --test agent_protocol_controls`, and `cargo test --locked -p coh --lib doctor::` with `scripts/check-generated.sh`. These are source/host checks for the eight effective flag combinations, malformed profile refusal, startup override refusal and existing REST availability. They cannot close MCP/A2A listener or job conformance; those belong to M28d and M28e. M28 shared jobs and standing authority require their separate focused recovery/concurrency checks and the live `m28-jobs-live` and `m28-authority-live` cases before milestone acceptance. M28a selects the generated registration and existing executor recovery checks, then runs `m28a-workloads-live` separately for the useful reference and one independently configured adaptation on each advertised native owner. A case report covers one fresh admission and records exact source, target, package, device, input, output and independent pixel verification. The milestone ledger must link all required reports and the `m28a-recovery-live` cancellation and interruption record. Direct native smoke, Python output checks and an older M28 target result do not replace these admitted exact-source observations. M28c's first gate is `m28c-platform-live` on the supported macOS 27 Apple Silicon host. It binds a development-signed installed app and extension, Shortcuts action discovery and native Apple AI/Metal availability to source. For M28c completion, retain source-bound selected component evidence for an installed Shortcuts start/inspect/cancel and revoked-scope path using original authenticated service-job identities and a source-linked on-device explanation. The final installed app must resolve a saved action and refuse the revoked scope; this does not claim a fresh authorised service effect on final bytes. Also collect actual local MLX training and final installed SwarmUI inference/evaluation with observed Metal, model/data/adapter hashes and resource bounds, and signed vMLX 1.6.65 loopback compatibility with disposable before/after-hashed fused model bytes and four frozen operational responses. Final app and extension bytes must pass Developer ID signature, Apple notarisation, stapling and Gatekeeper checks. The native-action, local-compute, optional-vMLX, assistance and final-installed records keep separate proof limits; focused tests cannot substitute for these live observations. Spoken Siri invocation is outside M28c's developer value gate. Local MLX/vMLX results carry no Cohesix job, accepted generation or release authority. Milestone 28c1 owns fresh `m28c1-mlx-live` and `m28c1-vmlx-live` admission, canary and governed rollback evidence before those deployment claims or mixed MLX/CUDA use cases are accepted. M28d selects `cargo test --locked -p coh-rtc --test agent_protocol_controls`, `cargo test --locked -p hive-gateway --test agent_protocol_controls --test mcp_protocol --test mcp_workflows`, the focused `cohsh` and SwarmUI manual checks, and `scripts/check-generated.sh`. The compiler checks all eight flag combinations; the endpoint checks real HTTP/stdout framing, auth, Origin, revision, malformed and oversized input, scoped discovery and original identity mapping. They are host contracts, not live CUDA or PEFT outcomes. The selected native NeMo 1.9.0 `mcp_client` with MCP SDK 1.29.1 probes the Streamable HTTP revision and custom-header mode against the running gateway. For milestone acceptance, `m28d-mcp-live` binds exact source, KVM image, selected private PEFT profile, native Orin artifacts, ordinary client package and configuration, original job identities and shared-verifier results. The named client must complete CUDA and PEFT comparison/deployment with refused and uncertain/recovered cases; a passing native client probe is insufficient. M28e selects `cargo test --locked -p coh-rtc`, `cargo test --locked -p hive-gateway --test agent_protocol_controls --test a2a_protocol --test a2a_jobs`, the focused A2A task-state unit test and `scripts/check-generated.sh`. These checks prove selected routes, scoped card, malformed/authenticated traffic, bounded state mapping and disabled surfaces. `m28e-a2a-live` must then bind a named `a2a-sdk==0.3.26` peer, original CUDA/PEFT job and native result to exact source, QEMU image, Jetson provider and verifier evidence. Recover across a gateway restart under `ticket.id`, request cancellation and retain uncertain outcomes without a new effect. The native NeMo 1.9.0 `a2a_client` must read the card and use task lookup/cancel; its timeout is not a provider terminal. This is selected component qualification, not Pi or Release B acceptance. The `m28d-mcp-live` and `m28e-a2a-live` cases each require a named standard client to complete and recover selected real work independently. When MLX and CUDA are advertised together, each case includes a selected workflow using both native providers on their qualified host profiles. Record exact provider/host selection, bounded capacity and transfer decisions, original stage/job identities, native outcomes and shared-verifier result. Test wrong host, incompatible artifact, exhausted capacity/budget, unapproved transfer, lost response and reconnect without a new effect. The installed Python wheel must project the same shared operation through its CLI/REST path, including pending/refused/failed states; focused Python tests cannot replace live MCP, A2A, MLX or CUDA evidence. JetPack observations apply only to a host profile that actually uses JetPack; another NVIDIA platform needs its own qualification. The planned `m28b-verified-weight-distribution` extension is a prerequisite before either protocol advertises model-weight distribution. Its separate `m28b-weight-distribution-live` case needs an actual model weight of at least 1 GiB, frozen bake-off winner and source digest, admission before bytes move, observed host-to-host payload route and byte count, destination chunk checks, reassembly and independent full-file SHA-256. Record Pi/Queen reference bytes separately from payload bytes; publishing `coh-ref-c/v1` references through a real FUSE mount does not prove that the Pi carried the model. Interrupt the transfer and recover the same durable identity without duplicating verified chunks; reject wrong host/format, revoked export, insufficient space, corrupt chunk and cross-subject receipt access. The MCP-only and A2A-only live cases must each invoke and recover that accepted transfer independently if their catalogue advertises it. Check that MCP tool success, A2A task completion and Python status agree with the scoped destination receipt; none is a substitute for the native byte/digest proof or a separate serving activation canary. Before claiming vMLX compatibility, pin the installed app, model, MCP SDK, transport and peer versions. Run its configured MCP client against the real Cohesix server for discovery, admitted CUDA/PEFT work, refusal and recovery; record vMLX tool policy and the original shared operation identity. For A2A, use a named A2A-capable peer with vMLX as its selected model endpoint, then exercise delegation, interruption and reconnect against Cohesix. Record the model endpoint, A2A peer and GPU provider separately. A vMLX API response, model tool call or task state is not native CUDA/MLX execution or a verified PEFT release. If no vMLX model session or compatible peer exists, report the live slice blocked and leave generic MCP/A2A and NeMo evidence separate. For selected `m28f-nemo-install` and `m28f-nemo-live`, record the pinned NeMo Agent Toolkit version and exercise its native MCP transport/auth, selected tool discovery and call, plus its per-user A2A Agent Card, task lookup and cancellation helpers. The workflow must use an actual selected CUDA job and HF PEFT adapter/base/served generation, including a denied or failed candidate. Test two verified subjects, client timeout, disconnect and cross-protocol retry: the same underlying job remains recoverable, and one subject cannot read the other's scoped task or evidence. Toolkit user-ID resolution, tool output and agent traces do not supply gateway authentication or native provider proof. If a single Toolkit process is claimed to host both protected per-user clients, require a startup and authenticated call on each path for that exact version; otherwise report separate native workflows and the combined-mode limitation. M28g starts with the exact 1.2.0 archive, version-aligned Python/NeMo inputs and independently authenticated native installer artifacts. Qualify the Mac `.pkg` on clean supported Apple Silicon sessions and both ARM64 `.deb` packages on the selected JetPack 7.2.1 / L4T 39.2.1 reference. Retain its JetPack, L4T, Ubuntu base, CUDA toolkit and package-manager identities alongside the receipt, installed byte readback, publisher verification, graphical launch from each advertised desktop path, headless controller use, 1.1.0-beta migration, failed/repeated install, upgrade, rollback, remove and purge observations. Archive inspection, synthetic staging and a GUI screenshot alone do not close those cases. Run the published CUDA, PEFT, Apple and NeMo journeys from the installed candidate, with shared native identity and verifier outcome across CLI, Python, SwarmUI, MCP and A2A where advertised. Freeze adoption, quality, control-latency and recovery budgets before those runs. Preserve all four effective protocol modes and the disabled master override. Then apply the complete selected QEMU, physical Pi and conditional release gates at the assembled source/image identities. A missing installer, host profile, live result or a named clean-install walkthrough blocks M28g completion. The release owner may attest a personally completed full walkthrough in writing when raw timings, screenshots or notes were not retained. Record the exact scope and source/package identities, label it owner-attested, and report measurements as unrecorded. This accepts only the human adoption evaluation; it does not supply native job, protocol, target, installer receipt or quantitative budget proof. `release_qualify.py installer` checks independently selected publisher trust, package-manager receipt and installed byte readback on the current host; its PASS is one input to the Mac or Linux installer row, not the graphical launch, lifecycle, named evaluator walkthrough or assembled release verdict. | Contract at risk | Required kind of evidence | Closure owner | | --- | --- | --- | | Parsing, bounds, arithmetic, serialization, ABI/layout, policy, replay, pure state transitions | Small deterministic tests with independent expected truth | Stage 01 and AGENTS Test Discipline | | Source/generated/profile/image agreement | Generated consistency, exact target compilation, validated image and launch/readback identity | Target-entry integrity; Stages 01/02; hardware runbook | | seL4 capabilities, IPC/MCS, service/Worker startup, SMP, faults and recovery | Real READY and operation; bounded negative/recovery path; post-recovery state and liveness | QEMU/Pi target gates and 26e component/root/system evidence | | Physical ownership, MMIO, IRQ, DMA/cache, timers, USB/HDMI, GENET/CYW43 | Fresh exact-image Pi boot, real device operation and operator observation | Conditional F; independent performance and repeatability gates | | Console/TCP, Secure9P, REST and gateway | Exact protocol terminals, refusal/error paths, reconnect/close and bounded backpressure on a live target | Stages 03/04; Conditional A where selected | | Shipped host tools, Python, tickets, CAS, GPU/PEFT receipts and providers | Focused host contracts plus exact-version live integration for supported modes; real providers for provider claims | Conditional B/B1; generated host integration graph; Python projection | | SwarmUI | Deterministic presentation/replay plus separately identified packaged/live workflow checks | UI tier; host integration and release qualification | | Capacity, latency, coexistence, sustained load and delivery | Qualified workload, before/after state, pressure/repeatability, extracted bundle execution and governance | Conditional B2/D/E/F/G as applicable; Stage 05 | - Follow [AGENTS Test Discipline](../AGENTS.md#test-discipline): preserve useful pure-contract tests even when a target also exercises the path. Add a host regression after a target defect only when its cause is an independently understood deterministic invariant. Do not recreate the whole target in mocks. - Test the applicable boundary and refusal cases, not only success: malformed or oversized input, wrong/stale authority or generation, exhaustion, partial construction/publication, timeout/fault, cancellation and teardown. Require typed failures, bounded resource use, no unintended authority or side effects, and continued service where the contract requires recovery. Terminal containment must remain terminal; do not demand a nonexistent restart path. - The catalog owns command selection. A named regression identifies an obligation, not an additive command after its owning complete suite. Repeat a suite only for a distinct feature/profile, required fresh boot, workload, host platform, changed input, or explicitly required repeatability. Keep the broad Rust pre-merge/audit gates; this rule does not waive them. - Reuse only validator-approved immutable evidence under the resume contract. Shared source bytes do not make Mac/HVF and Linux/KVM builds, QEMU and Pi, Wi-Fi and GENET, or first-install and saved-policy boots interchangeable. Stage 05's current advisory checks still run on every acceptance refresh. - The owner-approved 1.0.0-beta publication mode in [HOST_TOOLS.md](HOST_TOOLS.md#release-factory) separates documentation and obsolete-distribution cleanup from an immutable tested source. The factory checks the complete allowlisted delta and preserves exact tested binaries, generated runtime contracts, source identities and passing records. Optional publication metadata identifies the later packaging commit. This is reuse for assembly only: no stage marker, target result or acceptance claim is rebound to that publication commit, and the mode runs no tests. Changes to runtime source, policy, manifests or evidence machinery still require their applicable qualification and cannot enter this mode. - The release owner's 2026-09-13 instruction to complete canonical release builds without tests admits `release_bundle.sh --build-only`. It assembles newly recorded `release.build-only` artifacts from one clean source; source, native profiles, image identity, compiler inventory and content checks remain mandatory. It accepts no transport result and records `NOT_RUN` in bundle provenance. Existing Stage 5 acceptance and original source/image identities remain separate. This assembly path grants no staged or target PASS. - DD30's recurring acceptance requirement was permanently retired by Lukas Bower on 2026-09-14. [DD30 retirement decision](audit/AUDIT_REPORT_2026-09-13.md#dd30-retirement) records the repaired historical finding as P1 / `RETIRED_ACCEPTED_GAP`, with dynamic fault/wake still UNEXECUTED. Stage 05 checks its retirement record without milestone/release selection, whole-file hashes or expiry. `DD_MILESTONE_ID` is obsolete. No other finding, target evidence or staged prerequisite is waived; a demonstrated IPC regression follows the ordinary finding lifecycle. - Technical Rust review remains required and may be agent-led, including independent review where required. Individual changes, commits, merges and component/milestone acceptance do not require human sign-off. Follow the [validation matrix and merge baseline](../CONTRIBUTING.md#5-validate-locally) and [Test Discipline](CODING_GUIDELINES.md#test-discipline); existing safety, scope, exception and evidence controls remain mandatory. - Only an overall assembled Cohesix release requires explicit approval by a named human release owner before publication or promotion. Bind approval to the exact source/artifacts, evidence, limitations and residual risks. Agent review or a passing gate cannot supply release approval. Runtime capability, deployment and physical-operation authority checks remain unchanged. - Historical [M27](audit/DD30_M27_APPROVAL.toml) and [M27a](audit/DD30_M27A_APPROVAL.toml) human approvals can be checked explicitly with `scripts/ci/due_diligence_gate.sh --check-rust-review 27|27a`. This binds only their original implementations and ignores historical DD30 expiry; it is not a recurring per-change or milestone gate. No new DD30 approval record is required. A finding preflight is neither code review nor release sign-off. - For this M27a completion run, the owner's later instruction is: “Only run tests required to mark this milestone ‘Complete’, not the full suite”. [M27A_COMPLETION_EVIDENCE.md](audit/M27A_COMPLETION_EVIDENCE.md) maps the milestone's individual definition-of-done checks to exact retained evidence and focused validation. Earlier complete suites retain their source identities; the interrupted candidate-F common run is not a staged PASS. This run does not claim a complete final-source five-stage chain or create replacement markers. Required M27a contracts, target observations, compatibility regression outputs, the pre-27a gateway benchmark comparison and security checks remain required. The separate 2026-09-14 [completion decision](audit/M27A_COMPLETION_EVIDENCE.md#completion-decision) accepts M26d without its missing status-baseline JSON and waives that comparison for M27a closure only. The comparison remains NOT_PERFORMED; this decision creates no performance-equivalence claim, target PASS or runner bypass. - The separate 2026-09-14 [M27 completion decision](audit/M27_COMPLETION_EVIDENCE.md) permits marking Milestone 27 Complete using the disclosed final-source QEMU Stages 01–05, live TCP/REST operator checks and Pi Stages 01–02. The final-image Pi boot, live operator checks and Stages 03–05 remain unexecuted. This is a one-time milestone-status approval, not Pi staged acceptance: it creates no PASS, marker, or source rebind, and adds no runner bypass. Future target claims retain the complete applicable staged requirements and exact provenance. - Review all affected host tools, `tools/cohesix-py`, generated contracts, benchmark workloads/report schemas, fixtures, and docs in the same change. Record reviewed surfaces requiring no change. Use [HOST_TOOLS.md](HOST_TOOLS.md) and the generated integration graph as the inventory, not a partial tool list. - When hardware, a provider, or a required host platform is unavailable, report BLOCKED or unexecuted for that claim. A mock, replay, help/version smoke, or another target cannot fill the gap. No unrelated test cleanup or test-count target is authorized by this plan. ## Development order: target first, acceptance complete During active target development, obtain real QEMU or Pi 4 evidence as early as safely possible. Use host/unit tests to preserve pure contracts and codify host-testable target-discovered invariants. Use the complete staged pipeline when producing acceptance evidence. This is a sequencing correction, not a reduction in test coverage, provenance, or release rigor. Two workflows are deliberately separate: - **Convergence evidence** is a development and diagnosis aid. It is emitted as `NON-CLAIMING TARGET DIAGNOSTIC`, uses the distinct `cohesix-test-plan-convergence/v1` schema, and cannot write Stage PASS attestations or Milestone 26e acceptance records. It does not require an earlier Stage 01, 02, 03, 04, or 05 attestation. - **Acceptance evidence** is the only basis for a milestone or release claim. The existing Stage 01-05 runner, target-specific attestations, immutable artifacts, pressure/repeatability evidence, due diligence, and conditional claim tiers remain authoritative and fail closed. A convergence PASS never satisfies, bypasses, or promotes an acceptance gate. Run the full acceptance workflow again against the exact final source, image, target, profile, and topology before making a claim. Milestone 26e `bi`, `caps mcs`, and `smp mcs` are convergence diagnostics. Parser/fixture and `mcs_operator_inspection` tests prove bounded rendering and snapshot behavior. The `caps mcs` records must preserve every field within the 77-byte Pi linked-HDMI fallback width. QEMU proves only QEMU; Pi observations require a fresh exact image. Neither diagnostic may call `seL4_SchedContext_Consumed`. ### Target-first convergence entry point Use the separate runner during active Milestone 26e work: ```bash # Explicit focus is preferred when a dirty tree spans several surfaces. scripts/ci/test_plan_converge.sh \ --target qemu \ --focus root-mcs \ --path apps/root-task/src/kernel.rs # Reuse a previously built immutable out/cohesix launch set. scripts/ci/test_plan_converge.sh \ --target qemu \ --focus worker \ --launch-existing # Let the declarative trigger paths choose the focus for a bounded change set. scripts/ci/test_plan_converge.sh \ --target qemu \ --changed-from origin/main ``` `--focus` accepts `root-mcs`, `ninedoor`, `console-network`, `worker`, `pi4-driver`, `live-transport`, `python-sdk`, `swarmui`, `test-plan-tooling`, or `docs`; use `--list-focus` for the generated inventory. Automatic selection fails rather than silently choosing QEMU for a Pi-first path or choosing a focus for an unmatched path. The selected actions come from `configs/test_plan_actions.toml`; there is no second command catalog. A selected focus proves only its changed path. For changes spanning independent surfaces, retain the full changed-path list and run the additional relevant focuses; an explicit `--focus` does not waive another surface's canary or final claim-tier requirements. Review the selection before spending a target boot. Every run creates a fresh directory below `out/test-plan-convergence/` and records the Git commit, complete dirty-tree/source digest, catalog digest, target, exact profile, focus, changed paths, selected action IDs, session/run ID, action logs and hashes, target observation, built-image and immutable identity hashes where applicable, UART/serial path and hash, result (`PASS`, `FAIL`, or `BLOCKED`), first failed proof layer, and optional `--hypothesis`/`--note`. Results are immutable candidate observations with `claiming=false` and `promotion_eligible=false`. ### Changed-path convergence routing The convergence focus and action metadata below are generated from the same catalog used by acceptance. The first authoritative evidence is selected by the most specific matching focus; broad host closure is not added before a target canary merely because it is part of final acceptance. | Focus | Target | First authoritative evidence | Exact profile | | --- | --- | --- | --- | | `pi4-driver` | pi4 | one exact-image Pi boot, touched service/device liveness, one live operation, UART liveness, and no unexpected target fault | `pi4_production / configs/root_task_pi4_uboot_aarch64.toml` | | `worker` | qemu | canonical QEMU boot, real Worker READY, and one bounded startup/teardown/restart recovery operation | `qemu_smp_production / configs/root_task.toml` | | `ninedoor` | qemu | canonical QEMU boot, isolated NineDoor READY, and one real 9P operation | `qemu_smp_production / configs/root_task.toml` | | `console-network` | qemu | canonical QEMU boot, isolated console READY v6, and the fixed one-socket HELP/NETSTATS/SMP/CACHELOG matrix | `qemu_smp_production / configs/root_task.toml` | | `root-mcs` | qemu | canonical QEMU boot, root steady state, one real target operation, and no unexpected seL4 fault | `qemu_smp_production / configs/root_task.toml` | | `live-transport` | qemu, pi4 | one authenticated operation over the changed live target transport | `selected target production profile` | | `python-sdk` | qemu, pi4 | focused Python SDK tests | `host-only Python SDK` | | `swarmui` | qemu, pi4 | focused SwarmUI package tests | `host-only SwarmUI` | | `test-plan-tooling` | qemu, pi4 | focused test-plan tooling tests and catalog/document consistency | `test-plan tooling` | | `docs` | qemu, pi4 | documentation metadata and generated-contract consistency | `generated documentation contracts` | Representative results are normative: - root task, MCS, IPC, capability, Worker, or service-isolation changes start with a QEMU target canary; - Pi boot, MMIO, DMA, IRQ, timer, cache, driver-runtime ABI, networking, or physical ownership changes start with a Pi 4 target canary; - TCP, REST, gateway, or cohsh changes perform one live operation against the selected real target; - Python SDK and SwarmUI changes run their focused host suite; and - documentation-only changes run documentation/generated consistency checks. ### Target-entry integrity versus broad host closure Stage 01 remains intact for compatibility and final acceptance, but its responsibilities have two different positions during development: **Target-entry integrity** is the minimum safe pre-target set: validate the generated contracts used by changed target code, validate the selected feature/profile, compile the exact target release configuration, run only cheap required ABI/layout checks, and optionally run one narrow test when it directly protects the target-entry contract. **Broad host closure** remains mandatory for acceptance but normally follows a successful target canary during active target work: workspace-wide tests and Clippy, complete root/runtime feature suites, SwarmUI, mock `coh`, Python SDK and examples, unrelated drivers, broad regressions, and dependency/risk/ governance closure. A host-only focus may run its focused host test first because the host surface itself is the changed execution path. ### QEMU convergence proof order For root-task behavior, MCS, isolation, Workers, capability/IPC/scheduling, fault handling, SMP, image construction, or startup, run only: 1. generated contracts and selected configuration required by the change; 2. exact `qemu_smp_production` target compilation; 3. genuinely required cheap ABI/layout/static checks; 4. the immutable canonical Milestone 26e QEMU image boot; 5. `Cohesix console ready` root steady state; 6. the changed service READY marker, or real `WORKER_TASK_READY` for a Worker; 7. one real operation through the changed target path; 8. absence of unexpected seL4 faults, capability errors, scheduler failures, timeouts, or runtime panics; 9. one bounded budget/timeout/fault-recovery operation when the change affects scheduling or recovery, selected with `--operation-script` when the default probe is not the changed recovery path; then 10. the smallest appropriate regression guard; use a host guard only for a useful deterministic host-testable invariant. Stop at the first failed layer. Do not run broad workspace, UI, mock-client, Python SDK, unrelated driver, or general regression suites before this canary unless one of those surfaces is itself the selected focus. `--launch-existing` validates and launches the bound `cohesix-qemu-launch-artifacts.json`; it never restages or silently rebuilds an immutable diagnostic artifact. The convergence runner emits `cohesix-target-observation/v2`. In addition to the UART and QEMU command records, it binds `operation_log` to the exact authenticated `cohsh` transcript. QEMU evidence consumers require that transcript to contain remote NineDoor readiness, successful authentication and Queen attachment, and one successful `CAT`; a boot-only UART marker cannot stand in for the operation. The Worker restart canary publishes a distinct single-use policy approval before each create/kill/create write. Each asynchronous lifecycle step uses the existing bounded `WAIT 2000 TAIL` contract to observe the expected READY record in the Queen log or terminal telemetry before the next step. Read a new Worker's telemetry only after READY publishes its namespace. Two completed authenticated Queen-log exports retain the actual `WORKER_TASK_READY` generations and the intervening complete `WORKER_TASK_TEARDOWN`. The validator requires the same slot, increasing lease/supervisor/capability generations, and every terminal containment field; UART mirroring and admission acknowledgements cannot substitute for these records. The observation binds `worker_restart_evidence` to this parsed proof and still requires successful `cohsh` completion, including response framing and QUIT/EOF. The QEMU proof ladder is: ```text source/config identity -> exact target build -> image validity -> target boot -> root steady state -> changed service/Worker READY -> one real operation -> changed failure/recovery path -> focused regression guard -> broader integration regressions -> pressure/repeatability -> final acceptance ``` ### Pi 4 convergence checkpoints and proof order Use Pi early when QEMU cannot authoritatively model firmware/U-Boot, physical MMIO/IRQ/timers, DMA/cache coherency, the driver-runtime ABI, physical device ownership/networking/concurrency, or shared root capability construction that may differ on hardware. The convergence runner does not discover or overwrite an SD device and never fabricates physical evidence. Prepare and independently preserve the exact readback and live boot record, then provide them explicitly: ```bash scripts/ci/test_plan_converge.sh \ --target pi4 \ --focus pi4-driver \ --pi4-target-evidence out//target-evidence.json \ --pi4-readback-image out//readback.img \ --pi4-identity-metadata out//readback.img.identity.json \ --pi4-serial-log /absolute/path/to/current-nonempty-uart.log \ --pi4-host ``` The first Pi diagnostic proves only: exact source/image identity; flash/readback identity; one real boot; root and touched service/device readiness; one real operation through the selected path; UART liveness across that operation; and no unexpected fault in the bound boot. Missing hardware inputs produce `BLOCKED`, never synthetic PASS. Full cold/warm repeatability, pressure, TCP matrices, RF claims, benchmark, and hardware qualification remain later acceptance activities. Milestone 26e requires lightweight Pi checkpoints: 1. after the first complete MCS root boot is stable under QEMU; 2. after isolated critical services work under QEMU; 3. after Worker loading and fault recovery work; 4. before resource/capability/topology ABI assumptions are frozen; and 5. before Milestone 26e acceptance. Each checkpoint is a new image/source-bound Pi observation, not permission to reuse old hardware proof. The first four may use the lightweight convergence lane. The fifth must be followed by the complete required Pi qualification and acceptance evidence. Follow the canonical build -> flash -> readback -> boot -> saved boot/profile policy -> device/network proof -> console/liveness proof -> target-qualified Test Plan -> benchmark/repeatability ladder in [HARDWARE_BRINGUP.md](HARDWARE_BRINGUP.md); this document does not redefine it. ### Rabbit-hole prevention rules These rules are normative during target convergence: 1. No more than two speculative target-code edits may occur without rerunning the relevant QEMU or Pi diagnostic. 2. A target failure overrides host PASS results when diagnosing target behavior. 3. Do not add broad tests while the target remains red unless a new test distinguishes one specific observed target hypothesis. 4. Stop at the first failed proof layer. Do not debug TCP while boot, image identity, capability construction, service readiness, IRQ delivery, or an earlier layer remains unresolved. 5. Do not optimize or broadly refactor a failing path before its target failure mechanism is understood. 6. Every target fix must identify the observed target failure, hypothesis, code change, and target observation that proves or disproves the hypothesis. 7. Once the target fix is proved, preserve its smallest independently understood invariant with an appropriate regression guard or target evidence. A new host test is not mandatory for every target defect. 8. Unit tests are not authoritative evidence for live scheduling, capability installation, real IPC, IRQ delivery, DMA/cache correctness, or physical device behavior. ### Candidate collection, validation, and acceptance promotion Target observations may be collected early as candidate convergence evidence and validated for schema, source, image, profile, target, action-log, and UART integrity. They are never promoted in place. Acceptance evidence is created only by a new complete staged run after all required stages pass and current source, image, target/profile/topology, pressure, repeatability, and hardware identities still match. A stale convergence result, even a PASS, cannot become accepted Milestone 26e evidence. ## Mandatory Acceptance Execution Contract This contract is normative whenever a milestone, release, or claim-tier result is being produced. It is unchanged by the development convergence lane. 1. Run `scripts/ci/test_plan_run.sh --list`, then use the staged runner with a dedicated state directory. The runner resumes digest-valid evidence by default: - `scripts/ci/test_plan_run.sh --target qemu --state-dir out/test-plan/` - `scripts/ci/test_plan_run.sh --target pi4 --state-dir out/test-plan/` 2. Use `--force` to replace an active stage result after a failure or input change. The old immutable attempt remains under `evidence/`; a failed, interrupted, or INCOMPLETE rerun cannot leave a reusable PASS marker. 3. Use `--stage --iteration` only for focused debugging. Iterations have a separate evidence namespace and never write, remove, or refresh full-pass attestations. 4. Use `--reuse-common-from ` only for source-, configuration-, toolchain-, catalog-, and artifact-identical common evidence. Target-specific actions are never relabelled across QEMU and Pi 4. Missing or legacy provenance fails closed. 5. Fix a failed stage before progressing. Skips write INCOMPLETE evidence and fail; a platform check may be `NA` only where the catalog explicitly permits it. 6. Keep `configs/test_plan_actions.toml`, this document, and the scripts aligned. `scripts/ci/check_test_plan.sh` must pass. 7. Before making a claim for a change set, select its conditional tiers from the catalog without losing paths that contain whitespace: `git diff --name-only -z -- | python3 scripts/ci/test_plan_catalog.py recommend --stdin0 --format tiers`. Preserve the reported action IDs with the run evidence. An unmatched path selects every catalog action, and a conditional tier cannot be reported unless its named evidence action has also passed. 8. Keep the developer host responsive. The runner defaults build, libtest, and Rayon concurrency to half the detected logical CPUs (maximum six); on the 10-core macOS development host this is five jobs. Playwright defaults to two workers. Set `TP_HOST_JOBS=` or `TP_UI_WORKERS=` to lower the cap. Oversubscription requires the explicit `TP_ALLOW_OVERSUBSCRIBE=1` opt-in. The Pi image builder consumes the same budget when invoked from a staged or conditional test-plan run. The five staged entrypoints remain: - Stage 01: `scripts/ci/test_plan_stage_01_integrity.sh` - Stage 02: `scripts/ci/test_plan_stage_02_host_fast.sh` - Stage 03: `scripts/ci/test_plan_stage_03_qemu_tcp_regression.sh` - Stage 04: `scripts/ci/test_plan_stage_04_rest_multiplexer.sh` - Stage 05: `scripts/ci/test_plan_stage_05_due_diligence.sh` ### Milestone 26e executable enforcement Requirements must have an executable owner, not a second checklist document. The stage scripts consume the catalog; the milestone runner calls the strict evidence validators after all five stage attestations verify. | Boundary | Executable enforcement | | --- | --- | | Deterministic contracts and feature coverage | Stage 01 executes the catalog's complete host suites and Python discovery, including compiler admission, ABI/layout, service/Worker/driver contracts, and the distinct `host.console-network-direct-genet` suite. `check_driver_test_coverage.py` checks the declared driver-suite mapping. | | Exact target/profile and image | Stage 02 validates the selected seL4 profile and target build; Stage 03's `qemu_artifact.py` validators bind launch/image/source bytes or the supplied Pi transport identity. Physical readback remains a separate hardware gate. | | Live protocol behavior | Stage 03 runs and validates the complete selected TCP matrix; Stage 04 runs the live REST batches and Python smoke on the bound target. Neither substitutes for fault, pressure, or hardware evidence. | | Worker, critical-TCB and full-system outcomes | `test_plan_run.sh --m26e-evidence-kind component\|root\|system` invokes `worker_task_evidence.py` with explicit inputs. Its validators require the exact outcome matrices, generated topology, role integrations, raw-evidence hashes, and target/session bindings. Missing or partial input prevents that acceptance record. | | Physical proof and performance | Conditional F uses `pi4_gate_proof.sh`, `pi4_trace_normalize.py`, image-identity and repeatability validators over real captures. Conditional B2 and the qualified Pi lane use the pressure collectors, `rest_perf_harness.py`, and target comparator. These are additional claim gates, not automatic work performed by an ordinary five-stage run. | | Governance and release promotion | Stage 05 revalidates prior attestations and current due diligence. `worker_task_evidence.py promote-release` validates all six QEMU/Pi component/root/system records; `release_bundle.sh` and `release_qualify.py` separately validate the required projections and delivered artifacts. | Ordinary Stage 01-05 PASS is not Milestone 26e PASS. Every applicable conditional, component, root, system, Python, hardware, pressure/repeatability, and release gate must also be complete. A script validates supplied physical observations; it cannot manufacture a Pi boot, keyboard input, HDMI observation, or provider result. Keep those collection procedures and pass criteria here and in HARDWARE_BRINGUP, not in an auxiliary test inventory. When a scoped requirement lacks a test, validator, or required physical observation, record the enforcement gap under its BUILD_PLAN task and block the affected acceptance claim until it is closed. Do not label a prose rule, unexecuted action, diagnostic PASS, or historical candidate record as enforced acceptance. Add a missing check to its owning suite/catalog or validator, with a focused regression for the observed gap; do not duplicate the whole stage. Milestone 26e host-integration inventory and evidence use the generated [`host-integration-dependency/v1`](../configs/generated/host_integration_dependency.json) graph. `scripts/ci/check_host_integration_inventory.py` checks exhaustive host surface, six-scenario, and nine-playbook coverage. The bounded runner is: ```bash scripts/ci/host_integration_run.sh \ --matrix configs/host_integration_acceptance.toml \ --matrix-only \ --state-dir out/host-integration/m26e-matrix ``` Target-session lanes accept `qemu` or `pi4` only, reject stale or wrong-target identity, and emit one `cohesix-worker-integration-evidence/v1` record per dependency row. The mandatory target rows are exactly `worker-control`, `gpu-receipt-path`, and `peft-receipt-path`; external provider observations remain separate and cannot be promoted by receipt fixtures. The target runner consumes target proof and row observations as separate, caller-produced records; launching QEMU does not synthesize either record: ```bash python3 scripts/worker_task_evidence.py emit-qemu-target-session \ --repo-root . \ --qemu-out out/cohesix \ --resolved-manifest configs/generated/root_task_resolved.json \ --topology configs/generated/root_task_topology.json \ --out-dir out//session ``` The standalone emitter takes no digest arguments. It verifies the immutable QEMU launch record and launch bytes, exact Worker and driver archive/manifests, stable git-visible source bytes, Worker ABI sources, and generated-topology / resolved-manifest parity before creating a new output directory. It refuses aliases, drift, malformed archives, unignored in-repository output, and any existing output directory. On PASS it atomically publishes exactly `source-inventory.json`, `worker-abi-identity.json`, `qemu-cyw43-coexistence.json`, and `target-session.json`; none is direct target execution or acceptance evidence. ```bash scripts/ci/host_integration_run.sh \ --matrix configs/host_integration_acceptance.toml \ --mode live \ --target qemu \ --target-session out//session/target-session.json \ --observations out//host-integration-observations.json \ --state-dir out/host-integration/ ``` `target-session.json` is an exact object containing `target` plus lowercase SHA-256 fields `source_sha256`, `manifest_sha256`, `kernel_sha256`, `root_image_sha256`, `driver_archive_sha256`, `driver_manifest_sha256`, `cyw43_coexistence_record_sha256`, `worker_archive_sha256`, `worker_image_manifest_sha256`, and `worker_abi_sha256`. The observations record uses `cohesix-host-integration-observations/v1`, binds the exact dependency-graph and resolved-manifest hashes, and supplies one mode, bounded outcome list, and sorted raw-evidence list for every selected row. Capability material and secrets are rejected; output state directories must be empty. ### Milestone 26e direct target acceptance evidence Worker-component and root-TCB acceptance are separate from the normal staged claim tiers. A QEMU launch, reachable gateway, stage marker, integration mock, or packaged image never creates either record. The staged runner emits a 26e record only after Stages 01-05 have independently passed and the caller supplies the complete direct-observation inputs through `--m26e-evidence-kind`. For clean live-QEMU acceptance builds, use a dedicated disposable checkout with fresh `target/` and `out/` directories, then build through the canonical GICv3 script. Keep retained evidence outside that checkout. Never delete the working repository's `out/`, logs, or immutable attempts to obtain a clean run. The build and artifact validators must reject stale ELF/archive/manifest input: ```bash scripts/cohesix-build-run.sh \ --clean \ --cargo-target aarch64-unknown-none \ --no-run out/toolchain/arm-gnu-toolchain-15.2.rel1-darwin-arm64-aarch64-none-elf/bin/aarch64-none-elf-gdb \ --version ``` The script rejects any machine/GIC override and builds QEMU evidence symbols only for the selected `release-qemu,bootstrap-trace` profile. Keep the unstripped ELFs under `target/aarch64-unknown-none/release/`, the Worker archive and manifest under `out/cohesix/worker-images/`, and the canonical external driver archive at `out/cohesix/driver-runtimes/cohesix-driver-runtimes.cpio`. The driver archive is byte-verified inside rootserver but is intentionally not duplicated in the system CPIO. Rootserver also retains exactly one byte-identical Worker archive and manifest for target loading; their system-CPIO copies remain the host and release projection. A build fails if either embedded payload differs from the validated source or target code treats typed BootInfo FDT bytes as a CPIO. Capture each boot through macOS `script`. The ordinary preflight boot remains running while the operator drives Worker turns from another terminal; terminal service injection uses the separate fresh boots defined below: ```bash RUN=out/m26e-qemu mkdir -p "$RUN/preflight" script -q "$RUN/preflight/uart.log" \ scripts/cohesix-build-run.sh \ --cargo-target aarch64-unknown-none \ --raw-qemu \ -- \ -pidfile "$RUN/preflight/qemu.pid" \ -gdb tcp:127.0.0.1:1234 ``` Run `qemu-gdb` once for each role, preserving argument order. Each invocation stays attached for three generations: the operator spawns the role for the pre-READY fault, recreates it and submits an approved `kill` for the during-IPC standard fault, then repeats that lifecycle call for MCS budget exhaustion before creating a final READY instance. The instrumented pre-READY injection redirects the selected Worker's PC to unmapped address zero. The target must report a role-matched Standard fault and terminal teardown; a GDB injection marker or READY timeout cannot pass. The later during-IPC injection retains its target evidence fault symbol. The instrumented `cohesix_worker_qemu_evidence_call_dispatch` hook runs after validating the received call and before its dispatch or reply. All three passive roles can therefore exercise the real shutdown IPC path; Heartbeat needs no autonomous publish turn. The runner reads the exact READY identity through the Python SDK over direct TCP and publishes fresh GPU fixture inventory before each GPU spawn. These direct sessions close before the gateway first attaches. The separate seven-action matrix continues to use real v2 host tickets and receipts. ```bash GDB=out/toolchain/arm-gnu-toolchain-15.2.rel1-darwin-arm64-aarch64-none-elf/bin/aarch64-none-elf-gdb COMMON_GDB="--gdb $GDB --remote 127.0.0.1:1234 --target-session $RUN/session/target-session.json --generated-inventory configs/generated/root_task_topology.json --worker-image-manifest out/cohesix/worker-images/cohesix-worker-image-manifest.json" python3 scripts/worker_task_evidence.py qemu-gdb $COMMON_GDB \ --worker-elf worker-heartbeat=target/aarch64-unknown-none/release/worker-heart \ --worker-elf worker-gpu=target/aarch64-unknown-none/release/worker-gpu \ --worker-elf worker-lora=target/aarch64-unknown-none/release/worker-lora \ --inject-role worker-heartbeat \ --out "$RUN/preflight/gdb-worker-heartbeat.log" python3 scripts/worker_task_evidence.py qemu-gdb $COMMON_GDB \ --worker-elf worker-heartbeat=target/aarch64-unknown-none/release/worker-heart \ --worker-elf worker-gpu=target/aarch64-unknown-none/release/worker-gpu \ --worker-elf worker-lora=target/aarch64-unknown-none/release/worker-lora \ --inject-role worker-gpu \ --out "$RUN/preflight/gdb-worker-gpu.log" python3 scripts/worker_task_evidence.py qemu-gdb $COMMON_GDB \ --worker-elf worker-heartbeat=target/aarch64-unknown-none/release/worker-heart \ --worker-elf worker-gpu=target/aarch64-unknown-none/release/worker-gpu \ --worker-elf worker-lora=target/aarch64-unknown-none/release/worker-lora \ --inject-role worker-lora \ --out "$RUN/preflight/gdb-worker-lora.log" COH_AUTH_TOKEN="$QUEEN_TOKEN" \ TEST_PLAN_CONVERGENCE_QEMU_OUT_DIR=out/cohesix \ python3 scripts/ci/test_plan_converge.py \ --target qemu --focus ninedoor --launch-existing \ --state-dir "$RUN/authenticated-ninedoor" SERVICE_GDB="--gdb $GDB --remote 127.0.0.1:1234 --target-session $RUN/session/target-session.json --generated-inventory configs/generated/root_task_topology.json --qemu-out out/cohesix --auth-observation $RUN/authenticated-ninedoor/target-observation.json" python3 scripts/worker_task_evidence.py qemu-service-gdb $SERVICE_GDB \ --service ninedoor-service --mode during-call-standard \ --service-elf target/aarch64-unknown-none/release/nine-door-runtime \ --out "$RUN/ninedoor-during-call/service.gdb.log" python3 scripts/worker_task_evidence.py qemu-service-gdb $SERVICE_GDB \ --service ninedoor-service --mode between-calls-revoke \ --service-elf target/aarch64-unknown-none/release/nine-door-runtime \ --root-elf target/aarch64-unknown-none/release/root-task \ --out "$RUN/ninedoor-between-calls/service.gdb.log" python3 scripts/worker_task_evidence.py qemu-service-gdb $SERVICE_GDB \ --service console-network --mode during-call-standard \ --service-elf target/aarch64-unknown-none/release/console-network-runtime \ --out "$RUN/console-standard-fault/service.gdb.log" ``` Each NineDoor command above attaches to its own fresh exact-artifact boot; its matching `service.uart.log` is frozen after terminal teardown. The first is triggered by one authenticated ordinary Secure9P Call. The between-Calls probe resolves root-local evidence hooks by their exact defined, demangled Rust symbols; those hooks remain deliberately non-exported and the collector must not require a global control symbol. It counts two successful root post-prepare returns and requests local revoke between them and the next Call. Console-network Standard injection uses its own fresh exact-artifact boot because its containment is terminal and has no same-boot replacement. It is triggered by one authenticated control turn; the runner neither waits for reconstruction nor attempts a second child handler after teardown. Natural-postpone budget liveness is exercised under the retained pressure boots rather than by the obsolete terminal timeout-spin injection. No VM command or namespace fault-injection authority exists. The four critical-duty observation hooks occur during startup, so collect them on a separate halted boot of the exact same image/session. The collector first continues to a fifth, post-SMP arm hook immediately before any restricted TCB resumes, then replaces that breakpoint with the four duty breakpoints. This post-secondary-core re-arm is required because accelerator hardware-debug state can be reset while seL4 initializes secondary cores. It changes no guest scheduling, budget, capability, or service behavior: The five `release-qemu,bootstrap-trace` hooks carry distinct opaque identity tags so release linking cannot fold separate duty addresses together. They perform no I/O, scheduling, or authority change; the collector rejects a missing or aliased address before it attaches GDB. ```bash mkdir -p "$RUN/critical" script -q "$RUN/critical/uart.log" \ scripts/cohesix-build-run.sh \ --cargo-target aarch64-unknown-none \ --raw-qemu \ -- \ -pidfile "$RUN/critical/qemu.pid" \ -gdb tcp:127.0.0.1:1234 \ -S python3 scripts/worker_task_evidence.py qemu-critical-gdb \ --gdb "$GDB" --remote 127.0.0.1:1234 \ --target-session "$RUN/session/target-session.json" \ --generated-inventory configs/generated/root_task_topology.json \ --root-elf target/aarch64-unknown-none/release/root-task \ --out "$RUN/preflight/gdb-critical-duties.log" ``` After the same-boot integration records and the exact 7-by-3 receipt matrix are present, derive the component needed by the gateway before pressure. The collector treats cohsh `OK SPAWN`/`OK KILL` only as admission outcomes; READY, artifact, receipt, and proof axes come from identity-bound UART/pressure records, never from caller-supplied projection text. Use eight execution lanes for both receipt preflight and pressure. Preserve the completed `host-ticket-agent/` state, then copy its cursors, execution journals and lock files into `pressure-host-ticket-agent/` before pressure starts. Pressure resumes those cursors and journals with a separate log; starting from empty state would replay the deliberately retired tickets. Execution lane topology is part of journal identity and remains unchanged. ```bash python3 scripts/worker_task_evidence.py collect-qemu-preflight \ --target-session "$RUN/session/target-session.json" \ --generated-inventory configs/generated/root_task_topology.json \ --qemu-out out/cohesix \ --auth-observation "$RUN/authenticated-ninedoor/target-observation.json" \ --uart "$RUN/preflight/uart.log" \ --cohsh "$RUN/preflight/cohsh.log" \ --gdb-log "$RUN/preflight/gdb-worker-heartbeat.log" \ --gdb-log "$RUN/preflight/gdb-worker-gpu.log" \ --gdb-log "$RUN/preflight/gdb-worker-lora.log" \ --service-gdb-log "$RUN/ninedoor-during-call/service.gdb.log" \ --service-gdb-log "$RUN/ninedoor-between-calls/service.gdb.log" \ --service-gdb-log "$RUN/console-standard-fault/service.gdb.log" \ --service-uart "$RUN/ninedoor-during-call/service.uart.log" \ --service-uart "$RUN/ninedoor-between-calls/service.uart.log" \ --service-uart "$RUN/console-standard-fault/service.uart.log" \ --critical-gdb-log "$RUN/preflight/gdb-critical-duties.log" \ --worker-archive out/cohesix/worker-images/cohesix-worker-images.cpio \ --driver-archive out/cohesix/driver-runtimes/cohesix-driver-runtimes.cpio \ --worker-image-manifest out/cohesix/worker-images/cohesix-worker-image-manifest.json \ --worker-elf worker-heartbeat=target/aarch64-unknown-none/release/worker-heart \ --worker-elf worker-gpu=target/aarch64-unknown-none/release/worker-gpu \ --worker-elf worker-lora=target/aarch64-unknown-none/release/worker-lora \ --service-elf ninedoor-service=target/aarch64-unknown-none/release/nine-door-runtime \ --service-elf console-network=target/aarch64-unknown-none/release/console-network-runtime \ --root-elf target/aarch64-unknown-none/release/root-task \ --integration-dir "$RUN/integration" \ --out-dir "$RUN/preflight-component" ``` After separate fresh medium- and high-pressure boots, freeze each boot-local UART/GDB pair before writing its summary and run the final semantic collector: ```bash python3 scripts/worker_task_evidence.py collect-qemu \ --target-session "$RUN/session/target-session.json" \ --generated-inventory configs/generated/root_task_topology.json \ --qemu-out out/cohesix \ --auth-observation "$RUN/authenticated-ninedoor/target-observation.json" \ --preflight-uart "$RUN/preflight/uart.log" \ --preflight-gdb-log "$RUN/preflight/gdb-worker-heartbeat.log" \ --preflight-gdb-log "$RUN/preflight/gdb-worker-gpu.log" \ --preflight-gdb-log "$RUN/preflight/gdb-worker-lora.log" \ --preflight-service-gdb-log "$RUN/ninedoor-during-call/service.gdb.log" \ --preflight-service-gdb-log "$RUN/ninedoor-between-calls/service.gdb.log" \ --preflight-service-gdb-log "$RUN/console-standard-fault/service.gdb.log" \ --preflight-service-uart "$RUN/ninedoor-during-call/service.uart.log" \ --preflight-service-uart "$RUN/ninedoor-between-calls/service.uart.log" \ --preflight-service-uart "$RUN/console-standard-fault/service.uart.log" \ --preflight-critical-gdb-log "$RUN/preflight/gdb-critical-duties.log" \ --uart "$RUN/medium/uart.log" --gdb-log "$RUN/medium/gdb.log" \ --pressure "$RUN/medium/pressure.summary.json" \ --uart "$RUN/high/uart.log" --gdb-log "$RUN/high/gdb.log" \ --pressure "$RUN/high/pressure.summary.json" \ --cohsh "$RUN/cohsh.log" \ --worker-archive out/cohesix/worker-images/cohesix-worker-images.cpio \ --driver-archive out/cohesix/driver-runtimes/cohesix-driver-runtimes.cpio \ --worker-image-manifest out/cohesix/worker-images/cohesix-worker-image-manifest.json \ --worker-elf worker-heartbeat=target/aarch64-unknown-none/release/worker-heart \ --worker-elf worker-gpu=target/aarch64-unknown-none/release/worker-gpu \ --worker-elf worker-lora=target/aarch64-unknown-none/release/worker-lora \ --service-elf ninedoor-service=target/aarch64-unknown-none/release/nine-door-runtime \ --service-elf console-network=target/aarch64-unknown-none/release/console-network-runtime \ --root-elf target/aarch64-unknown-none/release/root-task \ --integration-dir "$RUN/integration" \ --run-dir "$RUN/test-plan" \ --out-dir "$RUN/accepted" ``` The QEMU-only GPU bridge snapshot remains `source=fixture`, `mode=fixture`, `profile=qemu`, `gate=bootstrap-trace`; it never becomes provider-live or production evidence. That same admitted snapshot projects exactly one read-only LoRA export job, `qemu-evidence-job`, containing only `telemetry.cbor`, `base_model.ref`, and `policy.toml`. Both fixtures disappear outside the explicit QEMU evidence gate. Publish it only through the existing bridge path: ```bash cargo run -p gpu-bridge-host --features rest -- \ --mock \ --publish \ --rest-url "$COHESIX_GATEWAY_URL" \ --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN" ``` The collector rejects missing, reordered, changed, symlinked, target-mismatched, or secret-bearing inputs and does not treat reachability or build markers as live proof. The QEMU Worker-component invocation is: ```bash scripts/ci/test_plan_run.sh \ --target qemu \ --state-dir out/test-plan/m26e-worker-qemu \ --m26e-evidence-kind component \ --m26e-target-session out/m26e-qemu/target-session.json \ --m26e-generated-inventory configs/generated/root_task_topology.json \ --m26e-observations out/m26e-qemu/worker-component-observations.json \ --m26e-integration-dir out/host-integration/m26e-qemu/integration ``` `target-session.json` uses the exact hash object documented above. The observations file uses `cohesix-worker-component-observations/v1` and contains exactly `schema`, `target`, `target_session_sha256`, `workers`, `outcomes`, `raw_evidence`, `verdict`, and `blockers`. Each of the three Worker rows is one bounded live Heartbeat/GPU/LoRA exemplar and records its five-part identity, state axes, image hash, READY/completion sequences, distinct endpoint/fault badges, core, exact passive scheduling-context values `0/0`, and full per-slot compiler-admission object inventory: TCB, CNode, VSpace, page table, ASID, frame, endpoint, notification, standard/timeout fault cap, Reply, scheduling context, CSpace slot, and untyped-byte totals. The emitter recomputes the compiler topology digest and derives the maximum-role inventory from the topology payload. It then requires each observed role's attach badge, fault badge, core, passive scheduling context, per-slot object inventory, allowlisted active executor donor, and generation-scoped Reply path to equal that generated truth. A fresh passive Heartbeat may have completion sequence zero: it has committed READY but has received no workload Call. The collector requires no uncompleted control/lifecycle Call for that live identity; the separate Heartbeat fault, shutdown, teardown and recreation proofs remain mandatory. GPU and LoRA require a positive completion sequence and a confirmed receipt. Separately, the topology digest seals all 256 Worker task rows and every generated non-Worker row. `temporal_authority.tasks` may exceed the generic 128-item evidence-list bound only when its exact length, order, identifiers, kinds, role classes, driver images, and fault-registry counts are derived from that compiler topology; every other arbitrary list retains the generic bound. A component PASS also requires the complete, sorted 26e event matrix and the exact live `gpu-receipt-path`, `peft-receipt-path`, and `worker-control` integration records. The emitter binds the observation, generated-topology, and target-session input bytes into the resulting raw-evidence graph and writes `worker-task-evidence.json` atomically only after validation. It also preserves the exact three referenced integration-record bytes under the output state directory's `integration/` subtree so downstream recursive validation does not depend on an unrecorded external directory. The QEMU root-TCB invocation is: ```bash scripts/ci/test_plan_run.sh \ --target qemu \ --state-dir out/test-plan/m26e-root-tcb-qemu \ --m26e-evidence-kind root \ --m26e-target-session out/m26e-qemu/target-session.json \ --m26e-worker out/test-plan/m26e-worker-qemu/worker-task-evidence.json \ --m26e-generated-inventory configs/generated/root_task_topology.json \ --m26e-observations out/m26e-qemu/root-tcb-observations.json ``` `coh-rtc` emits `configs/generated/root_task_topology.json` beside the selected resolved manifest. Its `cohesix-root-tcb-generated-inventory/v1` envelope has exactly `schema`, `profile`, `manifest_sha256`, `topology_sha256`, `topology`, and `inventory`. `topology_sha256` is the SHA-256 of deterministic compact JSON for the compiler-owned profile, root/driver topology, Worker runtime, temporal authority, resource admission, NineDoor service, and console-network service. The inventory is the admitted maximum: the exact fixed-object budget plus every executable slot in the one compiler-validated maximum role mix. It is not a kernel allocation or retype census. `scripts/check-generated.sh` compares the canonical QEMU output byte-for-byte. The direct input uses `cohesix-root-tcb-observations/v1` with the target-session digest and same topology, `inventory_scope=admitted-maximum`, the UART-projected admitted maximum, complete containment and operator-liveness outcomes, raw-artifact descriptors, and verdict. A PASS is impossible when the generated topology hash cannot be recomputed, its inventory cannot be derived, generated and projected admitted-maximum inventories differ, or the accepted Worker record names a different target session or topology. The resulting `root-tcb-acceptance.json` binds all three input files by digest. The strict release inventory packages this topology beside the resolved manifest; `scripts/release_bundle.sh --check-manifest --pi4-stage-dir ` rejects either an omission or an unexpected compiler-owned generated file rather than silently shipping a partial as-built contract. `ROOT_CRITICAL_OBJECTS scope=constructed-actual` separately records seven constructed critical TCBs: the five root duties plus the two active Worker executor lanes. Six are restricted children. Its active SC/Reply counts and minted standard/timeout fault-cap counts form the bounded actual critical- domain census and remain distinct from the complete generated fault-registry capacity; neither is inferred from the admitted maximum. Timeout caps remain minted and registered under NaturalPostpone even though the TCB has no installed timeout handler; this cap census is not a handler-installation count. Full-system evidence remains a verification-only layer over immutable accepted component/root records. Its explicit `cohesix-mcs-smp-run-input/v1` observation contains the target session, exact component/root digests, topology, four-core admission rows, the complete timeout/fault/Reply/liveness/performance outcome matrix, and raw evidence. It can be emitted after a full staged run with: ```bash scripts/ci/test_plan_run.sh \ --target qemu \ --state-dir out/test-plan/m26e-mcs-smp-qemu \ --m26e-evidence-kind system \ --m26e-worker out/test-plan/m26e-worker-qemu/worker-task-evidence.json \ --m26e-root out/test-plan/m26e-root-tcb-qemu/root-tcb-acceptance.json \ --m26e-observations out/m26e-qemu/mcs-smp-system-input.json ``` Missing, empty, wrong-target, stale, non-live, hash-mismatched, partial, or secret-bearing input fails before an acceptance record is published. Omitting `--m26e-evidence-kind` runs the ordinary staged plan but emits no 26e acceptance PASS. The same commands accept `--target pi4` only with independent fresh-Pi target-session, observations, integration records, and raw artifacts; QEMU files cannot be relabelled. Runtime release promotion still requires all six validated QEMU/Pi component, root, and full-system records, so QEMU-first work cannot produce Pi or Worker-runtime release acceptance. The controlled Milestone 26e refresh of tracked `seL4/build_UBOOT` is complete: it is a source-bound `pi4_production` SMP+MCS artifact set, and validation requires its exact contract hash, generated configuration, 4-node/16-bit-root- CNode profile, 54 MHz timer provenance, required artifacts, and complete tracked tree. Static profile validation, deterministic host composition, and a stage-only image build prove only their stated build and packaging contracts; none is Pi boot, hardware-driver, network, performance, repeatability, or acceptance evidence. Independent exact-image, fresh-target, pressure, and hardware gates must still pass before any Pi acceptance command above may pass. ### Milestone 26e Python package and target projection Build one target-neutral wheel, inspect its exact module and extras manifest, and install it without dependency resolution into isolated CPython 3.11 and 3.13 environments: ```bash python3 -m pip wheel --no-deps \ --wheel-dir out/python-wheels tools/cohesix-py scripts/ci/python_compat_run.sh \ --wheel-smoke \ --wheel-dir out/python-wheels \ --package-manifest out/python-compat/m26e-python-package.json \ --state-dir out/python-compat/m26e-wheel ``` The wheel must contain only target-neutral defaults. The package manifest binds that wheel to independently compiler-generated QEMU and Pi 4 `cohesix-python-profile/v2` contracts. A successful install or mock Worker observation remains host-model compatibility evidence, not target authority, READY proof, provider completion, runtime release acceptance, or production use-case acceptance. The smoke follows the selected authority policy. Compatibility profiles exercise mock spawn, READY observation and teardown. Strict profiles require refusal of an identity-free spawn and exact serialization of a versioned Queen intent with the selected writer epoch for each executable role. MockBackend retains those bytes without executing strict intents; its smoke therefore reports `strict-request-serialization` and does not claim READY or completed teardown. The separate live target projection and release lanes retain those obligations. After the direct QEMU role gate has emitted an accepted, live `cohesix-worker-integration-evidence/v1` record for `worker-control`, `gpu-receipt-path`, or `peft-receipt-path`, run the QEMU projection lane: ```bash scripts/ci/python_compat_run.sh \ --python-matrix 3.11,3.13 \ --target qemu \ --profile-contract configs/generated/cohesix_python_qemu_smp_production.json \ --wheel-dir out/python-wheels \ --package-manifest out/python-compat/m26e-python-package.json \ --matrix configs/host_integration_acceptance.toml \ --target-session out//worker-control.json \ --state-dir out/python-compat/m26e-qemu ``` The result is the release-required `python-sdk-projection` row. It consumes the direct role record and copies its exact target-session identities; it does not replace that record or raise its proof class. The Pi 4 invocation uses the Pi profile contract and an independently accepted fresh-Pi role record. Do not run or report the Pi lane from QEMU evidence. Stage 01 runs integrity first and then one broad host suite per distinct feature configuration. Its common-hermetic attestation may be imported into a second target state directory with `--reuse-common-from`. Stage 02 runs only the selected provisioned-target profile and release checks. Stage 05 verifies the immutable Stage 01-04 attestations and runs only unique release governance. Direct `scripts/ci/due_diligence_gate.sh` execution remains exhaustive. Pi Stage 02 uses the canonical `pi4-image-build.sh` workflow from a clean source commit. That builder generates the selected Pi contract, compiles its exact release components, seals the image, and restores the default QEMU generated files. The stage independently compares the retained Pi resolved-manifest hash to the compiler's selected Pi projection and checks the staged source/image identity against the verified image. A build failure or mismatched binding cannot pass. The default QEMU generated module is not a Pi build oracle. The resulting `selected-image-binding.json` is build evidence only; fresh Pi transport and hardware evidence retain their separate requirements. ## Claim tiers and PASS terminology Never report an unqualified “Test Plan PASS.” Report the exact claim tier(s): | Claim tier | What a PASS proves | What it does not prove | | --- | --- | --- | | `common-hermetic` | Catalog integrity, generated contracts, formatting, lint, workspace/default tests, complete production-feature host suites, Python discovery, no-std Pi runtime compile, and risk ratchet. | Provisioned root-task target builds, QEMU boot, live transport, Pi hardware, performance, UI, federation, or bundles. | | `qemu-integration` | `common-hermetic` plus the provisioned QEMU profile/release check, content-bound QEMU artifacts, and fresh Stage 03/04 boots with TCP and REST regression results. | Pi transport or hardware. | | `pi4-transport` | `common-hermetic` plus the provisioned Pi profile/release check and TCP/REST results bound to one caller-supplied Pi target/boot/image evidence record. | Reflash/readback, RF, driver ownership, repeatability, benchmark, or hardware acceptance. | | `pi4-hardware` | A separate machine-validated bundle containing image/readback identity, fresh serial proof, capture manifest, driver-task proof, and required repeatability report. | Performance unless the performance tier also passes. | | `ui` | Deterministic replay-mode Playwright presentation and transcript coverage. | Control-plane authority or protocol correctness. | | `performance` | Named no-retry/error-budget matrices with reviewable summaries. | General functional or hardware acceptance. | | `federation` | Named three-hive relay, dedupe, WAL resume, failover, timeline, and scale evidence. | Unrelated release or hardware claims. | | `release` | Unique advisory/governance checks; bundle validation is an additional release action when bundles are shipped. | Any target tier not explicitly included in the result. | The normal five-stage QEMU run produces `common-hermetic`, `qemu-integration`, and repository `release` governance evidence. The normal five-stage Pi run produces `common-hermetic`, `pi4-transport`, and repository `release` governance evidence. It must never be described as `pi4-hardware` without the separate hardware bundle. Conditional UI, performance, federation, Pi hardware, and bundle actions are selected by their catalog `trigger_paths` or by the active milestone. An unknown changed path selects the complete catalog conservatively. ## Immutable evidence and resume contract Each stage attempt records: - the Git HEAD plus every tracked and non-ignored untracked source file, including mode and submodule state; - `Cargo.lock`, selected manifests/generated outputs, non-secret selectors, selected seL4 profile identity, toolchain versions, OS, target, and exact action-catalog digest; - redacted argv, exit status, start/end timestamps, duration, and hashed logs for every action; - required artifacts and their content hashes, including the exact QEMU image or caller-supplied Pi target evidence where applicable; and - a terminal immutable stage manifest published atomically only after all assertions pass. `stage_XX.attestation` is an atomic reference to the immutable manifest. Input context v2 uses `scripts/ci/qemu_artifact.py source-digest` for the same complete Git, checkout-byte, executable-mode, and submodule identity used by standalone artifacts and the release factory. The context stores the untagged hex digest; artifact/result interfaces retain the `sha256:` prefix. The separate `source` object retains the staged Git-status and submodule-state metadata, bound by `source_state_digest` to preserve the existing resume/import guards alongside the shared source digest. Context v1 evidence remains historical and is never translated to v2; a changed context requires a fresh attempt through the normal fail-closed resume/import checks. The explicitly approved `1.0.0-beta` release is the sole carry-forward exception. [RELEASE_1_0_0_BETA_CARRY_FORWARD.toml](audit/RELEASE_1_0_0_BETA_CARRY_FORWARD.toml) binds the owner's approval, the four authentic predecessor attestations and contexts, the reviewed host-only DD30 successor, closure-file scope and a fixed set of later evidence receipts with their original verdicts. `scripts/ci/release_stage5_acceptance.py` validates those immutable records and the exact clean successor input, then completes the unique release governance checks. It emits a separate `release-stage5-acceptance.json` with `PASS_WITH_RESIDUAL_RISK`; it never creates current-source Stage 01–04 markers. The ordinary runner retains the current-context checks below. This exception does not waive release-bundle source or content integrity, other target proof, performance thresholds, or any future source/release requirement. Compatibility `.done` files are not authority and are published only after the attestation verifies. Missing/malformed provenance, changed inputs, tampered logs/actions/artifacts, an iteration result, target mismatch, or a failed attempt blocks resume. `target.env` is created once and cannot be overwritten with a different target or start identity. A state directory has one writer: the runner holds its lock across all selected stages, and a concurrent writer must use a different state directory. Secrets named like tokens, passwords, tickets, credentials, API keys, or authorization values are redacted from command logs and structured evidence. Pass secrets through inherited environment variables; do not interpolate them into logged shell command strings. ## Target-qualified runner matrix | Target | Stages | Required target-specific evidence | | --- | --- | --- | | `qemu` | 01-05 | Stage 03 builds one immutable artifact per unique manifest, content-binding all eight packaged host executables (`cas-tool`, `coh`, `cohsh`, `gpu-bridge-host`, `hive-gateway`, `host-sidecar-bridge`, `host-ticket-agent`, and `swarmui`), then uses a fresh boot for every regression group. Stage 04 reuses the validated default artifact but starts another fresh boot. Result manifests bind source, profile, manifest, image, scripts, boot identity, counts, and log hashes. | | `pi4` | 01-05 | Stage 03 requires the Pi TCP endpoint, initial target evidence and `COHSH_PI4_BOOT_COLLECTOR`; it retains a fresh boot of the exact image for each group. Stage 04 requires an existing gateway bound to the final base-group boot/image. These stages yield only `pi4-transport`. `TP_PI4_HARDWARE_EVIDENCE_FILE`, when required, must validate the stronger hardware bundle and is never synthesized by the runner. | Declare the Pi TCP host, external gateway URL, client/policy overrides and external broker response-timeout settings before Stage 02, and preserve them through Stages 03–05. These selectors are part of target-stage input identity; introducing them only when a later stage needs them correctly invalidates the earlier context. The boot/image-bound target-evidence file may be supplied after the fresh image has booted. Keep the gateway stopped during direct TCP testing, then start it with those declared settings for Stage 04. A Pi Stage 03 run refuses loopback unless `TP_PI4_ALLOW_LOOPBACK=1` records an intentional tunnel. A Pi Stage 04 run without an existing gateway fails rather than creating misleading local-QEMU evidence. ## Canonical action catalog `configs/test_plan_actions.toml` is the sole staged command inventory. It owns action IDs, exact commands, feature sets, stages, claim tiers, targets, trigger paths, timeouts, expected evidence, and zero-test policy. `scripts/ci/test_plan_catalog.py` validates semantic duplicates and forbids zero-match-prone filtered library-test actions. The table below is generated from that catalog. | Action | Stage | Claim tier | Scope / target | Command or proof | | --- | ---: | --- | --- | --- | | `integrity.cargo-metadata` | 1 | `common-hermetic` | common / qemu, pi4 | `cargo metadata --locked --no-deps` | | `integrity.generated-contracts` | 1 | `common-hermetic` | common / qemu, pi4 | `scripts/check-generated.sh` | | `host.format` | 1 | `common-hermetic` | common / qemu, pi4 | `cargo fmt --all -- --check` | | `host.clippy` | 1 | `common-hermetic` | common / qemu, pi4 | `CARGO_INCREMENTAL=0 cargo clippy --workspace --all-targets -- -D warnings` | | `host.workspace-check` | 1 | `common-hermetic` | common / qemu, pi4 | `CARGO_INCREMENTAL=0 cargo check --workspace` | | `host.workspace-tests` | 1 | `common-hermetic` | common / qemu, pi4 | `CARGO_INCREMENTAL=0 cargo test --workspace --exclude swarmui --exclude pi4-driver-runtime` | | `host.swarmui-tests` | 1 | `common-hermetic` | common / qemu, pi4 | `CARGO_INCREMENTAL=0 cargo test -p swarmui` | | `host.coh-mock-tests` | 1 | `common-hermetic` | common / qemu, pi4 | `cargo test -p coh --features mock` | | `host.root-task-qemu-features` | 1 | `common-hermetic` | common / qemu, pi4 | `cargo test -p root-task --no-default-features --features driver-tests-qemu --lib -- --test-threads=1 --skip drivers::driver_task_net` | | `host.root-task-pi4-features` | 1 | `common-hermetic` | common / qemu, pi4 | `cargo test -p root-task --no-default-features --features driver-tests-pi4 --lib -- --test-threads=1` | | `host.root-task-net-console` | 1 | `common-hermetic` | common / qemu, pi4 | `cargo test -p root-task --no-default-features --features net-console --lib -- --test-threads=1` | | `host.console-network-direct-genet` | 1 | `common-hermetic` | common / qemu, pi4 | `cargo test -p console-network-runtime --features direct-genet` | | `host.pi4-runtime-tests` | 1 | `common-hermetic` | common / qemu, pi4 | `cargo test -p pi4-driver-runtime -- --test-threads=1` | | `host.pi4-runtime-target-check` | 1 | `common-hermetic` | common / qemu, pi4 | `cargo check -p pi4-driver-runtime --target aarch64-unknown-none` | | `host.cache-maintenance-tests` | 1 | `common-hermetic` | common / qemu, pi4 | `cargo test -p root-task --no-default-features --features cache-maintenance --test cache_maintenance` | | `host.coh-doctor-smoke` | 1 | `common-hermetic` | common / qemu, pi4 | `cargo run -p coh --features mock -- doctor --mock` | | `host.swarmui-dependency-policy` | 1 | `common-hermetic` | common / qemu, pi4 | `python3 scripts/ci/check_swarmui_dependencies.py` | | `host.driver-coverage-contract` | 1 | `common-hermetic` | common / qemu, pi4 | `python3 scripts/ci/check_driver_test_coverage.py` | | `host.python-tests` | 1 | `common-hermetic` | common / qemu, pi4 | `scripts/ci/python_test_gate.sh --tests` | | `host.python-examples` | 1 | `common-hermetic` | common / qemu, pi4 | `scripts/ci/python_test_gate.sh --examples` | | `host.rust-risk-bootstrap` | 1 | `common-hermetic` | common / qemu, pi4 | `python3 scripts/ci/test_rust_risk_gate.py` | | `host.rust-risk-ratchet` | 1 | `common-hermetic` | common / qemu, pi4 | `env -u CARGO_HOME scripts/ci/rust_risk_gate.sh --baseline docs/audit/rust_risk_baseline.toml` | | `target.qemu-profile` | 2 | `qemu-integration` | provisioned-target / qemu | `"${TEST_PLAN_ROOT}/out/toolchain/sel4-profile-venv/bin/python" scripts/sel4_profile.py validate --profile qemu_smp_production --build-dir "${TEST_PLAN_ROOT}/out/sel4/profile-v2/qemu-smp-production" --require-source --require-artifacts --for-runtime` | | `target.root-task-qemu-release` | 2 | `qemu-integration` | provisioned-target / qemu | `scripts/ci/test_plan_target_root_check.sh --target qemu --sel4-build "${TEST_PLAN_ROOT}/out/sel4/profile-v2/qemu-smp-production" --profile qemu_smp_production --features release-qemu --timer-clock-hz 24000000` | | `target.pi4-profile` | 2 | `pi4-transport` | provisioned-target / pi4 | `"${TEST_PLAN_ROOT}/.venv/bin/python" scripts/sel4_profile.py validate --repo-managed --profile pi4_production --build-dir "${TEST_PLAN_ROOT}/seL4/build_UBOOT" --require-artifacts --for-runtime` | | `target.root-task-pi4-release` | 2 | `pi4-transport` | provisioned-target / pi4 | `scripts/ci/test_plan_target_root_check.sh --target pi4 --sel4-build "${TEST_PLAN_ROOT}/seL4/build_UBOOT" --profile pi4_production --features release-pi4 --timer-clock-hz 54000000` | | `qemu.tcp-regression` | 3 | `qemu-integration` | target / qemu | `scripts/cohsh/run_regression_batch.sh` | | `qemu.production-tcp-smoke` | conditional | `qemu-integration` | conditional / qemu | `COHSH_PRODUCTION_STRICT=1 COHSH_BATCH_GROUPS=base scripts/cohsh/run_regression_batch.sh` | | `pi4.tcp-regression` | 3 | `pi4-transport` | target / pi4 | `scripts/cohsh/run_regression_batch.sh` | | `qemu.rest-regression` | 4 | `qemu-integration` | target / qemu | `scripts/ci/test_plan_stage_04_rest_multiplexer.sh` | | `pi4.rest-regression` | 4 | `pi4-transport` | target / pi4 | `scripts/ci/test_plan_stage_04_rest_multiplexer.sh` | | `release.unique-governance` | 5 | `release` | target / qemu, pi4 | `scripts/ci/due_diligence_gate.sh` | | `diagnostic.qemu-canary` | NON-CLAIMING diagnostic | `non-claiming` | conditional / qemu | `scripts/ci/test_plan_target_canary.sh --target qemu` | | `diagnostic.pi4-canary` | NON-CLAIMING diagnostic | `non-claiming` | conditional / pi4 | `scripts/ci/test_plan_target_canary.sh --target pi4` | | `diagnostic.guard-root-mcs` | NON-CLAIMING diagnostic | `non-claiming` | conditional / qemu | `cargo test -p root-task --no-default-features --test mcs_activation_order -- --test-threads=1` | | `diagnostic.guard-worker` | NON-CLAIMING diagnostic | `non-claiming` | conditional / qemu | `cargo test -p root-task --no-default-features --test worker_fault_lifecycle -- --test-threads=1` | | `diagnostic.guard-ninedoor` | NON-CLAIMING diagnostic | `non-claiming` | conditional / qemu | `cargo test -p root-task --no-default-features --test ninedoor_service_isolation -- --test-threads=1` | | `diagnostic.guard-console-network` | NON-CLAIMING diagnostic | `non-claiming` | conditional / qemu | `cargo test -p console-network-abi && cargo test -p console-network-runtime && cargo test -p console-network-runtime --features direct-genet && cargo test -p root-task --test console_network_service && cargo test -p root-task --test direct_genet_network_phasing && cargo test -p root-task --no-default-features --test isolated_virtio_network_phasing -- --test-threads=1 && .venv/bin/python -m pytest -q tests/test_console_network_runtime_packaging.py tests/test_qemu_tcp_response_matrix.py scripts/ci/test_run_regression_batch.py` | | `diagnostic.guard-pi4-driver` | NON-CLAIMING diagnostic | `non-claiming` | conditional / pi4 | `cargo test -p root-task --no-default-features --features driver-tests-pi4 --lib -- --test-threads=1` | | `diagnostic.guard-pi4-driver-contracts` | NON-CLAIMING diagnostic | `non-claiming` | conditional / pi4 | `cargo test -p root-task --no-default-features --test driver_task_mcs -- --test-threads=1 && cargo test -p coh-rtc --test pi4_profile && cargo test -p console-network-runtime --features direct-genet && cargo test -p root-task --test direct_genet_network_phasing` | | `diagnostic.guard-live-transport` | NON-CLAIMING diagnostic | `non-claiming` | conditional / qemu, pi4 | `cargo test -p cohsh --no-default-features --features tcp` | | `diagnostic.python-sdk` | NON-CLAIMING diagnostic | `non-claiming` | conditional / qemu, pi4 | `scripts/ci/python_test_gate.sh --sdk-tests` | | `diagnostic.test-plan-tooling` | NON-CLAIMING diagnostic | `non-claiming` | conditional / qemu, pi4 | `python3 scripts/ci/test_test_plan_catalog.py && python3 scripts/ci/test_test_plan_converge.py` | | `ui.swarmui-playwright` | conditional | `ui` | conditional / qemu, pi4 | `scripts/ci/swarmui_ui_gate.sh --run` | | `performance.gateway-telemetry` | conditional | `performance` | conditional / qemu, pi4 | evidence-only: telemetry-summary-matrix, ops-csv, ramp-csv, ramp-svg | | `federation.m28-jobs-live` | conditional | `federation` | conditional / qemu, pi4 | `scripts/ci/provider_conformance_run.sh --matrix configs/provider_conformance.toml --case m28-jobs-live --reference-config "${RELEASE_B_REFERENCE:?}" --host-profile "${RELEASE_B_HOST_PROFILE:?}" --state-dir "${RELEASE_B_EVIDENCE:?}/m28-jobs-live"` | | `federation.m28-authority-live` | conditional | `federation` | conditional / qemu, pi4 | `scripts/ci/provider_conformance_run.sh --matrix configs/provider_conformance.toml --case m28-authority-live --reference-config "${RELEASE_B_REFERENCE:?}" --host-profile "${RELEASE_B_HOST_PROFILE:?}" --state-dir "${RELEASE_B_EVIDENCE:?}/m28-authority-live"` | | `federation.m28a-workloads-live` | conditional | `federation` | conditional / qemu, pi4 | `scripts/ci/provider_conformance_run.sh --matrix configs/provider_conformance.toml --case m28a-workloads-live --reference-config "${RELEASE_B_REFERENCE:?}" --host-profile "${RELEASE_B_HOST_PROFILE:?}" --state-dir "${RELEASE_B_EVIDENCE:?}/m28a-workloads-live"` | | `federation.m28a-recovery-live` | conditional | `federation` | conditional / qemu, pi4 | `scripts/ci/provider_conformance_run.sh --matrix configs/provider_conformance.toml --case m28a-recovery-live --reference-config "${RELEASE_B_REFERENCE:?}" --host-profile "${RELEASE_B_HOST_PROFILE:?}" --state-dir "${RELEASE_B_EVIDENCE:?}/m28a-recovery-live"` | | `federation.m28b-peft-live` | conditional | `federation` | conditional / qemu, pi4 | `scripts/ci/provider_conformance_run.sh --matrix configs/provider_conformance.toml --case m28b-peft-live --reference-config "${RELEASE_B_REFERENCE:?}" --host-profile "${RELEASE_B_HOST_PROFILE:?}" --state-dir "${RELEASE_B_EVIDENCE:?}/m28b-peft-live"` | | `federation.m28b-serving-live` | conditional | `federation` | conditional / qemu, pi4 | `scripts/ci/provider_conformance_run.sh --matrix configs/provider_conformance.toml --case m28b-serving-live --reference-config "${RELEASE_B_REFERENCE:?}" --host-profile "${RELEASE_B_HOST_PROFILE:?}" --state-dir "${RELEASE_B_EVIDENCE:?}/m28b-serving-live"` | | `ui.m28c-platform-live` | conditional | `ui` | conditional / macos | `scripts/ci/provider_conformance_run.sh --matrix configs/provider_conformance.toml --case m28c-platform-live --reference-config "${M28C_PLATFORM_REFERENCE:?}" --host-profile macos-apple-silicon --state-dir "${M28C_EVIDENCE:?}/m28c-platform-live"` | | `federation.m28c1-mlx-live` | conditional | `federation` | conditional / qemu, macos | `scripts/ci/provider_conformance_run.sh --matrix configs/provider_conformance.toml --case m28c1-mlx-live --reference-config "${M28C1_MLX_REFERENCE:?}" --host-profile mac-apple-m4-macos27 --state-dir "${M28C1_EVIDENCE:?}/mlx-live"` | | `federation.m28c1-vmlx-live` | conditional | `federation` | conditional / qemu, macos | `scripts/ci/provider_conformance_run.sh --matrix configs/provider_conformance.toml --case m28c1-vmlx-live --reference-config "${M28C1_VMLX_REFERENCE:?}" --host-profile mac-apple-m4-macos27 --state-dir "${M28C1_EVIDENCE:?}/vmlx-live"` | | `federation.m28g-adoption-live` | conditional | `federation` | conditional / qemu, pi4 | `scripts/ci/provider_conformance_run.sh --matrix configs/provider_conformance.toml --case m28g-adoption-live --reference-config "${M28G_ADOPTION_REFERENCE:?}" --host-profile mac-apple-m4-macos27 --state-dir "${M28G_EVIDENCE:?}/m28g-adoption-live"` | | `federation.m28g-integration-live` | conditional | `federation` | conditional / qemu, pi4 | `scripts/ci/provider_conformance_run.sh --matrix configs/provider_conformance.toml --case m28g-integration-live --reference-config "${M28G_INTEGRATION_REFERENCE:?}" --host-profile "${M28G_HOST_PROFILE:?}" --state-dir "${M28G_EVIDENCE:?}/m28g-integration-live"` | | `federation.three-hive-relay` | conditional | `federation` | conditional / qemu, pi4 | evidence-only: federation-result-manifest, relay-counter-snapshots, evidence-timeline, scale-summary | | `pi4.hardware-acceptance` | conditional | `pi4-hardware` | conditional / pi4 | evidence-only: pi4-image-readback-identity, pi4-gate-proof, pi4-capture-manifest, pi4-repeatability-report | | `release.bundle-validation` | conditional | `release` | conditional / qemu, pi4 | `python3 scripts/release_qualify.py verify --macos-result "${TP_RELEASE_MACOS_RESULT:?}" --linux-result "${TP_RELEASE_LINUX_RESULT:?}" --pi4-result "${TP_RELEASE_PI4_RESULT:?}" --releases-dir "${TP_RELEASE_DIR:?}" --output "${TP_RELEASE_RESULT:?}"` | | `release.bundle-validation-b` | conditional | `release` | conditional / qemu, pi4 | `python3 scripts/release_qualify.py verify --macos-result "${TP_RELEASE_MACOS_RESULT:?}" --linux-result "${TP_RELEASE_LINUX_RESULT:?}" --pi4-result "${TP_RELEASE_PI4_RESULT:?}" --macos-installer-result "${TP_RELEASE_MACOS_INSTALLER_RESULT:?}" --linux-installer-result "${TP_RELEASE_LINUX_INSTALLER_RESULT:?}" --releases-dir "${TP_RELEASE_DIR:?}" --output "${TP_RELEASE_RESULT:?}"` | `performance.gateway-telemetry` may be selected alongside either target, but its retained evidence is Conditional D's host-model gateway comparator; it is not QEMU or Pi target-performance evidence. ### M28g release selection — 2026-09-28 For the stable 1.2.0 cut, the release owner selected the completed checks below as the acceptance gate. Keep each result's original source, artifact, host and proof class; this selection does not rewrite test output. 1. Canonical QEMU and physical Pi Stages 01–05; physical packaged-SD readback, fresh boot and TCP qualification; exact Mac and Jetson host-archive checks; Jetson medium/high KVM pressure and the retained 60-minute Pi diagnostic. 2. The five-part strict `release.bundle-validation-b` result for the same three distributed archives and both native installer sets. Publisher signatures, package-manager receipts and installed-byte readback were checked at their recorded hashes. Review later documentation and proof-tool commits through `scripts/release_publication.py`; keep the sealed artifacts byte-identical. 3. Live M28d MCP, M28e A2A and M28f NeMo component results, their independent CUDA/PEFT outcome checks, the Mac MLX component result, and the focused `agent_protocol_controls` real-gateway test covering both, MCP-only, A2A-only and neither. These are separate component and host observations. 4. The named owner's full `m28g-adoption-live` manual-attestation PASS bound to the installed package receipts. Report absent numeric setup, download and first-job measurements as `not-recorded`. The catalogued `m28g-integration-live` case and an additional raw capability-by-capability cross-client matrix remain available for expanded qualification; they are not selected gates for this 1.2.0 cut. Public claims use the focused, component-live, installed and target results above. Release approval and PyPI publication retain their own records. For `m28g-integration-live`, first run `cargo test --locked -p hive-gateway --test agent_protocol_controls` against the selected source. Its real gateway processes use a mock Queen to check both, MCP-only, A2A-only and neither route availability, plus explicit launch-value refusal and REST route retention. This is a focused gateway contract check. The live acceptance case still needs an exact installed gateway and target, authenticated protocol clients, one shared native job, scope/budget/revocation and disable/restart recovery evidence in all four effective modes. A gateway route status cannot establish those results. The live integration reference (`cohesix-m28g-integration-reference/v1`) names the clean source commit, extracted qualified host bundle and matching installed `hive-gateway`, live Queen host/profile/manifest, private credential references, shared standing ledger/scopes and one terminal `/host/tickets/current/` path with its byte hash. The runner starts the installed gateway on loopback four times, with only explicit `false` launch controls, and rechecks authenticated `/proc/boot`, the exact original terminal and enabled/disabled route statuses each time. It then revalidates ordinary MCP and A2A SDK records against one native result, an independent CUDA output report and separately retained refusal, Queen-loss and latency observations. Each external observation has raw SHA-256 attachments; retain its collection commands and source/target identities for review. The latency attachment contains call-level CSV samples plus a raw client log, and the runner recomputes the nearest-rank p95 from those samples. Execute the case on each advertised gateway host. Its KVM path binds the live rootserver and source marker on JetPack; its physical Pi path requires the packaged SD media and fresh-boot qualification record. QEMU and Pi pressure and repeatability acceptance remain separate gates. No mode probe or JSON attachment alone qualifies a provider or release. `m28g-adoption-live` uses one private JSON reference with schema `cohesix-m28g-adoption-reference/v1`. It names absolute paths for the clean qualified source root, its commit, the assembled `release_qualify.py verify` result, both native `installer` results and a named evaluator's walkthrough. Keep the reference and walkthrough in ignored private evidence, with credentials omitted. The walkthrough schema is `cohesix-m28g-adoption-walkthrough/v2`: record the evaluator's kind, identity, independence and assistance. Lukas Bower may evaluate this sole-developer release with `kind=person` and `independent=false`; label that result owner-evaluated, while other evaluators must be independent. Record separate Mac and JetPack Linux installation steps, time, downloads, interventions and package hashes; time to the first verified CUDA result; the three selected journey IDs and original job IDs; Finder/Spotlight/Dock and GNOME grid/search launch; doctor and data-preserving rollback/uninstall results. Bind raw Mac/Linux GUI, doctor, lifecycle and journey reports by absolute path, byte size and SHA-256. The runner rechecks both native installer qualifications, exact source and every attachment against the frozen BENCHMARKS limits. Its `live_host` PASS is an installed adoption record; the native outcomes and target still need their separate live provider and integrated release cases. Alternatively, the named release owner may submit `cohesix-m28g-owner-manual-attestation/v1` after personally completing the full walkthrough. It records his statement, date, exact seven walkthrough areas and native installer package hashes. The runner checks the owner identity, selected source, five-part release result and both installed package receipts, then reports `owner_attested_host` with measurements `not-recorded`. This route makes no numeric setup or first-work claim and does not reverify CUDA/PEFT/client outcomes. Preserve the owner's original statement and any independently retained native evidence; a manual PASS may not be reused for `m28g-integration-live` or a target/provider gate. Resumable critical service loops use the selected generated NaturalPostpone policy. Retain their reserved timeout caps and independent standard-fault routing, and require successful post-refill work and receipts under pressure. Passive Worker timeout injection, NineDoor recovery, emergency fail-stop, per-core reservations, and all existing pressure thresholds remain required. Kernel `scConsumed` is cumulative since its last reset; do not report it as single-operation WCET. A debugger-interrupted kernel snapshot is diagnostic evidence and cannot replace uninterrupted timing or repeatability results. ## GitHub Actions gate mapping `.github/workflows/ci.yml` is the sole repository-authored workflow and keeps the stable required check `ci` directly; there is no aggregate fan-in job. - Per-commit CI is deliberately a small health signal. On a clean macOS 26 runner it checks locked Cargo resolution and generated contracts, checks the shipped host workspace, runs the default workspace tests once, compiles the QEMU and Pi root-task feature test binaries without executing their complete behavior matrices, and checks `pi4-driver-runtime` for `aarch64-unknown-none`. - Formatting, strict lint/risk ratchets, complete production-feature behavior matrices, repository-wide Python tests, Playwright, examples, and target execution remain change-focused developer or canonical Test Plan work. They are not repeated on every push. A green `ci` job is not `common-hermetic`, QEMU, Pi, UI, performance, federation, or release evidence. - `dependency-audit` runs only on the weekly schedule and explicit manual dispatch, using pinned `cargo-audit` and `cargo-deny` versions. Network-fetched advisory data therefore remains visible without making ordinary source changes depend on it. - CI retains native GitHub job logs and no custom evidence uploads. Reusable provenance-bound evidence still comes only from the staged Test Plan. ## Preflight and guardrails - `scripts/ci/test_plan_run.sh --list` (verify scripted stage inventory before execution) - `scripts/ci/check_test_plan.sh` - If IR or manifest changes: `cargo run -p coh-rtc` then `scripts/check-generated.sh`. - Ensure `SEL4_BUILD_DIR` points at the validated production SMP kernel build (`$REPO/out/sel4/profile-v2/qemu-smp-production` by default). Preserved `seL4/build` or `seL4/SMP_build` trees may be selected explicitly for diagnostic comparison only and are claim-ineligible unless they pass a named profile contract. - Default QEMU SMP topology is four single-threaded cores; set `COHESIX_QEMU_SMP=1` for single-core baselines or `COHESIX_QEMU_SMP_TOPO` for explicit topologies. - seL4 16 QEMU artifact trees must be configured with `ElfloaderRootserversLast=ON`, scalar pre-MMU elfloader/libcpio code, and an embedded QEMU `virt,gic-version=3,virtualization=off` DTB whose PSCI method is `hvc`. The wrapper must select HVC `CPU_ON` only for an HVC DTB, retain the upstream SMC path for SMC-selected platforms, and reject a duplicate or missing PSCI SMP driver. Apple-Silicon/macOS uses HVF, `cortex-a57`, and `kernel-irqchip=off`; AArch64 Linux uses KVM, the `host` CPU, and the in-kernel GICv3. Generated `TIMER_CLOCK_HZ` and the console-network descriptor must equal the guest-visible virtual-counter frequency: 24,000,000 Hz for macOS `qemu_smp_production` and 31,250,000 Hz for Linux `qemu_smp_kvm_production`. TCG, `-icount`, or a mismatched timer frequency is diagnostic-only and cannot establish QEMU acceptance or performance evidence. - Milestone 26d profile closure requires all five fresh `out/sel4/profile-v2/*` defaults to pass the fail-closed aggregate validator with source and artifacts required. QEMU build, release, regression, and publication entrypoints consume and revalidate `qemu_smp_production` by default. This makes it the canonical GICv3 build input; it is still not QEMU boot evidence. The repo-managed `seL4/build_UBOOT` exact-image CYW43 input remains separately coordinated and is not Pi hardware or release evidence. Exact-image composition treats that tracked tree as immutable input: the image wrapper validates relocated artifact digests, fingerprints the complete tree, reconstructs the tracked baseline elfloader byte-for-byte as a toolchain oracle, and relinks the new rootserver only in disposable output. Derived provenance binds the canonical stamp/tree, tool identities and oracle, and exact rootserver/CPIO/wrapper tuple; `--skip-build` may reuse only that provenance-bound assembly. No Pi seL4 source or build input may be selected below `out/`, and CMake or Ninja must never mutate `seL4/build_UBOOT`. - macOS: FUSE mount coverage is optional unless the MacFUSE runtime is installed and approved (verify `/dev/macfuse0` exists, or `/dev/osxfuse0` on older OSXFUSE). - On Linux, KVM requires `/dev/kvm`, `-cpu host`, and the in-kernel GICv3; `kernel-irqchip=off` is the macOS HVF envelope and is invalid for this KVM configuration. The launcher must agree with the generated GICv3 and 31,250,000 Hz timer truth of `qemu_smp_kvm_production`. - Before any QEMU TCP run, start tcpdump and confirm the log path (example: `logs/tcpdump-new-YYYYMMDD-HHMMSS.log`). Use the same path in TCP correlation checks. Observe the guest's `root-console.start.ok` serial marker before the first authenticated console request. Do not probe the forwarded console port with an unauthenticated connection: QEMU may defer that connection until the guest listens, consuming its sole console slot during a later test operation. For automated runs without host BPF privileges, set `COHESIX_QEMU_CAPTURE_DIR` to a private evidence directory and `COHESIX_QEMU_TCPDUMP` to a tcpdump executable that supports pcap input through `-r -`. Both canonical QEMU launchers start tcpdump before QEMU and retain the complete guest `net0` stream, decoded log, launch command and completion record in a fresh directory for each boot. Check every `result.json` has `complete=true`; an early decoder exit or truncated stream fails the launcher. These captures observe guest network traffic; use a host-interface capture when diagnosing host-side REST traffic. Keep the capture root outside a pressure runner's disposable `out/` tree. - Headless Linux requires `xvfb-run` (`sudo apt-get install -y xvfb` if missing). - Ensure `/updates` and `/host` are enabled for host tool tests: - `cas.enable = true` (and `ui_providers.updates.*` as needed) - `ecosystem.host.enable = true` with providers set - Re-run `coh-rtc` and `scripts/check-generated.sh` if toggled. - Use a fresh run/log directory. Retain failed attempts and their exact inputs; do not clear shared log or evidence directories before a rerun. The hosted `driver-tests-pi4` suite selects compiler-generated Pi tables from `apps/root-task/tests/support/generated/pi4/`. This preserves the default QEMU tables while exercising the Pi registry, bounds and admission configuration. After changing the Pi manifest, regenerate this test profile with `scripts/check-generated.sh --update-pi4-test-profile`; the ordinary generated check verifies both profiles. These host tests do not constitute Pi boot or hardware acceptance. ## Performance baselines (Authoritative) - Performance evidence is only valid when it is **stored and reviewable**: - Preserve the canonical harness `*.summary.json` plus associated logs or target proof under `logs/bench/`, `out/bench/`, or a milestone-approved committed evidence path; and - Record its baseline/comparison/claim and artifact hashes in the owning BUILD_PLAN task or audit record. Follow `docs/BENCHMARKS.md` for methodology; do not duplicate the measurement history there or here. - Do not use "last local run" as a baseline. If you need a new baseline, preserve and index its artifacts in the same change. Commit raw evidence only when the active milestone requires it; otherwise retain the immutable bundle. ## Staged and conditional procedures For acceptance, run in order. Skips produce INCOMPLETE markers and the stage will fail. During active target convergence, use the separate non-claiming entry point above; its result cannot satisfy any procedure in this section. - Scripted runner (recommended): `scripts/ci/test_plan_run.sh --state-dir out/test-plan/` ### Automated Stage 01 — Reusable common-hermetic closure Run `scripts/ci/test_plan_stage_01_integrity.sh`. The generated catalog table is the sole Stage 01 command inventory; do not replay named tests as filters. Stage 01 deliberately runs one complete harness for each distinct configuration: - locked Cargo metadata and generated-contract/catalog integrity; - workspace formatting, Clippy, check, and default tests, partitioned so SwarmUI and `pi4-driver-runtime` are not rerun by the workspace action; - complete SwarmUI, `coh --features mock`, root-task QEMU, root-task Pi 4, minimal `net-console`, cache-maintenance, direct-GENET console runtime, and isolated Pi runtime suites; - the `aarch64-unknown-none` Pi runtime compile, host dependency/driver policy, Python discovery and examples, and the Rust-risk bootstrap/ratchet. Normal harness parallelism is retained. Serialization is limited to the stateful root-task feature suites and isolated Pi runtime boundary recorded in the catalog. `scripts/ci/check_driver_test_coverage.py` maps the documented HAL/driver invariants to those broad suites and fails if the feature or target closure drifts. Every catalogued test action enforces a non-zero inventory, so a renamed or removed test cannot turn an empty filtered run green. The Python actions use `scripts/ci/python_test_gate.sh`: a shared virtual environment keyed by the canonical Python executable and hashed requirements, with exact `pytest`, `pyserial`, and Python-package build-backend (`setuptools`) pins. Repository, client, due-diligence, and runner contract tests execute in one pytest process; four mock examples execute once in a separate smoke action. A missing Python lane is INCOMPLETE, never PASS. `scripts/check-generated.sh` already invokes `scripts/ci/check_test_plan.sh`, so Stage 01 does not repeat that check. Fixture regeneration is intentionally outside the normal pass and is allowed only when fixtures change: - `COHESIX_WRITE_TRACE=1 cargo test -p cohsh --test trace` - `COHESIX_WRITE_TRACE=1 cargo test -p swarmui --test trace` The explicit NineDoor scale proof remains conditional rather than part of the fast common closure: - `cargo test -p nine-door --features scale-tests --test shard_scale sharded_attach_1k_scale_gate_exports_metrics -- --nocapture` Stage 01 proves bounded host-model and feature behavior, including the catalogued Secure9P, operator-liveness, driver-ring, CYW43/SDIO, GENET, USB, HDMI, and scheduling invariants. It does not prove QEMU boot, live Pi hardware, RF/DHCP, physical throughput, or image/readback identity. Pi trace normalization, image identity, and repeatability remain post-capture hardware workflows under Conditional F. ### Automated Stage 02 — Provisioned-target checks Stage 02 runs only the catalogued checks for the selected target after a fresh or imported Stage 01 common-hermetic attestation: - QEMU profile validation against `out/sel4/profile-v2/qemu-smp-production`, followed by the `release-qemu` AArch64 root-task check. The check builds fresh Worker, NineDoor, console-network, and driver-runtime identities inside the Stage 02 attempt and binds them to the root check under the selected 24 MHz profile. The console child selects `direct-virtio` independently of diagnostic tracing. - Pi 4 profile validation against the immutable `seL4/build_UBOOT` `pi4_production` artifacts, followed by the `release-pi4` AArch64 root-task check. Its independently built component bindings use the selected 54 MHz header and the canonical Pi build's `direct-genet` feature and console-network/smoltcp optimization settings; this remains compile evidence, not Pi boot or hardware acceptance. The remaining Pi-specific material in this section defines evidence semantics for Conditional F. It is not additional Stage 02 execution and must not cause the common host suites to be replayed. #### Pi 4 post-capture and hardware evidence semantics Stage 01 runs the identity/normalizer and HAL/runtime contract tests; it does not create physical evidence. Conditional F validates actual owner, DMA/cache/IRQ, serial, USB/HDMI, and CYW43/GENET observations with the image, gate-proof, normalizer, and repeatability tools. Preserve their raw inputs and failure verdicts. Missing, stale, duplicate, wrong-image, or cross-boot evidence cannot be repaired by a passing compile or a historical transcript. ### Automated Stage 03 — QEMU or Pi transport regression - `scripts/ci/test_plan_stage_03_qemu_tcp_regression.sh` - Stage 03 sets resilient readiness defaults for clean hosts: `TP_STAGE3_READY_TIMEOUT=900`, `TP_STAGE3_PORT_TIMEOUT=60`, and `TP_STAGE3_AUTH_READY_TIMEOUT=120` (override as needed). The separate authentication-readiness bound applies to Pi/live transport; QEMU evidence workloads perform their own exact AUTH exchange after the UART marker. - QEMU boots must emit exact `[mark] root-console.start.ok` before the first authenticated response-matrix or `.coh` workload. A listening TCP socket is not root-console readiness, and the runner must not consume a throwaway QEMU authentication connection before the evidence workload. - `scripts/cohsh/run_regression_batch.sh` builds one immutable artifact for the default manifest and one for the gated manifest. Base, telemetry, and shard groups reuse the default artifact bytes; every group still receives a fresh QEMU boot. - The gated manifest may change only its named audit, replay, policy, model, sidecar, UI, and client feature gates. It must preserve the selected base QEMU operational topology, including the root, Worker, temporal-authority, NineDoor, console-network, resource-admission, and timer contracts exactly. Audit/replay, model, and Modbus support must be enabled for their positive gated fixtures; repository-only CAS trust remains separate from production. - The batch snapshots generated projections and restores them in an EXIT trap, including failure and interrupt paths. Each artifact and boot result has a machine-readable source/profile/manifest/image/action/log binding. - QEMU close success is a same-connection protocol assertion. The fixed matrix and every `.coh` client must receive exact `OK QUIT` followed by target EOF; timeout, another post-terminal frame, or a client error fails the workload. The isolated console child owns close and relisten, so Stage 03 must not wait for the legacy root-stack UART string `audit tcp.conn.close`. Authentication by the next workload on that same boot proves listener restoration without a reconnect retry. Pi/live retains its lifecycle resume and per-script ledger. - Pi 4 uses the same official runner with `COHSH_BATCH_TARGET=pi4`, `COHSH_TCP_HOST=` and `COHSH_TCP_PORT=31337`. A multi-group run also requires `COHSH_PI4_BOOT_COLLECTOR=`: Worker slots and IDs are boot-local, so every group needs a fresh boot of the same exact image. Declare that selector before Stage 02 and retain it through Stage 05. The runner executes `base-telemetry`, `base-shard`, `gated`, then `base`, preserving the existing Stage 04 requirement to continue on the base boot. It retains lifecycle resume and the per-script ledger, and stops before another boot if a group fails. A single selected group may use an existing fresh boot for focused diagnosis; it cannot produce full Stage 03 PASS. `COHSH_LOG_ROOT` selects the evidence directory; the default remains `out/regression-logs/pi4-full-`. - The full Stage 03 scripts exercise development authority, including legacy `/queen/ctl` operations. A production manifest has `legacy_queen_ctl=false`; its refusal is required. Release assembly instead requires the separate catalogued `qemu.production-tcp-smoke` result on each native QEMU host. Set `COHSH_BASE_MANIFEST` to the selected production TOML, `COHSH_PRODUCTION_STRICT=1`, `COHSH_BATCH_GROUPS=base`, and `TEST_PLAN_ACTION_ID=qemu.production-tcp-smoke` when invoking `scripts/cohsh/run_regression_batch.sh`. Retain its `base.json` and exact `qemu-artifact.json` together. This boot runs nine production-safe scripts and the fixed response matrix, with selected MCP, A2A and standing controls checked before launch. It supplies production TCP proof for the archive; the full development Stage 03 and live protocol/provider cases remain separate gates. - The operator-owned collector is invoked without shell evaluation with `--group --out --prior-evidence ` and `--source-digest sha256:`. It must boot the selected image, retain exact BUILD/image/serial and any RAM or media verification receipts in that directory, and write `target-evidence.json` through `qemu_artifact.py record-pi4-evidence`. `pi4_regression_boot.py` limits each invocation to 600 seconds, terminates its owned process group, records the executable hash and outcome, and rejects source/image/host/gateway changes or any reused boot ID. It never supplies reboot, flash, sudo, serial ownership or hardware-proof authority on its own. Each group result binds its own immutable receipt; the complete aggregate still requires all 17 scripts. The final base receipt is published for Stage 04 continuity. Do not keep a serial reader or gateway open across collector calls unless that collector explicitly owns it. - The `9p_batch.coh` fixture asserts append admission, successful CAT and oversized-write refusal. TCP and REST batch runners additionally invoke `cohsh_regression_output.py`, which requires exactly `batch-1`, `batch-2`, `batch-3` in that order in the completed CAT data stream. Boot/audit records may interleave. The bounded ACK preview cannot satisfy this assertion; running the `.coh` file alone is not the complete append-order gate. - Before the staged Pi 4 transport run, create its source/boot/image/endpoint binding and pass the result as `TEST_PLAN_TARGET_EVIDENCE_FILE`: ```sh source_digest="$(scripts/ci/qemu_artifact.py source-digest --repo-root .)" scripts/ci/qemu_artifact.py record-pi4-evidence \ --output out/test-plan//pi4-target-evidence.json \ --source-digest "${source_digest}" \ --boot-id \ --image-identity sha256: \ --target-host \ --gateway-url http://: ``` Use `--gateway-target-host ` instead when the public URL is recorded separately. This caller-declared record prevents accidental target switching during Stages 03/04; it cannot independently detect a reboot or backend replacement and is transport evidence, not Pi hardware acceptance. The initial record selects the image; per-group collectors supply fresh records. Stage 04 uses the final immutable base receipt, so omit an obsolete initial `TEST_PLAN_TARGET_EVIDENCE_FILE` override when starting Stage 04. - Stage 03 archives per-script logs under the stage state dir (for example `out/test-plan//qemu-regression-logs/`). - Manual runs of `scripts/cohsh/run_regression_batch.sh` default to `out/regression-logs/` unless `COHSH_LOG_ROOT` is set. - Focused Stage 03 iteration may use `COHSH_BATCH_GROUPS=base`, `base-telemetry`, `base-shard`, or `gated` with `--iteration`. Without `--iteration`, any subset writes an INCOMPLETE record and cannot produce Stage 03 PASS evidence. For focused manual diagnosis only (not a second mandatory Stage 03 pass), start QEMU from the exact source artifact or bundle, then verify: - Capture QEMU serial to `logs/qemu-console.log` (example: `./qemu/run.sh | tee logs/qemu-console.log`). - `cohsh` (queen): `help`, `attach queen` (skip if you launched cohsh with `--role`), `log`, `tail /log/queen.log`, `ls /`, `cat /log/queen.log`, `test --mode quick`, `test --mode full`, `test --mode smp` (fresh boot), `spawn heartbeat ticks=100`, `ls /worker`, `kill worker-`, `ping`, `tcp-diag`, `quit` - If policy gating is enabled (see `/policy/rules`), enqueue approvals before `spawn` and `kill`: - `echo {"id":"spawn-1","target":"/queen/ctl","decision":"approve"} > /actions/queue` - `echo {"id":"kill-1","target":"/queen/ctl","decision":"approve"} > /actions/queue` - Capture cohsh output to `logs/cohsh-session.log` (example: `... | tee logs/cohsh-session.log`). - Success criteria: - No unexpected `ERR` lines or reconnect loops; negative tests must match their exact expected error and prove its required side-effect boundary. - ACK/ERR/END ordering stable. ### Conditional A — TCP reliability smoke Use this smoke for a changed TCP reliability path or its selected claim; it is not an extra full regression pass after every Stage 03 run. Run while QEMU is up: - Repeat `tcp-diag` 5–10 times and record results (example: `... | tee logs/tcp-diag.log`). - Run `pool bench path=/log/queen.log ops=500 batch=8 payload_bytes=64` and record throughput/latency (example: `... | tee logs/pool-bench.log`). - Functional smoke acceptance: - `tcp-diag` has zero failures. - `pool bench` completes with non-zero operations. This is a liveness smoke, not a performance claim. - The `performance` tier is qualified by the lane that owns the claim: Conditional B2 for executable QEMU target pressure, a separate fresh-Pi target-performance path for Pi, and Conditional D for the exact packaged gateway's host-model large-telemetry comparator. Any regression claim also needs reviewable baseline artifacts indexed in `docs/BENCHMARKS.md`; never compare against unpublished local runs. - Capture logs: - cohsh: `logs/cohsh-session.log` - QEMU serial: `logs/qemu-console.log` - tcpdump: recorded tcpdump log path - Fail if any unexpected disconnects: - QEMU log: `rg -n "audit tcp\\.conn\\.close reason=error|audit tcp\\.send\\.partial|audit tcp\\.send\\.error|console\\.emit\\.failed" logs/qemu-console.log` - cohsh log: `rg -n "\\[cohsh\\]\\[tcp\\] connection lost" logs/cohsh-session.log` - tcpdump: `rg -n "Flags \\[R\\]" ` - Acceptable disconnects: explicit `quit` or EOF; anything else is a defect. - `audit tcp.flush.blocked` lines before any client connects are expected; do not treat them as failures. ### Conditional B — Host tools integration Require exact packaged-tool/source identity and target-session correlation. Historical staged PASS does not close missing host-tool, Python, provider, or performance claims. The CAS boundary and rejection cases below remain required. - QEMU log correlation (required): - Record a short note per tool in `logs/host-tool-runs.md` with start/stop time and tool name. - In the QEMU log, locate matching `audit tcp.conn.open`/`audit tcp.conn.close` lines for the same window. - Verify the session ends cleanly (`reason=quit`/`eof`) and no TCP errors are present in that window. - Use: `rg -n "audit tcp\\.conn\\.open|audit tcp\\.conn\\.close|audit tcp\\.send\\.partial|audit tcp\\.send\\.error|console\\.emit\\.failed" logs/qemu-console.log` - `cohsh` (already covered in Section 3). - Control grammar sanity (requires `control_plane.*` + `/proc` observability enabled): - `echo {"id":"sched-1","role":"worker-gpu","priority":2,"ticks":3,"budget_ms":120} > /queen/schedule/ctl` - `cat /proc/schedule/summary` and `cat /proc/schedule/queue` - `echo {"op":"grant","id":"lease-1","subject":"queen","resource":"gpu0","ttl_s":300,"priority":5} > /queen/lease/ctl` - `echo {"op":"preempt","id":"lease-1","reason":"timeout"} > /queen/lease/ctl` - `cat /proc/lease/summary`, `cat /proc/lease/active`, `cat /proc/lease/preemptions` - `echo {"op":"open","id":"export-1","ttl_s":900} > /queen/export/ctl` - `echo {"op":"close","id":"export-1","reason":"window-complete"} > /queen/export/ctl` - `echo {"op":"apply","id":"rev-2026-02-03","sha256":"<64-hex>"} > /policy/ctl` - `echo {"op":"rollback","id":"rev-2026-02-03"} > /policy/ctl` - `coh` (TCP console; requires `configs/generated/coh_policy.toml`): - `./bin/coh gpu list --host 127.0.0.1 --port 31337` - `./bin/coh gpu lease --host 127.0.0.1 --port 31337 --gpu GPU-0 --mem-mb 4096 --streams 1 --ttl-s 60` - `./bin/coh run --host 127.0.0.1 --port 31337 --gpu GPU-0 -- echo ok` - `./bin/coh gpu status --host 127.0.0.1 --port 31337 --gpu GPU-0` - `./bin/coh telemetry pull --host 127.0.0.1 --port 31337 --out ./out/telemetry` - PEFT file-registry smoke (requires live GPU bridge publish; not native adapter execution): - `demo/peft_adapter` is a documented placeholder fixture. Its synthetic metrics and text weights exercise only export/import/registry plumbing; use `coh peft release` with genuine enrolled native inputs for the separate 1.1.0 release journey in [PRIVATE_LORA_RELEASE.md](PRIVATE_LORA_RELEASE.md). - Preflight: `./bin/cohsh --transport tcp --tcp-host 127.0.0.1 --tcp-port 31337 --role queen -c "ls /queen/export/lora_jobs"` - If `/queen/export/lora_jobs` is missing in dev-virt, **skip this registry smoke** and retain the missing prerequisite (this indicates no export job was seeded in the VM). Mock PEFT tests remain a separate host-only result. - `./bin/coh --host 127.0.0.1 --port 31337 peft export --job job_0001 --out ./out/peft_export` - `./bin/coh --host 127.0.0.1 --port 31337 peft import --publish --model demo-model --from demo/peft_adapter --job job_0001 --export ./out/peft_export --registry ./out/peft_registry` - `./bin/coh --host 127.0.0.1 --port 31337 peft activate --model demo-model --registry ./out/peft_registry` - Verify in `cohsh` (after closing SwarmUI): `ls /gpu/models/available` and `cat /gpu/models/active` - Optional FUSE: `./bin/coh mount --host 127.0.0.1 --port 31337 --at /tmp/coh-mount` (requires a FUSE runtime; on macOS this means MacFUSE installed and approved, typically `/dev/macfuse0`). - `swarmui` live (console + observability; do not attach cohsh simultaneously): - macOS: `./bin/swarmui` - headless Linux: `xvfb-run -a ./bin/swarmui` - Live telemetry (required for milestone-flagged UI changes): - Preconditions: Queen session reachable, workers emitting telemetry, and no parallel `cohsh` session attached. - In SwarmUI: set role `queen` + ticket, click **Connect**, then **Hive Start**. - Click a worker dot; confirm the detail panel updates within 1–2 poll intervals. - Click a telemetry overlay card; confirm the dot selection updates and the detail panel matches the card agent. - Confirm overlays show recent lines (no empty state) for at least one worker. - Record the result in `logs/host-tool-runs.md` with timestamps. - SwarmUI console exposes the core console verbs; CLI-only commands remain in `cohsh`. - `swarmui` replay: - Source tree: `./bin/swarmui --replay-trace "$(pwd)/tests/fixtures/traces/trace_v0.trace"` - Release bundle: `./bin/swarmui --replay-trace "$(pwd)/traces/trace_v0.trace"` - Source tree: `./bin/swarmui --replay "$(pwd)/tests/fixtures/traces/trace_v0.hive.cbor"` - Release bundle: `./bin/swarmui --replay "$(pwd)/traces/trace_v0.hive.cbor"` - headless Linux: prefix with `xvfb-run -a` - `cas-tool`: - Validate the selected generated host contract before using it. For the default profile, `limits.max_chunks` must be exactly `8`, `limits.max_payload_bytes` must be exactly `1024`, and `chunk_bytes` must be exactly `128`. A template that omits `limits` is legacy and falls back to the shared eight-chunk manifest-v1 maximum; if `limits` is present, neither a smaller nor a larger value is accepted. The optional `--chunk-bytes` argument must equal the selected template and cannot override it: ```bash jq -e ' .chunk_bytes == 128 and .limits.max_chunks == 8 and .limits.max_payload_bytes == (.chunk_bytes * .limits.max_chunks) ' configs/generated/cas_manifest_template.json ``` - Prepare two independent payloads. Preserve the full trace as replay truth and as the exact nine-chunk negative input. Pad the dedicated positive 359-byte fixture at `sha256:7f97db91e95d67b8cdd7baa194c563e2a03f94045a4e5c3282b4bb8e7fc3fda1` as a whole to exactly eight unique chunks using the deterministic nonzero tail byte `((index * 73 + 19) % 251) + 1`; never truncate or relabel the trace. Use the source-tree invocation from the repository root or the release invocation from the release root, never a path guessed across the two packages: ```bash prepare_cas_payloads() { CAS_TRACE_SOURCE="$1" CAS_POSITIVE_SOURCE="$2" python3 - <<'PY' import hashlib import os from pathlib import Path chunk_bytes = 128 max_chunks = 8 capacity = chunk_bytes * max_chunks output = Path("out/cas") output.mkdir(parents=True, exist_ok=True) trace = Path(os.environ["CAS_TRACE_SOURCE"]).read_bytes() trace_padded = trace + b"\0" * ((-len(trace)) % chunk_bytes) assert len(trace) == 1073 assert len(trace_padded) == 1152 (output / "trace_v0.padded").write_bytes(trace_padded) positive = Path(os.environ["CAS_POSITIVE_SOURCE"]).read_bytes() assert len(positive) == 359 assert hashlib.sha256(positive).hexdigest() == ( "7f97db91e95d67b8cdd7baa194c563e2a03f94045a4e5c3282b4bb8e7fc3fda1" ) if len(positive) > capacity: raise SystemExit("positive fixture exceeds manifest-v1 capacity") tail_len = capacity - len(positive) positive_padded = positive + bytes( ((index * 73 + 19) % 251) + 1 for index in range(tail_len) ) assert len(positive_padded) == 1024 positive_chunks = [ positive_padded[offset : offset + chunk_bytes] for offset in range(0, capacity, chunk_bytes) ] assert len({hashlib.sha256(chunk).digest() for chunk in positive_chunks}) == 8 (output / "max_chunks_v1.padded").write_bytes(positive_padded) PY } ``` - Source-tree invocation, from the repository root: ```bash cargo build --locked --release -p cas-tool CAS_TOOL=(./target/release/cas-tool) CAS_FIXTURE_SIGNING_KEY=./resources/fixtures/cas_signing_key.hex test -x "${CAS_TOOL[0]}" test -f "$CAS_FIXTURE_SIGNING_KEY" shasum -a 256 "${CAS_TOOL[0]}" prepare_cas_payloads \ tests/fixtures/traces/trace_v0.trace \ tests/fixtures/cas/max_chunks_v1.txt ``` - Release-bundle invocation, from the release root instead of the source invocation: ```bash CAS_TOOL=(./bin/cas-tool) unset CAS_FIXTURE_SIGNING_KEY test -x "${CAS_TOOL[0]}" shasum -a 256 "${CAS_TOOL[0]}" test "$(tr -d '\n' < cas/max_chunks_v1.txt.sha256)" = \ "7f97db91e95d67b8cdd7baa194c563e2a03f94045a4e5c3282b4bb8e7fc3fda1" prepare_cas_payloads \ traces/trace_v0.trace \ cas/max_chunks_v1.txt ``` - Negative pack preflight: require a fresh nonexistent `./out/cas/trace-v0-over-limit.bundle`, run `pack` on `./out/cas/trace_v0.padded`, require nonzero exit with exact causal text `CAS manifest capacity exceeded: payload_bytes=1152 chunk_bytes=128 chunks=9 max_chunks=8 max_payload_bytes=1024`, and require the bundle path to remain absent. This is local proof: do not start or contact QEMU, Pi, or a gateway for the negative case: ```bash test ! -e ./out/cas/trace-v0-over-limit.bundle if "${CAS_TOOL[@]}" pack \ --epoch 1 \ --input ./out/cas/trace_v0.padded \ --out-dir ./out/cas/trace-v0-over-limit.bundle \ --chunk-bytes 128 \ >./out/cas/trace-v0-over-limit.log 2>&1; then exit 1 fi rg -F "CAS manifest capacity exceeded: payload_bytes=1152 chunk_bytes=128 chunks=9 max_chunks=8 max_payload_bytes=1024" \ ./out/cas/trace-v0-over-limit.log test ! -e ./out/cas/trace-v0-over-limit.bundle ``` - Exact-capacity fixture pack (source-tree gated-profile invocation only): ```bash test -n "${CAS_FIXTURE_SIGNING_KEY:?source-tree fixture key is required}" "${CAS_TOOL[@]}" pack \ --epoch 1 \ --input ./out/cas/max_chunks_v1.padded \ --out-dir ./out/cas/max-chunks-fixture.bundle \ --chunk-bytes 128 \ --signing-key "$CAS_FIXTURE_SIGNING_KEY" ``` This repository key is fixture-only. Upload that bundle only to the explicitly gated QEMU artifact whose selected verification key is the matching fixture key; never upload it to operational base QEMU or Pi. - Operational pack: ```bash test -n "${COH_CAS_SIGNING_KEY:?set external CAS signing-key path}" "${CAS_TOOL[@]}" pack \ --epoch 1 \ --input ./out/cas/max_chunks_v1.padded \ --out-dir ./out/cas/max-chunks-operational.bundle \ --chunk-bytes 128 \ --signing-key "$COH_CAS_SIGNING_KEY" ``` Its public key must match the selected profile's `cas.signing.verification_key_path`. - Direct upload of the bundle appropriate to the selected artifact: ```bash "${CAS_TOOL[@]}" upload \ --bundle \ --host 127.0.0.1 \ --port 31337 \ --auth-token "$COH_AUTH_TOKEN" \ --ticket "$QUEEN_TICKET" ``` Upload must validate manifest chunk count before reading chunks or opening the socket. A foreign or legacy nine-chunk bundle must therefore fail locally with zero target connection. - Passing local preflight proves only the fixed manifest shape. The target's independent global store may still return typed `buffer-full` for an eight-chunk manifest when previously retained chunks or models consume capacity. Preserve that result; do not retry, increase capacity, or present the positive fixture as target success unless the exact target accepted it. - `gpu-bridge-host`: - `./bin/gpu-bridge-host --mock --list` - Optional NVML: `./bin/gpu-bridge-host --list` (enabled by default on Linux builds; omit NVML with `--no-default-features`) - Live publish: `./bin/gpu-bridge-host --publish --tcp-host 127.0.0.1 --tcp-port 31337 --auth-token "$COH_AUTH_TOKEN" --interval-ms 1000 --registry "$COH_GPU_REGISTRY"` - On macOS without a real compiled GPU backend, run `--mock --list` only. Fixture snapshots are rejected by the operational target and are not live evidence. - `host-sidecar-bridge`: - `./bin/host-sidecar-bridge --mock --mount /host --provider systemd --provider k8s --provider docker --provider nvidia` - `./bin/host-sidecar-bridge --tcp-host 127.0.0.1 --tcp-port 31337 --auth-token "$COH_AUTH_TOKEN" --watch` (requires `/host` enabled in `configs/root_task.toml`) - `host-ticket-agent`: - `./bin/host-ticket-agent --mock --run-once` - `./bin/host-ticket-agent --tcp-host 127.0.0.1 --tcp-port 31337 --auth-token "$COH_AUTH_TOKEN" --run-once` - REST mode (gateway required): `./bin/host-ticket-agent --rest-url http://127.0.0.1:8080 --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN" --run-once` ### Conditional B1 — Control-ticket matrix (Milestone 25g) All runs are required unless explicitly marked `NA` by platform constraints. - Ticket namespace and bounds: - Append valid ticket JSONL to `/host/tickets/spec`; verify success. - Append malformed or over-bound lines; verify deterministic `ERR`. - GPU/PEFT ticket flow: - Submit `gpu.lease.grant` and `gpu.lease.release` tickets (or `NA` if no GPU surface). - Submit `peft.import`, `peft.activate`, and `peft.rollback` tickets against a test registry. - Verify lifecycle receipts under `/host/tickets/status|deadletter`. - systemd/docker remediation flow: - Submit `systemd.restart` and `docker.restart` tickets for allowlisted targets. - Verify receipts are deterministic and bounded. - Verify non-allowlisted actions are rejected before execution. - K8s coexistence translation flow: - Use Python translation helpers (`K8sRbacIntent` -> `/host/tickets/spec`) to submit `k8s.cordon`, `k8s.drain`, and `k8s.lease.sync`. - Verify no scheduler-replacement semantics are introduced (coexistence only). - Evidence/timeline replay flow: - `./bin/coh evidence pack --host 127.0.0.1 --port 31337 --out ./out/evidence/tickets --with-telemetry` - `./bin/coh evidence timeline --in ./out/evidence/tickets` - Verify `timeline.ndjson` contains host ticket correlation keys (`id + idempotency_key`). - `hive-gateway` (REST gateway, Linux/systemd required for this section): - Install unit + env file (examples): - `sudo cp resources/systemd/hive-gateway.service /etc/systemd/system/` - `sudo tee /etc/cohesix/hive-gateway.env >/dev/null <<'EOF'` - `COH_TCP_HOST=127.0.0.1` - `COH_TCP_PORT=31337` - `COH_AUTH_TOKEN=changeme` - `COH_ROLE=queen` - `COH_TICKET=` - `HIVE_GATEWAY_BIND=127.0.0.1:8080` - `HIVE_GATEWAY_BROKER_CONTROL_RESPONSE_TIMEOUT_MS=120000` - `HIVE_GATEWAY_BROKER_TELEMETRY_RESPONSE_TIMEOUT_MS=120000` - `EOF` - `sudo systemctl daemon-reload` - `sudo systemctl enable --now hive-gateway` - Validate REST responds: `curl -sS http://127.0.0.1:8080/v1/meta/bounds | jq .` - Restart QEMU and confirm auto-reconnect: - stop QEMU, wait 5–10s, restart QEMU - `journalctl -u hive-gateway -n 200 --no-pager | rg -n "reconnect|connected|disconnected"` - Re-run: `curl -sS http://127.0.0.1:8080/v1/meta/bounds | jq .` - `sudo systemctl stop hive-gateway` - Multiplexer regression (REST gateway, QEMU running; `hive-gateway` is the sole console client): - REST API smoke (manifest + namespace + log tail): - `curl -sS http://127.0.0.1:8080/v1/meta/bounds | jq .` - `curl -sS 'http://127.0.0.1:8080/v1/fs/ls?path=/' | jq .` - `curl -sS 'http://127.0.0.1:8080/v1/fs/cat?path=/proc/lifecycle/state&max_bytes=64' | jq .` - `curl -sS 'http://127.0.0.1:8080/v1/fs/tail?path=/log/queen.log&max_bytes=512&lines=64' | jq .` - Failure classification is part of the contract: a preserved target `ERR` or exact in-process host-model schedule/lease/export semantic-capacity refusal returns HTTP `200` with `GatewayResponse.status="ERR"`; capacity retains `ERR ECHO reason=quota detail=buffer-full path= error=buffer full`. HTTP `429` means bounded broker queue backpressure, HTTP `503` means transport or session unavailability, and HTTP `504` means the broker accepted work but the backend response exceeded its response timeout. - REST `/proc` bounds (schedule + lease): - `curl -sS 'http://127.0.0.1:8080/v1/fs/cat?path=/proc/schedule/summary&max_bytes=128' | jq .` - `curl -sS 'http://127.0.0.1:8080/v1/fs/cat?path=/proc/schedule/queue&max_bytes=256' | jq .` - `curl -sS 'http://127.0.0.1:8080/v1/fs/cat?path=/proc/lease/summary&max_bytes=128' | jq .` - `curl -sS 'http://127.0.0.1:8080/v1/fs/cat?path=/proc/lease/active&max_bytes=256' | jq .` - Policy approval (only if `/policy/rules` exists): - `curl -sS -X POST http://127.0.0.1:8080/v1/fs/echo -H "Authorization: Bearer ${HIVE_GATEWAY_REQUEST_AUTH_TOKEN}" -H 'Content-Type: application/json' -d '{"path":"/actions/queue","line":"{\"id\":\"approve-rest-1\",\"target\":\"/queen/ctl\",\"decision\":\"approve\"}"}'` - REST spawn (heartbeat) through the gateway: - `curl -sS -X POST http://127.0.0.1:8080/v1/fs/echo -H "Authorization: Bearer ${HIVE_GATEWAY_REQUEST_AUTH_TOKEN}" -H 'Content-Type: application/json' -d '{"path":"/queen/ctl","line":"{\"spawn\":\"heartbeat\",\"ticks\":120,\"budget\":{\"ttl_s\":300,\"ops\":500}}"}'` - `curl -sS 'http://127.0.0.1:8080/v1/fs/ls?path=/worker' | jq .` - Host publishers over REST: - `./bin/gpu-bridge-host --publish --rest-url http://127.0.0.1:8080 --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN" --interval-ms 1000` - `./bin/host-sidecar-bridge --rest-url http://127.0.0.1:8080 --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN" --watch --provider systemd --provider nvidia` - Validate: `curl -sS 'http://127.0.0.1:8080/v1/fs/ls?path=/gpu' | jq .` and `curl -sS 'http://127.0.0.1:8080/v1/fs/ls?path=/host' | jq .` - `coh` REST path coverage (queen role via gateway): - `./bin/coh gpu --rest-url http://127.0.0.1:8080 --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN" list` - `./bin/coh gpu --rest-url http://127.0.0.1:8080 --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN" lease --gpu GPU-0 --mem-mb 2048 --streams 1 --ttl-s 120` - `./bin/coh run --rest-url http://127.0.0.1:8080 --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN" --gpu GPU-0 -- echo ok` - `./bin/coh telemetry --rest-url http://127.0.0.1:8080 --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN" pull --out ./out/telemetry-rest` - The following PEFT commands are the same file-registry smoke as the TCP lane. The placeholder `demo/peft_adapter` and pointer activation do not prove native training, evaluation or serving. - `./bin/coh peft --rest-url http://127.0.0.1:8080 --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN" export --job job_0001 --out ./out/peft_export_rest` (skip if no export job is seeded) - `./bin/coh peft --rest-url http://127.0.0.1:8080 --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN" import --publish --model demo-model --from demo/peft_adapter --job job_0001 --export ./out/peft_export_rest --registry ./out/peft_registry_rest` - `./bin/coh peft --rest-url http://127.0.0.1:8080 --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN" activate --model demo-model --registry ./out/peft_registry_rest` - `./bin/coh peft --rest-url http://127.0.0.1:8080 --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN" rollback --registry ./out/peft_registry_rest` - REST mount exclusivity: - `./bin/coh mount --rest-url http://127.0.0.1:8080 --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN" --at /tmp/coh-mount-rest` - In a second shell: `./bin/coh mount --rest-url http://127.0.0.1:8080 --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN" --at /tmp/coh-mount-rest-2` → must fail with an exclusive lock error. - Read/write smoke (supported MIME types only): - `cat /tmp/coh-mount-rest/proc/lifecycle/state` (must be non-empty) - `head -n 5 /tmp/coh-mount-rest/log/queen.log` (must be non-empty) - `DEV=tp-mount-xfer-1; printf '{"new":"segment","mime":"text/plain"}\n' >> "/tmp/coh-mount-rest/queen/telemetry/${DEV}/ctl"` - `printf 'hello-from-test-plan ts_ms=%s\n' "$(date +%s000)" >> "/tmp/coh-mount-rest/queen/telemetry/${DEV}/seg/seg-000001"` - `cat "/tmp/coh-mount-rest/queen/telemetry/${DEV}/latest"` (expects `seg-000001`) - `cas-tool` REST upload: - Use the separately named exact-eight-chunk bundle prepared in Conditional B; never use a truncated prefix of `trace_v0.trace`. - Run: ```bash "${CAS_TOOL[@]}" upload \ --bundle \ --rest-url http://127.0.0.1:8080 \ --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN" ``` - Require zero retry. A local over-limit refusal must occur before a REST request; a target `buffer-full` refusal for an eligible bundle remains a valid target-capacity failure, not a host-preflight failure. - `cohsh` REST CLI: - `./bin/cohsh --transport rest --rest-url http://127.0.0.1:8080 --rest-auth-token "$HIVE_GATEWAY_REQUEST_AUTH_TOKEN"` - `attach queen` - `cat /proc/schedule/summary` - `cat /proc/lease/summary` - SwarmUI via gateway (REST transport enabled by default): - `SWARMUI_TRANSPORT=rest SWARMUI_REST_URL=http://127.0.0.1:8080 ./bin/swarmui` - Confirm Live Hive view renders telemetry and the console panel accepts standard verbs. - Open DevTools and run `window.__SWARMUI_HIVE_DEBUG.getMetrics()`; confirm renders advance and the UI stays responsive. - Deterministic replay via cohsh (no QEMU needed): - Source tree: `./bin/cohsh --transport mock --replay-trace ./tests/fixtures/traces/trace_v0.trace` - Release bundle: `./bin/cohsh --transport mock --replay-trace ./traces/trace_v0.trace` ### UI Presentation Layer — SwarmUI (Playwright) **Additive UI-only layer.** Playwright tests **DO NOT** assert control-plane correctness and **MUST NOT** introduce new verbs, protocols, or semantics. They validate presentation (rendering, wiring, and transcript parity) using deterministic replay fixtures. #### 1) Scope - Covers: SwarmUI launch, Spectrum shell control wiring, canvas presence, deterministic replay rendering, mint-ticket UI wiring, and embedded `>coh` console transcript output. - Excludes: control-plane logic, NineDoor semantics, ticket validation correctness, and any non-UI behavior already covered by `.coh` scripts or regression pack. #### 2) Modes - **Replay mode (required, gating):** UI is driven from trace/snapshot fixtures and deterministic transcript outputs. - **Native mode (required for desktop integration completion):** the real app and Rust bridge use an authenticated gateway and accepted QEMU image. This is separate from Playwright fixture claims and from target/release qualification. #### 3) Test Categories - Launch + smoke (UI loads and renders key panels). - Replay visual regression (banner/shell screenshot baseline). - Spectrum shell controls (buttons, text fields, pickers mount and remain wired to existing IDs/flows). - Interactive `>coh` prompt (type commands, assert transcript lines). - Mint ticket flow (UI-only assertion that the host-returned token is surfaced back into the session field). - Live Hive UX (labels, role colors, and dot selection wiring). - Native command argument parity (`detailAgent`, `snapshotKey`), selected detail refresh while the canvas is offscreen, and changing/absent telemetry when only an overlay is available. - Structured Worker state (declaration, lifecycle, receipt, artifact, and proof render independently; absent axes render as unknown). - Opaque Worker identity (a role-looking id prefix never supplies a role, READY state, receipt, artifact, or proof). - Live Hive performance harness (bounded render cadence and backlog checks). - Failure UI (auth error, disconnected state) as UI-only states. #### 4) Determinism Rules - Replay-first: all UI assertions are driven from replay fixtures. - Version-1 model-only snapshots migrate without inferring structured READY or target proof from their legacy role/id strings. - Avoid unbounded timing-based assertions; use explicit replay fixtures and the Live Hive metrics harness for bounded render/backlog checks. - Transcript-based assertions only (match `OK`, `ERR`, `END` and static help lines). - Shell assertions use stable roles, labels and replay contracts. Layout IDs may evolve with the workbench; existing backend transcript and authority semantics remain canonical. #### 5) CI Positioning - Runs **after** `.coh` scripts and the regression pack. - **Blocking for the `ui` claim:** replay-mode UI tests (snapshot + transcript parity + Live Hive UX + performance harness). - **Blocking for the native desktop claim:** identity-bound source and installed-candidate native walkthroughs; no injected bridge state. See [SwarmUI native acceptance](SWARMUI.md#native-acceptance-walkthrough). **Focused native desktop gate:** `scripts/ci/swarmui_native_e2e.sh` launches an exact app/tool pair and records bounded command-boundary metadata. The operator uses the visible native UI for credential refusal, gateway connection, canonical namespace/boot reads, Live Hive pressure and Worker evidence, reviewed host-tool execution, disconnect/reconnect, verified reference and failed-canary playback, offline refusal and clean shutdown. The final verifier binds exact app/tool, registry, source and accepted QEMU image hashes plus original screenshots. A second lane uses the installed candidate; Linux AArch64 additionally runs on the reference Jetson desktop. Retained CUDA/LoRA verification is historical and does not substitute for new GPU or physical Pi execution. `stage_swarmui.py` supplies exact offline browser companions for the existing `SWARMUI_RELEASE_DIR` gate. Source and candidate run the same presentation and performance assertions. `swarmui_showcase_capture.sh` accepts only native PASS records and unchanged screenshot hashes. Its deterministic frame walkthrough cannot establish live readiness or new target evidence. **Playwright commands (macOS ARM64):** - `cd tools/swarmui-ui-tests` - `npm ci` - `npx playwright install webkit chromium` - Source UI (default harness target): `npm test` - Explicit source UI override: `SWARMUI_UI_ROOT=../../apps/swarmui/frontend npm test` - Release bundle verification: `SWARMUI_RELEASE_DIR=../releases/ npm test` - Update snapshots only when UI changes are intended: `npm run test:update` **Notes** - The Playwright harness targets the **source SwarmUI frontend** by default so UI regressions are measured against the current shell, not a stale release bundle. - Set `SWARMUI_RELEASE_DIR` to verify a packaged release bundle; set `SWARMUI_UI_ROOT` only when overriding the default source path. - The harness injects a deterministic Tauri `invoke` mock for UI-only replay and transcript assertions; it does not exercise control-plane behavior. - The current shell uses a vendored Spectrum Web Components layer for operator controls; Playwright interacts with the effective editable/button controls while keeping the canonical SwarmUI element IDs stable. - The Live Hive performance harness reads `window.__SWARMUI_HIVE_DEBUG.getMetrics()`; `pending` must stay ≤ `swarmui.hive.pending_event_cap` and `renders` should advance without UI stalls under replay fixtures. - Browser binaries are installed into the user Playwright cache (not committed). - Snapshot coverage runs against the current browser matrix: `webkit-desktop` (baseline shell), `webkit-narrow` (responsive shell and scheduler), and `chromium-tablet` (interaction parity without snapshot gating). ### Automated Stage 04 — REST multiplexer regression Stage 04 is self-contained for local QEMU. Without an external gateway URL it starts the exact QEMU artifact and a local gateway with stage-local request authentication and free loopback ports. `TP_STAGE4_GATEWAY_BIND` and `TP_STAGE4_QEMU_TCP_PORT` override local placement. An existing gateway requires its explicit request-auth environment and declared deadlines; Pi always uses the external path bound to its supplied boot/image evidence. Python probes use the same canonical Python 3.11+ selection as Stage 01. - In a staged QEMU run, `scripts/ci/test_plan_stage_04_rest_multiplexer.sh` verifies and reuses the Stage 03 default artifact, then starts a fresh boot. Standalone Stage 04 may build its own canonical artifact. Set `COHESIX_GATEWAY_URL` or equivalent to target an existing gateway. - QEMU clients select the retained artifact's `cohsh` and `coh` policies by default, including readiness and REST FUSE checks. Explicit `COHSH_POLICY` and `COH_POLICY` selections remain subject to the clients' normal hash validation. The runner restores both inherited selectors before finalizing evidence; the checkout's restored default profile cannot replace the tested artifact's policy. - `COHESIX_GATEWAY_URL=http://: HIVE_GATEWAY_REQUEST_AUTH_TOKEN= scripts/cohsh/REST_regression_batch.sh` - Pass the token through the inherited environment. The runner records a redacted command and must not place the token value in retained logs. - Stage 04 composes one filesystem-operation response window from the gateway contract: `5,000 ms` bounded broker-queue admission plus `max(control_response_ms, telemetry_response_ms)` plus `5,000 ms` HTTP response-delivery grace. The canonical local gateway uses `120,000/120,000 ms`, so the canonical client window is `130,000 ms`. Metadata requests, name resolution, connection establishment, and response body transfer retain their separate short bounds. Because the HTTP library carries earlier send deadlines into response receipt, the filesystem agent applies the composed window to request send, request-body send, and response-header receipt; request and response byte bounds do not change. - Local runs resolve `TP_STAGE4_GATEWAY_CONTROL_RESPONSE_TIMEOUT_MS` and `TP_STAGE4_GATEWAY_TELEMETRY_RESPONSE_TIMEOUT_MS`, falling back to the matching `HIVE_GATEWAY_BROKER_*_RESPONSE_TIMEOUT_MS` service variables, and pass the resolved values explicitly to `hive-gateway`. Existing-gateway runs must declare both values; Stage 04 does not infer external process configuration. `TP_STAGE4_REST_CLIENT_TIMEOUT_MS`, with `COHSH_REST_RESPONSE_TIMEOUT_MS` as its service fallback, may select a larger client window but fails before target work when it is smaller than the composition. The Stage 04 runner accepts each declared broker response deadline only in `5,000..=1,200,000 ms` and the client window only from the composed minimum through `1,210,000 ms`. Conflicting declarations fail rather than choosing one. - The resolved client window applies identically to readiness, primary and pooled `cohsh` transports, concurrent core/parity batches, and the Python `RestBackend` smoke. This changes no batch concurrency, pool size, request, retry, script, or target timeout. The Stage 04 summary retains `gateway_timeout_declaration`, `gateway_broker_queue_wait_limit_ms`, `gateway_broker_control_response_timeout_ms`, `gateway_broker_telemetry_response_timeout_ms`, `rest_response_delivery_grace_ms`, and `cohsh_rest_response_timeout_ms`. - Before it exports resolved endpoint, deadline, auth, or FUSE helper values, Stage 04 snapshots the exact inherited presence and value of its complete runner-owned environment set. It restores that snapshot before normal `tp_stage_complete` and before cleanup or failure delegates to `tp_stage_exit_trap`; an inherited value must be restored exactly and a runner-created value must be unset. Focused regression `test_stage4_restores_runner_owned_environment_before_final_context` covers both finalization paths and the existing timeout composition. Runner-local child configuration must not become an input-context change. The local delegated issuer uses a secret-classified environment name, so its value is excluded from evidence selectors and redacted from retained output. Restore the inherited issuer reference, delegated ticket and private issuer variable alongside the endpoint settings before either finalization path. - Stage 04 runs two REST batches: - A concurrent "core" batch (boot, ingest, and root reachability): `scripts/cohsh/boot_v0.coh`, `scripts/cohsh/observe_watch.coh`, `scripts/cohsh/root_cut_basic.coh`. This is also the default selection for `REST_regression_batch.sh`. `session_pool.coh` remains a TCP check because REST batches only host ticket results. `host_absent.coh` remains under TCP because its entry-count assertion uses a console-specific ACK detail. - A strict "parity" batch (control-plane smoke): `scripts/cohsh/rest_control_plane_smoke.coh`. - Note: `scripts/cohsh/busy_backpressure.coh` and `scripts/cohsh/policy_gate.coh` remain covered by the TCP/QEMU regression matrix (Stage 03), where console-parser semantics are validated directly. - Stage 04 also runs a Python REST smoke (`tools/cohesix-py` `RestBackend`) that performs `LS /` and reads `/proc/lifecycle/state` against the same gateway. - Logs: - Scripted Stage 04 writes REST batch logs under the stage state dir (for example `out/test-plan//rest-regression-logs/`). - Manual runs of `scripts/cohsh/REST_regression_batch.sh` default to `out/regression-logs//