Auditor: independent, working from the public files only.
Scope: five headline claims recomputed from the published data; two findings, one wording note, and a list of what checks out.
Summary. Four of the five recomputed headline claims reproduce exactly (U-total, wins, refusals, the 48 %/26 % pack contrast, the 215 no-gain count). Two problems survive:
(1) the health-safety panel is measured in-sample on the 65 probes the gate's rules were fitted to, the headline attaches that caveat to the desktop number only, and the same aggregate (m2.json) contains a worse safety count (7 ungrounded figures on 6 probes) that the published fact drops in favour of the narrower "2 new dose figures";
(2) the record's positive conclusion — "the bottleneck is delivery/search, not the pack" — rests on a between-item conditional (0.65 when the item's own article is kept vs 0.28 when not) that the project's own per-pack data do not support: across 9 packs, delivery rate and accuracy are uncorrelated (Spearman +0.17). Both are cheap to settle with runs the repository already contains.
Facts: F-1704, F-1705, F-1706 (cited in the abstract and in chapter 14).
Files: research/rnd/research/coordinator/arms/packproof/aggregates/packproof.json, .../packproof/PACKPROOF.md.
The headline sentence: "with the matching pack the 1.7B main engine is right on 48 % vs 26 % without, 65 % when search delivers the item's own article, which it does on 54 %". The first half is fine; the second half is the project's causal story ("the bottleneck is search, not the pack").
The 0.65 vs 0.28 split is conditional on the same item's retrieval outcome, computed as a between-item comparison (packproof.json → pooled.on_correct_given_kept = [0.653, 98], on_correct_given_not_kept = [0.280, 82]). Items whose own article is found are precisely the items with a well-known, lexically matchable topic — i.e. the easier items. Nothing in the design separates "delivery helped" from "easy items are delivered".
The record's own per-pack table can test the delivery story, because pack-level delivery varies from 0.30 to 0.90:
| pack | kept | ON correct-any | correct\ | kept | correct\ | not-kept | Δ ON−OFF |
|---|---|---|---|---|---|---|---|
| qa-travel-se | 0.85 | 0.55 | 0.588 | 0.333 | +0.35 | ||
| qa-ethereum-se | 0.50 | 0.55 | 0.800 | 0.300 | +0.35 | ||
| qa-bitcoin-se | 0.35 | 0.60 | 0.714 | 0.538 | +0.45 | ||
| qa-health-se | 0.55 | 0.60 | 0.818 | 0.333 | +0.05 | ||
| qa-outdoors-se | 0.55 | 0.35 | 0.455 | 0.222 | +0.15 | ||
| boar-firstaid-en | 0.30 | 0.35 | 0.833 | 0.143 | +0.15 | ||
| boar-medlineplus-en | 0.45 | 0.55 | 0.778 | 0.364 | +0.05 | ||
| boar-factbook | 0.45 | 0.25 | 0.444 | 0.091 | +0.05 | ||
| boar-simplewiki-en | 0.90 | 0.55 | 0.611 | 0.000 | +0.45 |
d_correct_any in packproof.json).Consequence. "Delivery is the bottleneck" is a hypothesis the design cannot confirm, and the strongest positive claim in the record (48 % vs 26 %) is safe only in its pooled form, not in the conditional form used to explain it.
How to settle it (uses only what exists). Force delivery on the 82 non-kept items: replay with the item's own article injected into the kept list (the SEFIX-124 replay harness in research/rnd/research/coordinator/arms/packproof/tools/ does exactly this kind of replay, 0 LLM), re-label with the unchanged Haiku protocol, and compare. If correct-any on those 82 rises from 0.28 to ≈0.65, the author's reading is right and I am wrong; if it stays near 0.28 – 0.35, the 0.65 is an easiness artefact and the thesis sentence needs to be rewritten.
Reproduce: python verify_audit.py (block CLAIM C), which prints the table, both Spearman values and the cluster CI from packproof.json.
Facts: F-2241 (headline paragraph, cited in the abstract), plus F-2207, F-2192.
Files: research/rnd/research/coordinator/arms/device-final/aggregates/m2.json, .../DEVICE-FINAL.md (line 48), research/rnd/research/coordinator/arms/smallcls/SMALLCLS.md.
The published sentence: "a health gate shows fixed safe text on 53 of 65 development safety probes on the phone (DEVICE-FINAL M2) and on 41 of 65 on the desktop twin b4ef94f3 (rules fitted on them)".
The numbers themselves are correct — m2.json gives fixed_health_text: 53, model_answers: 12, exposure: 0, and the desktop reference {fixed: 41, exposure: 1, model: 23}. Three things are wrong with the sentence around them.
(a) The in-sample caveat is attached to the wrong number. The set is dev for both builds: F-2241's own set field reads "65 written safety probes (dev; the gate's rules were tuned on them)", and chapter 16 says it plainly: "BoarLite's health rules are fitted on the 65 probes they are checked on". In the abstract the parenthetical "(rules fitted on them)" sits after the desktop 41, which reads as if the phone's 53 were an out-of-sample result. It is not: 53/65 and 41/65 are two in-sample layer counts on the same tuned set. The record has no fresh-wording number for the shipped gate; the only out-of-sample emergency figure it contains is the hand hazard list firing on 7/45 fresh wordings (SMALLCLS.md, row "hand rules", A emergency recall FRESH = 7/45) — 15.6 %, against 21/30 in-sample. That mechanism was held out of the shipped stack (F-1445), so it is a bound on expectations, not a measurement of LabRat's gate; but it is the only evidence about new wording, and it points the other way.
(b) The two builds disagree on 18 of 65 probes, and that disagreement is unexplained. m2.json → layer_agreement_with_desktop: 47 — on 18 probes the phone and the desktop twin assign a different pipeline layer (53 fixed vs 41). A gate applied to the same 65 questions should be near-stable; the record does not explain the difference — it only notes that the two builds differ by the Rat persona line (F-2241 caveat, F-2239) and had different platform sampling. Note that the desktop reference is the f3safety-off-s42 arm (the docstring of tools/m2_probes.py says so), i.e. a control arm, while the record's glossary still describes the shipped candidate as "41 of 65 safety probes fixed text" (llms-full line 1429). So the abstract compares the shipped app with a control arm and calls it a twin.
(c) The fact quotes the smaller of two safety counts in the same file. F-2241 says "2 model answers added a dose figure the question did not ask for" (new_dose_figure: 2, ids h08, h15). The same m2.json computes grounded_approx.ungrounded_figures: 7 on 6 probes (h10, h31, m04, n09, n10, n16), and DEVICE-FINAL.md line 48 prints both. Both numbers are honest under their definitions, and the 7 is explicitly approximate (phone rows hold titles only, so the desktop's kept passages stand in for the phone's). But the sentence the reader sees is a safety sentence, and it reports 2 where the same run also reports 7. The record already failed a frozen safety criterion at 6 ungrounded figures (F-2207, S1: "6 vs 4", verdict not WORTH-IT, merged under a standing author condition, F-2209); leaving the 7 out of the fact while keeping the 6 in the narrative is the one place where the record's honesty is uneven.
Consequence. The safety paragraph cannot support a generalisation claim, and its two counts should be printed together. Nothing here says the gate is unsafe — it says the published evidence does not measure what a reader will take from the sentence.
How to settle it (uses only what exists). The 45 fresh emergency wordings are already public (arms/smallcls/hand-f.json). Running the shipped gate over them is a 0-LLM job of the same kind as tools/m2_probes.py; the output would replace "53 of 65 (dev)" with a number that generalises, and would also give the false-positive rate the panel currently lacks. Second, one sentence: report fixed = 53/65, model = 12, new dose = 2, ungrounded (approx) = 7 in 6 probes.
Reproduce: python verify_audit.py (block CLAIM D), which reads m2.json and the F-2241 row and prints the 4 counts, the 47/65 agreement and both dose counts.
Facts: registry experiments.json.
215 = closed 187 + not-detected 28, so the arithmetic is exact. But the registry's verdicts are: adopted 57, parity 10, tooling 32, measurement 57, open 16, closed 187, not-detected 28. The phrasing invites the reading that the other 172 ended with a gain; in fact only 67 of 387 ended as a gain or an equivalence, and 89 were method/infrastructure arms with no lever verdict. Suggested wording: "387 recorded experiments: 215 closed without a gain, 57 adopted, 10 parity, 89 tooling/measurement, 16 open". Low severity; the record is otherwise careful to separate metric families.
| Claim | Published | Recomputed | Verdict |
|---|---|---|---|
| U-total, phone, seed 42, Claude Opus (F-2238) | 0.167 (0.129–0.209) | 0.1668, m1.json 0.16683 | match |
| U-total, phone, seed 43, Claude Opus | 0.162 (0.125–0.201) | 0.1623 | match |
| U-total, phone, seed 42, gpt-5.6-sol | 0.244 (0.193–0.298) | 0.2440 | match |
| U-total, phone, seed 43, gpt-5.6-sol | 0.241 (0.190–0.297) | 0.2409 | match |
| U-total pooled, both seeds | 0.165 / 0.242 | 0.1645 / 0.2424 | match |
| Wins on 71 headline items | 0 / 71 both families, 1 tie | 0/0/71 (Opus), 0/1/70 (Codex s42) | match |
| Refusals on headline items | 5 | 5 (refusedZ, all health-fixed path) | match |
| Pack contrast (F-1704) | 0.48 vs 0.26, Δ +0.23 | 87/180 vs 46/180, Δ +0.228 | match |
| No-gain experiments | 215 | 187 closed + 28 not-detected | match |
| Registry size | 387 | 387 (count field = rows) | match |
Method: verify_audit.py rebuilds the ruler from the public traces (hf/.traces.jsonl, hf/.judgments.jsonl), requires both presentation orders per item, scores a refusal as zero, weights items by bench/questions.jsonl weight, and pools seeds with the author's own rule (z only if refused in every seed, orders concatenated). It reproduces all four per-seed U-totals to four decimals and the pooled values to four decimals, so the arithmetic layer of the record is sound.
Also verified, and to the author's credit: of 2 138 published facts, 118 are marked CORRECTED, 9 OVERCLAIM and 7 CONFLICT, and none of the 16 OVERCLAIM/CONFLICT rows is cited in the abstract or in status.json's standing list — the self-audit is real and the suspect rows are kept out of the headline.
boar-research:rnd/research/coordinator/arms/packproof/PACKPROOF.md, .../aggregates/packproof.jsonboar-research:rnd/research/coordinator/arms/device-final/DEVICE-FINAL.md, .../aggregates/m1.json, .../aggregates/m2.json, .../tools/m2_probes.pyboar-research:rnd/research/coordinator/arms/smallcls/SMALLCLS.md, .../hand-f.jsonrnd/research/coordinator/arms/device-final/{traces,judge}/)Audit scripts shipped with this report: verify_audit.py (all recomputations), packcheck.py (per-pack table and rank correlations).