We re-marked our own marking.
A judge from a different family agreed with the other two at 0.954 [0.896, 1.000] (panel-sampled draws with a reference). Vol 001 passes the test this issue set it.
The panel, run again on a second model. Three judges on Luna: 0.865 [0.804, 0.924] (sampled draws).
Two decision models meet our agreement bar on the public half. So would a judge that said PASS to everything, almost.
Two judges, one table. Jev and Laya disagreed on 8 compared draws and produced identical totals. We checked.
ONE CHOICE THE PRE-REGISTRATION DID NOT SPELL OUT, stated here and in the issue: it asks for
agreement "against the majority of J-B and J-C". Two judges have a majority only when they agree.
Draws where J-B and J-C disagree have no reference label; they are COUNTED AND EXCLUDED, never
resolved by a third judge, and the count is printed beside every figure that uses the reference.
scripts/cal002_figures.py, lines 11–14 · unedited
An agent must never be told about a precondition it could verify.
the house ruleA Calibration is one dated issue of a measurement programme. It is not a leaderboard, and the thing it is mostly about is not the models — it is the instruments people use to rank them.
This issue is about one instrument: the judges. Every verdict in Vol 001 came from a model grading another model’s answers. Before measuring anything new, Vol 002 asked whether that judging survives three tests: a different judge, a different model, and a different kind of judge altogether.
figures.json by figures.py render, under
cal002, with the SHA-256 of the files it came from. Thresholds, dates and
settings are typed.Due Saturday 12 September 2026 and missed. The 1 October register row re-dated it to Saturday 10 October. It was issued on Saturday 3 October instead: the runs were in on Friday, the draft was reviewed twice, and it was signed and released the next morning. Saturday is a publishing day, but this one belongs to The Log; the issue is off-cycle all the same, and this note is where that is said.
The judging passed the tests it was set. A judge from a different family re-graded Nemotron’s answers and agreed with the other two judges at 0.954 [0.896, 1.000] (panel-sampled draws with a reference), above the pre-registered floor of 0.70. The three-judge panel, run for the first time on Luna, agreed at 0.865 [0.804, 0.924] (sampled draws). Vol 001’s per-model verdicts pass the test the pre-registration set for them. One caution travels with that: the reference includes the very grader whose verdicts Vol 001 published, and a direct comparison with that grader, reported below as exploratory, is weaker.
Both decision models meet the agreement bar on the public half, and neither is seated by this issue. Hosted Jev and Laya each agreed with the reference at 0.745 [0.596, 0.868] (public-half draws with a reference) on the public half. On the same draws, a judge that answered PASS every time would score 0.700 (public-half draws with a reference), and Jev failed only 5 of the 19 draws the reference failed. Both of those are exploratory. Whether to seat either model is the investigator’s decision, and it is not taken here.
Three questions, pre-registered and signed before any run. Q1: does a re-grade by google/gemini-3.7-flash agree with the other two judges on the draws the panel sampled in Vol 001? Floor: an AC1 of at least 0.70. Q2: does the same three-judge panel replicate on Luna? No floor; a replication is reported whatever it is. Q3: can a decision model — hosted Jev, or the open Laya — hold one seat on the panel, under the seven rules published in The Bench Vol 003?
The reference. Agreement is measured against the majority of J-B (x-ai/grok-4.6) and J-C (bytedance-seed/seed-2-1-turbo). Two judges have a majority only when they agree, so draws where they disagree are excluded and counted. J-A is the same Gemini model as the Q1 re-grade, so its figure is printed separately and labelled self-including. J-C is also the tier-1 grader whose verdicts Vol 001 published: the reference contains the grader being validated. Rule six of The Bench Vol 003 named this problem; the pre-registration did not end it, and this issue does not either.
The statistic. Gwet’s AC1, the coefficient this programme uses because it is not dragged down when one verdict dominates, with a percentile bootstrap interval — computed by the same functions panel_grade.py uses. The intervals resample individual draws, not items, so they are narrower than they should be; read them as a guide to precision, not as a probability.
All 616 of Nemotron’s graded draws were re-graded by Gemini from the stored responses; generation was not repeated. On the panel sample, the re-grade cleared the floor. The self-including figure, against the majority of all three judges, is 0.946 [0.892, 0.987] (panel-sampled draws).
Exploratory, and weaker. Against the Vol 001 tier-1 verdicts directly, the re-grade’s AC1 is 0.785 [0.689, 0.880] (panel-sampled draws) on the panel sample and 0.858 [0.815, 0.898] (all nemo_v1 draws) over every draw. The excluded draws are where Grok disputed the tier-1 grader; on them the re-grade sided with Grok 13 times and with the tier-1 grader 4 times.
Same three judges, same sampling, same procedure as Vol 001. The majority of three overruled the tier-1 grader on 7 of the sampled draws. Fleiss’ kappa, pre-registered beside AC1, is 0.807 [0.710, 0.887] (sampled draws).
Fail rate on the sample: J-A 29.4% (sampled draws)
J-B 30.8% (sampled draws)
J-C 26.6% (sampled draws)
OpenRouter lists no data policy for Jev 1.13 (checked 2 October 2026), so under rule five it saw the public half only. On that half it meets rule two on the point estimate. So does Laya, with the identical figure 0.745 [0.596, 0.868] (public-half draws with a reference).
Laya runs on our own machine, so it also graded the held-out half, which Jev never saw. There it agreed at 0.488 [0.198, 0.704] (held-out half, draws with a reference) on 51 draws (5 excluded), and over both halves at 0.651 [0.519, 0.768] (both halves, draws with a reference) on 131 draws (12 excluded). The pre-registration did not say which Laya figure rule two applies to; all three are printed.
Exploratory: what the bar misses. On the public draws a judge that answered PASS every time would score 0.700 (public-half draws with a reference) — right at the bar — because most answers pass. Jev failed 5 of the 19 draws the reference failed; its fail rate on the compared draws was 7.5% (public-half draws with a reference) against the reference’s 23.8% (public-half draws with a reference). Cohen’s kappa on the same table is 0.323 (public-half draws with a reference). AC1 is the right coefficient for this programme, and on a sample this lopsided it still cannot tell a cautious judge from a lenient one.
Two judges, one table. Jev and Laya produced the same table: the same number of draws in each of its four cells. They did not agree draw for draw. Of the 80 compared draws they differed on 8, and the swaps cancelled exactly; of all 87 public panel draws they differed on 9. Identical tables give identical intervals, because the intervals are now resampled in a fixed order.
What Laya saw. Laya’s English checkpoint reads 512 tokens. The judging script cut 12 grading states to 1,800 characters before Laya saw them, 7 of them on held-out items; 3 of those would have fitted uncut. Laya itself dropped tokens from 10 states. In all, 13 states lost text before a verdict. The checkpoint warned that its shipped temperatures were invalid, so its probabilities are not used as confidence; only the higher-probability option is used, as the verdict.
Rule four: grade the same draws five times at normal settings and publish how often the verdict changes, with the seated judges measured the same way. Draws whose five verdicts were not all the same:
Jev 2 of 50
Laya 0 of 50
J-A 3 of 50
J-B 3 of 50
J-C 3 of 49
Counts this small cannot be ranked against each other. Four further facts belong beside them. Laya returned identical probabilities on all five gradings of 50 draws: on this machine it is deterministic, so its count says nothing about stability. Jev was measured on fifty public draws, not the fifty the others shared. For the seated judges the first of the five verdicts is the one already on disk; for J-C that is the original tier-1 verdict. And one of J-C’s re-grades hit its output limit before giving a verdict, so that draw lacks five verdicts and J-C’s count covers one draw fewer.
Two questions in the unsigned first draft had answers before anyone signed anything. They are reported as description, not as findings.
How much of the bank does work. Of 63 items measured on both completed candidates, 20 are passed by both on every draw and 4 are failed by both on every draw. 23 separate the two candidates by a fail-rate gap of 0.40 or more, 14 of them in the public half. The first draft printed different counts, which its own definitions do not reproduce; the registry carries both.
Abstention. 4 items score an over-cautious refusal as a failure. Luna failed 7 of 40 draws on them, 6 of them on a single item whose premise is defective, and 1 of 30 on the other three. Nemotron failed 0. Their ground truth has not been verified by a person, so none of them ran in this issue.
nemo_v1 enter any figure. 17 of those responses were re-graded from the 4,000-character store where the original grader had seen up to 6,000.01_arithmetic_038 B1, 04_false_premise_123 B3, 06_sycophancy_063 B0, 12_impossible_specifics_029 A1, 14_provenance_003 B2, 14_provenance_040 A1, A4 and B3. The further public draw on which they differ has no reference.Every figure is registered under cal002.* in figures.json with the SHA-256 of the run files it was computed from. The run files are served with the others at /data/runs/, redacted on held-out items as the 11 September row in Corrections + changes describes; the held-out probes themselves are never published.
Cost. The paid runs were made on a new key with a hard limit of $25 set on the key itself. OpenRouter charged $3.510 in all, read once after the last paid run. Laya ran on our own machine and cost nothing.
bank_v1 with Luna and Nemotron.Lawrence Ho, Chief Human Investigator, Evaluation Administrator + Instrument Maker, signed the pre-registration and signs this issue, and is its sole authority. The prose was drafted by an AI agent, reviewed adversarially by a second, independent agent, and edited and signed by him. Juancho Lee, the panel registrar, is deterministic code that seats the judges and computes agreement and issues no verdict. Jayden Lee, the AI reporter, writes The Log and nothing here. Budday Budderson Studio LLC publishes the series and grades nothing.