The Calibration Vol 003 · ISSUED SAT20261003

Protocol
STAMP

The Calibration Vol 003 ISSUED SAT20261003

We asked a person.
On the same draws, our three judges agree with each other at 0.875 [0.789, 0.950] (the same labelled draws). With the person, at 0.685 [0.515, 0.819] (labelled draws) at best.

A judge that never fails anything would have scored higher. Answering PASS every time: 0.871 (labelled draws). That is what agreement means when most answers pass.

The decision models agreed most and caught least. Jev caught 3 of the person’s 10 failures; Laya 0.

A Budday Budderson Production Houston, Texas

From the harness

Shows the 87 public-half draws of the panel_luna_v1 sample, one at a time, in a fixed random order
(seed 20261003): the probe, the item's own pass criterion and the response. It never sends a judge's
verdict, an item's history or any rate to the page - the labels are blind by construction, because
the data needed to unblind them is not in what the page receives.

scripts/label_server.py, lines 7–10 · unedited

An agent must never be told about a precondition it could verify.

the house rule

About this publication

What a Calibration is

A Calibration is one dated issue of a measurement programme. It is not a leaderboard, and the thing it is mostly about is not the models — it is the instruments people use to rank them.

Vol 001 and Vol 002 asked whether the judges agree with each other. They do. This issue asks the question both of them left open: whether the judges agree with a person.

Signed before the first label. The questions were pre-registered and signed by Lawrence Ho at 12:00 CDT on Saturday 3 October 2026. The labelling page refused to save a label until the signature was on file; the first label was saved at 12:01.

Blind on the page. The labelling page showed the probe, the pass criterion and the response, and no verdict, item name or rate. What the labeller already knew is stated under One person.

Every computed number is cited. Each figure and count was substituted from figures.json under cal003, with the SHA-256 of the files it came from. Figures not asked by the pre-registration are labelled exploratory.

An issued Calibration is never edited. A correction is issued under a new date and this issue stands.

Issued the same day as Vol 002, off-cycle: this Saturday belongs to The Log, which follows this issue tonight. The next Calibration slot is Saturday 10 October.

The Calibration Vol 003 · Issued Sat 2026-10-03

The argument
Bottom line
What was asked, and how
Findings
01The judges agree with each other more than with a person
02The judges fail more than the person does
03Neither decision model meets rule three
The evidence
One person
Choices and departures, stated
Standing limitations
Who makes this

Bottom line

On these draws, our judges agree with each other more than any of them agrees with a person. The three seated judges agreed with one another at 0.875 [0.789, 0.950] (the same labelled draws), and with one person’s blind labels at 0.685 [0.515, 0.819] (labelled draws) (Gemini), 0.659 [0.479, 0.798] (labelled draws) (the tier-1 grader behind Vol 001) and 0.631 [0.453, 0.780] (labelled draws) (Grok). The intervals are wide and overlap, so this is a direction, not a measured gap.

Read every agreement figure here against one number. The person passed 77 of 87 draws. On a sample like that, a judge that answered PASS to everything would score 0.871 (labelled draws) — higher than every judge we use.

The judges fail more than the person does. The person failed 10 draws; each seated judge failed more than twice that many, most of them answers the person passed. They also missed some of the person’s failures. On Luna’s public half, then, the judges’ standard is stricter than this person’s. One labeller cannot say which standard is right.

The decision models agree with the person more because they fail almost nothing. Jev caught 3 of the person’s 10 failures and Laya 0. Neither meets rule three, so neither is eligible for a panel seat.

What was asked, and how

The labels. Lawrence Ho labelled all 87 public-half draws of the Luna panel sample from Vol 002, by each item’s own pass criterion, blind on the page to every judge: 77 PASS, 10 FAIL and 0 unsure, between 12:01 and 12:43 on 3 October, before the 18:00 freeze. A label could not be changed once saved.

The judges. Every judge’s verdict on these draws was already on disk from Vol 002: J-A google/gemini-3.7-flash, J-B x-ai/grok-4.6, J-C bytedance-seed/seed-2-1-turbo (the tier-1 grader), Jev typesafe/jev-1.13 and Laya convaiinnovations/laya. No model was called for this issue and nothing was spent.

The questions. Q1: each judge’s agreement with the person, as Gwet’s AC1 with a bootstrap interval. Q2: of the person’s FAILs, how many each judge caught; of the person’s PASSes, how many each judge failed. Q3: a decision model meets rule three if its agreement is no lower than the lowest seated judge’s and it catches at least half of the person’s FAILs.

Findings

01Pre-registered · exploratory context beside it

The judges agree with each other more than with a person

Agreement with the person (AC1), 87 draws
J-A Gemini 0.685 [0.515, 0.819] (labelled draws)
J-B Grok 0.631 [0.453, 0.780] (labelled draws)
J-C tier-1 0.659 [0.479, 0.798] (labelled draws)
Jev 0.846 [0.741, 0.935] (labelled draws)
Laya 0.779 [0.657, 0.885] (labelled draws)
J-B/J-C reference 0.684 [0.520, 0.826] (labelled draws) on 80 draws; 7 without a reference

Exploratory, on the same draws. The seated judges with one another: all three 0.875 [0.789, 0.950] (the same labelled draws); Gemini and Grok 0.905 [0.815, 0.981] (the same labelled draws); Gemini and the tier-1 grader 0.852 [0.737, 0.949] (the same labelled draws); Grok and the tier-1 grader 0.869 [0.763, 0.963] (the same labelled draws). A judge answering PASS to every draw would score 0.871 (labelled draws) against the person.

So two things are true at once. The judges agree with each other better than with the person, though the intervals overlap. And on a sample where the person passed most answers, a high agreement figure is easy to reach by passing everything; it is the failures that separate the judges.

No floor was pre-registered for this question. The comparisons above are description, not a test.

02Pre-registered

The judges fail more than the person does

Of the person’s 10 FAILs, caught · of the person’s 77 PASSes, failed
J-A 7 · 16
J-B 6 · 18
J-C 5 · 16
Jev 3 · 4
Laya 0 · 6
Reference, of 9 and 71: 5 · 14

Each seated judge failed more of the person’s PASSes than the person failed draws in total, and each also missed some of the person’s FAILs. In counts, the disagreement runs mostly one way: the judges fail answers the person passes. As shares of each pile, the misses on the person’s few FAILs are proportionally larger, so the judges are not simply a harsher copy of the person; they are stricter overall and also disagree about which answers fail.

What it would mean, on this sample. The failure rate Vol 001 published for Luna was set by the tier-1 grader. If this person’s reading is the standard, then on these sampled draws from Luna’s public half that grader fails more than a person would. The sample is not random: it was stratified on the tier-1 verdict and topped up with contested items. Vol 001’s other headline rates, chiefly Nemotron’s, have no human labels, so nothing here says how they would move.

03Pre-registered · answers Vol 002’s open question

Neither decision model meets rule three

The pre-registered test had two parts. On agreement, both pass: the lowest seated judge scored 0.631 (labelled draws), and Jev and Laya each scored higher — as would a judge that passes everything, at 0.871 (labelled draws). On failures, both fail: Jev caught 3 of 10 and Laya 0 of 10, where at least half was required.

The person’s FAILs fall on only 6 items, up to 4 on one, so the failure half of this test rests on a handful of items.

Consequence. Vol 002 left the seating of a decision model to the investigator. Neither model is eligible under the rule as signed, and neither is seated.

One person

A reference, not ground truth. One labeller, who built the bank and had seen some of these items before.
What he already knew. Earlier the same day he signed Vol 002, which lists by item and draw the eight draws where Jev and Laya differ; all eight of those draws are among the ones he labelled here. The page did not show which draw was which, but the probe text identifies an item to the person who built the bank, so the prior exposure is real.
The pace. All 87 labels took 42 minutes; the median time between labels was 11.000 seconds, and 10 were saved under five seconds apart. Most responses are short, but this is quick, and a reader should weigh it.

Choices and departures, stated

The analysis code was written after labelling. scripts/cal003_figures.py was written by the drafting agent in one pass once the labels were complete, and before any figure was computed. It implements the signed definitions; the exploratory figures were added after the independent review and are labelled as such.
Public half only. Held-out draws were not labelled, so every labelled probe and response can be published. The labels are served at /data/runs/ with this issue.
The reference needs two judges to agree. As in Vol 002, draws where J-B and J-C disagree have no reference and are counted.
The labeller saw the stored response. On 2 draws the stored response is shorter than the text the tier-1 grader originally graded.
Nothing was run. Every verdict compared here was recorded for Vol 002; their run files are already served.

Standing limitations

One labeller. A second, independent labeller is the next step.
One model, one half. Luna, public half, 87 draws from 35 items.
Intervals are optimistic. The bootstrap treats draws as independent; draws within an item are not.
Data terms for the seated judges are still unchecked, as Vol 002 recorded.

Who makes this

Lawrence Ho, Chief Human Investigator, Evaluation Administrator + Instrument Maker, signed the pre-registration, labelled every draw, and signs this issue, and is its sole authority. The prose was drafted by an AI agent, reviewed adversarially by an independent agent, and edited and signed by him. Budday Budderson Studio LLC publishes the series and grades nothing.