The Model Release Experiment
Revised October 9, 2026.
TLDR
- I am keeping a record alongside my impressions: one frozen story-and-math task, the requested settings, and each model's original response.
- Start with the model cards. Receipt checks cover arithmetic; judging the whole response takes a separate look.
I have an anecdotal feel for how Claude and Codex work. I want a record to put alongside that impression. The Model Release Experiment gives successive available models the same task, then keeps what each actually returned. The readable model cards organize the record by model; their linked run pages hold the original responses, receipts, technical evidence, and client-reported usage where available. An unavailable model keeps that status, rather than receiving a silent substitute.
The frozen task is PRINT-SHOP CHALLENGE v1.0. Mara runs a community print shop on its final afternoon before a repair deadline; Ivo wants her to prioritize the theater's banners. Eight possible orders compete for materials and 120 machine minutes. Before work begins, the materials budget falls from $80 to $66. A fixed $35 post-production expense comes out before the remaining contribution is compared with the $220 repair goal. A possible $100 donation is still unconfirmed and cannot count toward it. The model chooses its own voice, point of view, and tone, but the calculation must change the characters' decisions and relationship. It returns a titled 600–800-word story, one JSON receipt with a brief checkable explanation and one feasible competing plan, and a public notice of at most 80 words.
The protocol keeps the user prompt byte-identical, even though native provider and CLI instructions can differ. Standard profiles vary one supported native control at a time; factorial testing is separate. Codex distinguishes reasoning effort from output verbosity, with availability depending on the model and client. Claude's effort labels are calibrated per model, so matching labels across providers do not establish matching behavior. Claude Code's --verbose flag controls diagnostic logging and full turn-by-turn display, not model output verbosity. These records show requested configurations and reported outputs; they do not independently verify hidden internal reasoning.
The original responses stay unpolished, and saved failures remain visible. That matters when a receipt passes but the story disappoints, or when a useful response misses a constraint the receipt checker does not cover. The results below state the check's boundaries and link to the evidence for each profile.
The experiment runs
Results update
121 original responses from 21 models are now saved. 83 receipts matched both optimal order plans, their totals, and the unconfirmed donation. 38 receipts failed at least one of those checks. 5 earlier failed attempts remain in the record.
13 models returned both passing and failing receipts across their saved profiles. Each profile has one response, so these differences do not establish that a setting caused a result. The receipt check does not assess story quality, story consistency, or word limits.
Each run shows the requested thinking and verbosity settings, the original response, the receipt checks, and the usage reported by the client.