# 100-task harness evaluation

This pass@1 experiment holds `gpt-5.6`, high reasoning effort, prompt, benchmark tasks,
container images, and official grading fixed. The treatment is the Codex CLI release. The panel
contains 100 deterministic SWE-bench Multilingual tasks across seven releases (700 cells).

Generation produced 695 normal completions, four timeouts, and one agent error. One completed cell
had an empty patch. The official evaluator also reported the same patch-application error for
`jqlang__jq-2598` under every release. All of these are counted uniformly as unresolved; no failed
pass@1 cell was retried.

| Codex CLI | Release date | Resolved | Input tokens/task | Output tokens/task | Estimated USD/task |
|---|---:|---:|---:|---:|---:|
| 0.80.0 | 2026-01-09 | 56/100 | 561,403 | 5,957 | $0.618 |
| 0.98.0 | 2026-02-05 | 56/100 | 1,121,881 | 8,557 | $1.030 |
| 0.110.0 | 2026-03-05 | 56/100 | 1,039,662 | 8,044 | $0.960 |
| 0.120.0 | 2026-04-11 | 57/100 | 1,105,598 | 7,905 | $0.995 |
| 0.130.0 | 2026-05-08 | 52/100 | 925,290 | 6,507 | $0.843 |
| 0.140.0 | 2026-06-15 | 58/100 | 847,680 | 6,325 | $0.801 |
| 0.144.6 | 2026-07-18 | 56/100 | 909,975 | 6,760 | $0.851 |

Token and cost means use the 99 or 100 cells per release that completed generation and reported
usage. Correctness always uses all 100 planned tasks.

## Biggest finding

Harness age has no detectable relationship with correctness in this panel. The endpoint releases
both resolve 56/100. At task level, 0.144.6 gains five tasks and loses five relative to 0.80.0
(exact McNemar p=1.0). Seventy-six tasks are invariant across all releases and 24 discriminate.
The task-paired linear release-time statistic is exactly zero; a 100,000-draw within-task release
permutation therefore gives p=1.0. The observed sequence—56%, 56%, 56%, 57%, 52%, 58%, 56%—is
release-specific churn rather than a correctness scaling curve.

The real harness evolution is non-monotonic token efficiency. The 0.98.0–0.120.0 releases consume
far more tokens than 0.80.0. Starting at 0.130.0, usage recovers substantially while correctness
stays essentially flat. Across the 95 tasks with complete usage under every release, the late
0.130.0–0.144.6 group has median within-task reductions of 14% input tokens, 20% output tokens, and
13% estimated cost relative to 0.98.0–0.120.0. The reductions occur on 79%, 91%, and 81% of paired
tasks respectively (Wilcoxon p=5.1e-8, 4.2e-14, and 9.3e-11).

This is a partial recovery, not steady improvement. Compared directly with 0.80.0 on those same 95
tasks, 0.144.6 uses a median 51% more input, 12% more output, and costs 31% more. The oldest harness
is still the most token-efficient release tested.

## Independent 50-task replication

The second 50 tasks reproduce the late-release recovery more strongly. On the 45 new tasks with a
complete seven-release usage panel, the late group reduces median input by 18%, output by 22%, and
cost by 17% relative to the middle group. The corresponding reductions in the original 50 tasks
were 11%, 17%, and 10%.

| Tranche | Complete paired tasks | Late vs middle input | Late vs middle output | Late vs middle cost |
|---|---:|---:|---:|---:|
| Original 50 | 50 | -11% | -17% | -10% |
| Added 50 | 45 | -18% | -22% | -17% |
| Pooled 100 | 95 | -14% | -20% | -13% |

Thus the efficiency recovery is not an artifact of the first task slice. It replicated on new
repositories and issues, whereas the correctness ranking did not stabilize into a date trend.

## Accounting and limitations

The full experiment logged 646,606,896 input tokens and 4,970,501 output tokens, with an estimated
standard-tier cost of $605.53. The estimate excludes unreported cache-write tokens and
unreconstructable per-request long-context multipliers. Generation was parallelized by release and
task slice for throughput, so wall-clock latency is excluded from cross-release inference.

This experiment measures the net behavior of each released harness bundle. It cannot attribute the
efficiency jump or task churn to a particular internal change, and it cannot freeze server-side
behavior behind the `gpt-5.6` alias. Trace-level comparison around the 0.120.0 to 0.130.0 boundary is
the next useful analysis if the goal is to identify the mechanism behind the token reduction.