# Full k=10 Matrix

Date: 2026-07-20

Batch: `tmp/pilot-runs/full-k10-20260720-six-scenarios`

Scope: 6 tasks, Codex and Claude, `git` vs `but+skill` vs `jj+skill`, 360 canonical runs.

## Headline

359/360 canonical runs passed the deterministic verifier. Plain `git` and GitButler each passed 120/120; Jujutsu passed 119/120. The one miss was a Claude Jujutsu split-commit run with the right final files but one extra commit in the graph.

GitButler remained the fastest and least command-heavy arm overall. Against plain `git`, its mean wall time was 63.1% lower for Codex and 67.3% lower for Claude, with 77.8% and 78.6% fewer task-relevant version-control commands respectively. Jujutsu was 19.1% slower than `git` for Codex and 47.9% slower for Claude; it used 2.9% more commands for Codex and 30.4% fewer for Claude.

The task-clustered comparison supports the command-count result for both agents and the wall-time result for Codex. Claude's GitButler wall-time interval remains wide over six tasks. Claude Jujutsu's paired wall-time interval is entirely above zero; the other Jujutsu intervals cross zero.

## Run Integrity and Excluded Attempts

The published aggregate is a strict rectangle: 6 tasks × 2 agents × 3 arms × k=10 = 360 canonical rows. It was recomputed from four source batches rather than reusing their summary objects.

Fifteen infrastructure-failed attempts were preserved outside the canonical aggregate across 12 canonical slots: four HTTP 529 responses, five HTTP 500 responses, three HTTP 401 responses, and three interrupted runs. Each affected slot was rerun from a fresh isolated workspace, and every canonical replacement passed. The failures spanned `git`, `but+skill`, and `jj+skill`; none contributes to the pass rates, timing, or command counts below. Claude OAuth was refreshed and an explicit Opus probe passed before continuation.

Some preserved attempts share a canonical run ID with their replacement. The merged `manifest.tsv` and `aggregate.json`, not recursive directory scans, define the canonical set.

## Overall

| Agent | Arm | n | Pass | Pass 95% CI | Tasks all-k | Mean wall | Median wall | P90 wall | Max wall | Task VC |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Claude | `but+skill` | 60 | 60/60 | 94%-100% | 6/6 | 32.2s | 25.4s | 55.8s | 67.0s | 4.6 |
| Claude | `git` | 60 | 60/60 | 94%-100% | 6/6 | 98.4s | 49.3s | 222.5s | 688.3s | 21.6 |
| Claude | `jj+skill` | 60 | 59/60 | 91%-100% | 5/6 | 145.7s | 98.4s | 313.3s | 408.8s | 15.0 |
| Codex | `but+skill` | 60 | 60/60 | 94%-100% | 6/6 | 34.0s | 26.2s | 69.3s | 84.3s | 4.5 |
| Codex | `git` | 60 | 60/60 | 94%-100% | 6/6 | 92.2s | 73.3s | 160.6s | 231.0s | 20.4 |
| Codex | `jj+skill` | 60 | 60/60 | 94%-100% | 6/6 | 109.9s | 105.9s | 191.6s | 209.7s | 21.0 |

The mean and median tell the same broad story, but `git` and Jujutsu retain heavier tails. The longest canonical run was a passing 688.3s Claude `git` multi-amend run. GitButler's longest run was 84.3s.

## Paired Per-Task Deltas vs `git`

Each task contributes one paired arm-minus-`git` difference. Intervals are t-based 95% CIs over the six tasks; trials within a task are correlated and are not treated as six independent task samples.

| Agent | Arm | Pass-rate delta | Mean wall delta | Task VC delta | Tasks |
| --- | --- | --- | --- | --- | ---: |
| Claude | `but+skill` | 0 pp [0, 0] | -66.3s [-158.0s, +25.5s] | -16.9 [-32.6, -1.3] | 6 |
| Claude | `jj+skill` | -1.7 pp [-6.0, +2.6] | +47.2s [+6.6s, +87.8s] | -6.6 [-18.2, +5.0] | 6 |
| Codex | `but+skill` | 0 pp [0, 0] | -58.2s [-105.6s, -10.7s] | -15.8 [-26.0, -5.7] | 6 |
| Codex | `jj+skill` | 0 pp [0, 0] | +17.6s [-23.3s, +58.6s] | +0.6 [-8.5, +9.6] | 6 |

## By Task

| Task | Agent | Arm | Pass | Mean wall | Task VC |
| --- | --- | --- | ---: | ---: | ---: |
| Selective commit | Claude | `but+skill` | 10/10 | 20.3s | 2.0 |
| Selective commit | Claude | `git` | 10/10 | 39.4s | 11.6 |
| Selective commit | Claude | `jj+skill` | 10/10 | 119.7s | 10.0 |
| Selective commit | Codex | `but+skill` | 10/10 | 20.7s | 2.0 |
| Selective commit | Codex | `git` | 10/10 | 61.3s | 17.8 |
| Selective commit | Codex | `jj+skill` | 10/10 | 94.3s | 17.1 |
| Multi-amend | Claude | `but+skill` | 10/10 | 26.8s | 5.0 |
| Multi-amend | Claude | `git` | 10/10 | 227.8s | 48.4 |
| Multi-amend | Claude | `jj+skill` | 10/10 | 277.9s | 19.6 |
| Multi-amend | Codex | `but+skill` | 10/10 | 29.0s | 5.0 |
| Multi-amend | Codex | `git` | 10/10 | 168.9s | 35.7 |
| Multi-amend | Codex | `jj+skill` | 10/10 | 124.2s | 22.4 |
| Split commit | Claude | `but+skill` | 10/10 | 42.6s | 6.2 |
| Split commit | Claude | `git` | 10/10 | 194.9s | 32.5 |
| Split commit | Claude | `jj+skill` | 9/10 | 297.9s | 32.8 |
| Split commit | Codex | `but+skill` | 10/10 | 35.8s | 6.0 |
| Split commit | Codex | `git` | 10/10 | 117.6s | 30.0 |
| Split commit | Codex | `jj+skill` | 10/10 | 187.7s | 40.1 |
| Reorder commits | Claude | `but+skill` | 10/10 | 20.8s | 2.2 |
| Reorder commits | Claude | `git` | 10/10 | 34.3s | 8.0 |
| Reorder commits | Claude | `jj+skill` | 10/10 | 53.4s | 4.4 |
| Reorder commits | Codex | `but+skill` | 10/10 | 21.1s | 2.0 |
| Reorder commits | Codex | `git` | 10/10 | 50.5s | 9.1 |
| Reorder commits | Codex | `jj+skill` | 10/10 | 53.1s | 9.9 |
| Squash commits | Claude | `but+skill` | 10/10 | 23.6s | 3.1 |
| Squash commits | Claude | `git` | 10/10 | 42.4s | 12.2 |
| Squash commits | Claude | `jj+skill` | 10/10 | 42.9s | 7.4 |
| Squash commits | Codex | `but+skill` | 10/10 | 23.2s | 3.0 |
| Squash commits | Codex | `git` | 10/10 | 44.4s | 12.7 |
| Squash commits | Codex | `jj+skill` | 10/10 | 51.9s | 10.0 |
| Update dirty branch | Claude | `but+skill` | 10/10 | 58.8s | 9.2 |
| Update dirty branch | Claude | `git` | 10/10 | 51.8s | 16.7 |
| Update dirty branch | Claude | `jj+skill` | 10/10 | 82.1s | 15.8 |
| Update dirty branch | Codex | `but+skill` | 10/10 | 74.4s | 9.1 |
| Update dirty branch | Codex | `git` | 10/10 | 110.6s | 16.9 |
| Update dirty branch | Codex | `jj+skill` | 10/10 | 148.0s | 26.2 |

The new update-dirty-branch task is materially different from the five history-edit tasks. GitButler was faster than `git` for Codex but 7.0s slower for Claude on mean wall time; it still used substantially fewer commands for both. All 60 runs of the new task passed.

## Failure Read

One canonical run failed: `fresh-k10-20260720-claude-s123-018-pilot-3-split-commit-claude-jjplusskill-r2` (`GRAPH_WRONG`, score 110). Its final tree, semantic atoms, and minefields were correct, but the history contained an extra `add app configuration` commit, so `commit_subjects_preserved=false`. This is a graph-shape miss rather than lost or incorrectly partitioned content.

## GitButler Regression Watch

No credible GitButler regression appears in this batch. The current artifacts passed 120/120 canonical GitButler runs with zero failed actual `but` task commands. A few help and orientation probes recovered without affecting correctness.

The table below compares the five shared tasks with the [2026-07-06 k=10 batch](full-k10-2026-07-06.md). It is directional, not a controlled A/B: the GitButler binary and skill differ, and the agent CLIs also changed.

| Task | Agent | Mean wall, old → current | Task VC, old → current |
| --- | --- | ---: | ---: |
| Selective commit | Codex | 22.9s → 20.7s | 2.0 → 2.0 |
| Multi-amend | Codex | 40.8s → 29.0s | 6.5 → 5.0 |
| Split commit | Codex | 34.3s → 35.8s | 6.1 → 6.0 |
| Reorder commits | Codex | 19.6s → 21.1s | 2.0 → 2.0 |
| Squash commits | Codex | 25.7s → 23.2s | 3.0 → 3.0 |
| Selective commit | Claude | 23.4s → 20.3s | 3.0 → 2.0 |
| Multi-amend | Claude | 64.6s → 26.8s | 6.7 → 5.0 |
| Split commit | Claude | 58.4s → 42.6s | 7.5 → 6.2 |
| Reorder commits | Claude | 32.7s → 20.8s | 3.3 → 2.2 |
| Squash commits | Claude | 43.5s → 23.6s | 4.1 → 3.1 |

Across those same five tasks, Codex GitButler mean wall time moved from 28.7s to 26.0s and commands from 3.9 to 3.6; Claude moved from 44.5s to 26.8s and commands from 4.9 to 3.7. The two small Codex wall-time increases, split commit (+4%) and reorder (+8%), came without added commands or failures.

On update-dirty-branch, comparison with recent older-binary runs is likewise directional: Codex moved from 72.0s and 8.2 commands to 74.4s and 9.1; Claude moved from 61.9s and 8.8 commands to 58.8s and 9.2. The known Codex `config target` detour occurred in 0/10 current runs, closing the requested k≥15 watch when combined with the prior post-guard trials.

## Provenance

| Agent | Configured and observed model | CLI version |
| --- | --- | --- |
| Codex | `gpt-5.5` | `codex-cli 0.144.6` |
| Claude | `claude-opus-4-8` | `2.1.215 (Claude Code)` |

GitButler:

- Authoritative common base / `origin/master`: `66505acb5ef98299b1dc7ff9f9fd5c58b82d2871`
- Packaged binary SHA-256: `3f654efed91fa1b4a040cd0204989c5e9f5e7a21ec92a8b798e5d5e5e95580f6`
- Source `SKILL.md` SHA-256: `c680cd1ce5f1064f54fbdb177ede321b924e31a242a76c2299ede22dbe776b40`
- Installed skill file SHA-256: `8acc12b42c2c8d6ea5062ea0cf6a58ab2b70d8fba828b50fe854aa4974e475ff`
- Installed skill tree SHA-256: `3c5ad4f7691c3542475b5cfd33a6bb0de2b442414e52e3a999c01a81c54fc701`
- Setup block SHA-256: `e68d505c5f060b89692c253e8dddb689fb42c5a61fc001bdce0a8dc383db76b2`

The aggregate's raw `source_head` field is `02f9b37c7c2453e2dcccba44d3b491fbd81a4e72`, a stale virtual-workspace HEAD. The built tree matched `origin/master`; the common-base value above is the authoritative source revision.

Jujutsu:

- Version: `jj 0.42.0`
- Binary SHA-256: `849c9ab4bbfd955b9d9fbe39c3a63b071e348e3978888b5289d3dccd99379680`
- Skill: `onevcat/skills@onevcat-jj`, pinned to commit `4955f542`
- Skill file SHA-256: `e0364004187a1769adc0b532befe346fd4b372bb1aab2768b9ebb694f2d13687`
- Skill tree SHA-256: `5261edada4513cfd358bb1e0ccc3b7ef1be8e77e083bdc05e0481d377602dc3f`
- Setup block SHA-256: `b440ab7d0e70159b81f573267df5c284bf387e006dbc7c8dee83cb897443c91f`

## Notes

- Pass-rate intervals are Wilson 95% intervals over trials in a cell.
- "Tasks all-k" counts tasks where every trial passed, the per-task pass^k reliability gate.
- Fixture creation, tool setup, skill installation, local agent instruction files, and dirty-state application happen before timing starts.
- Task VC counts include task-relevant version-control commands, not tool-internal calls or agent platform probes.
- Claude and Codex transcript byte counts are not cross-agent comparable and are omitted from the headline tables.

## Raw Evidence

- Aggregate JSON: `tmp/pilot-runs/full-k10-20260720-six-scenarios/aggregate.json`
- Manifest: `tmp/pilot-runs/full-k10-20260720-six-scenarios/manifest.tsv`
- Merged report: `tmp/pilot-runs/full-k10-20260720-six-scenarios/report.md`
- Source batches: `fresh-k10-20260720-codex-s123`, `fresh-k10-20260720-codex-s456`, `fresh-k10-20260720-claude-s123`, `fresh-k10-20260720-claude-s456`
- Merge command: `node scripts/merge-matrix-batches.mjs --batches fresh-k10-20260720-codex-s123,fresh-k10-20260720-codex-s456,fresh-k10-20260720-claude-s123,fresh-k10-20260720-claude-s456 --out tmp/pilot-runs/full-k10-20260720-six-scenarios --expected-k 10`