What Prompt Precision Bought in 24 Agent Trials

By Josiah HunterOctober 2, 202630 min read

First published October 2, 2026; revised October 9, 2026.

TLDR

Observed: All 24 trials met the full intended specification on the first pass. No correction, timeout, error, or clarification request occurred, and every run executed its provided check.
Observed: Spelling-error runs stayed within 2% of clear mean output. Reduced-grammar runs used 5% more output for Task A and 18% more for Task B. Ambiguous-rule runs used 9% more for Task A and 27% more for Task B.
Interpretation: Ambiguous rules transferred rule selection to the agent. The selected conventional defaults matched the intended rules, but unconventional intent was not tested, so ambiguity was not shown safe.
Observed: This was a small experiment with one model and harness, two synthetic tasks, four conditions, and three repetitions per task-condition cell. The 24-of-24 result created a ceiling that prevented detection of outcome differences.

What the 24 trials changed

Observed: The experiment used claude-sonnet-5-5 through Claude Code CLI 2.1.287 at high effort. It covered two small synthetic tasks, four prompt conditions, and three repetitions per task-condition cell. Task A summarized orders by region. Task B repaired a billing-proration function. Independent validators recomputed the intended answers from frozen fixtures, and every run executed its provided check after the final file change. The models later used to write and review this article were not trial models.

Observed: The conditions were clear instructions, spelling errors, reduced grammar and punctuation, and ambiguous governing rules. The spelling variant misspelled about one in three ordinary words while leaving identifiers intact. The reduced-grammar variant removed punctuation, capitalization, and many function words. The ambiguity variant replaced an explicit rule with a vague sentence.

Interpretation: These were three different manipulations, not one scale from bad English to good English. Spelling and grammar changed the surface form of the instructions. Ambiguity changed who selected an important rule.

Spelling errors left the measured result unchanged

Observed: All six spelling-error runs succeeded. Mean output was 1,277 tokens for Task A versus 1,254 for clear, and 693 for Task B versus 700 for clear. Both means were within 2% of clear. Tool and turn behavior was broadly comparable, and no run mentioned the misspellings.

Interpretation: On these tasks, this model read through the tested misspellings. The experiment did not test errors in identifiers, paths, numbers, negations, or domain terms. A typo in one of those places could change the instruction itself.

Reduced grammar preserved the result but not identical output

Observed: All six reduced-grammar runs succeeded. Mean output was 1,317 tokens for Task A versus 1,254 for clear, a 5% difference. For Task B, the mean was 823 versus 700, an 18% difference.

Observed: One early Task B run, T01, made an extra Bash call to read the tests and produced 996 output tokens. The other two produced 732 and 740, so T01 accounts for most of the Task B gap.

Interpretation: The prompts remained understandable in these tasks, but the design did not isolate grammar as the cause of the output difference. Turn behavior and ordinary run-to-run variation remain possible explanations.

Ambiguous rules turned instructions into agent choices

Observed: The ambiguous Task A prompt said:

Leave out any orders that shouldn't count.

Observed: The ambiguous Task B prompt said:

Make sure the amounts come out right.

Observed: All six ambiguous runs reached the intended result. Task A selected completed orders, normalized statuses, and deduplicated records. Task B used inclusive billing days and half-up rounding. No run asked for clarification. Under the broad manual coding used for the final messages, four of six messages flagged a judgment or explained a choice beyond the tests.

Interpretation: The selected conventional defaults matched the intended rules. That explains the bounded result, not the success mechanism. The experiment did not test intent that departed from convention, so it did not show that ambiguity is harmless.

Observed: Ambiguous prompts produced 1,370 mean output tokens for Task A versus 1,254 for clear, a 9% difference. For Task B, they produced 889 versus 700, a 27% difference. Final messages were also longer: 993 versus 930 characters for Task A and 809 versus 569 for Task B.

Interpretation: The experiment did not identify which additional tokens were explanatory or establish ambiguity as the cause of the difference. It did show the practical cost of leaving a governing rule to the agent: even a correct conventional choice remains an agent-selected choice.

Tokens and historical API-equivalent estimates

Observed: Input-side totals changed little with wording except in two Task B runs with an extra turn. Total uncached input, cache-write, and cache-read tokens ranged from 33,569 to 35,902 per Task A run. Task B normally ranged from 22,642 to 22,825, except for totals of 31,318 and 31,173 in the two extra-turn runs.

Interpretation: Turn count is a plausible explanation for the larger Task B totals, but this design did not isolate causality from prompt wording.

Calculated estimate: Using the official Claude pricing retrieved on October 2, 2026, individual API-equivalent estimates ranged from $0.0131 to $0.0369. Mean estimates across both tasks were $0.0241 for clear, $0.0241 for spelling, $0.0257 for reduced grammar, and $0.0242 for ambiguous. The 24-trial total was $0.5890. These are API-equivalent estimates, not charges. Observed: The subscription runs produced no per-request billed cost, and subscription utilization could not be separated by trial.

Calculated estimate: Mean API-equivalent cost per successful completion ranged from $0.0281 to $0.0334 for Task A and from $0.0155 to $0.0202 for Task B, depending on condition. Because every trial succeeded, the mean per-success estimate equals the mean per-trial estimate in this sample.

Calculated estimate: The additional ambiguous-condition output corresponded to about $0.001 to $0.002 per task at the locked output price, based on 117 additional tokens for Task A and 190 for Task B.

Calculated estimate: Condition-level estimate differences were smaller than within-condition spread. Observed: Cache writes ranged from 492 to 4,376 tokens per run, and the earliest Task A run in every condition had the most cache writes.

Interpretation: Cache-write timing appears important, but the experiment did not isolate cache timing, turn count, or wording as causes.

Calculated estimate: Under the stated workload, model, settings, cache behavior, historical prices, and observed success rates, 1,000 comparable tasks project to about $24 to $26 per condition. The projected success counts use the observed 100% rate and are optimistic. With six successes in six trials per condition, the one-sided 95% exact binomial lower bound is 0.05^(1/6) = 0.606962 under independent trials with a common success probability. It is not a real-world guarantee.

Observed: The experiment used one model and harness, two small synthetic tasks with conventional intended answers, four conditions, and three repetitions per cell. Sampling settings were not exposed by the CLI. Cache-timing noise remained. Subscription use was not separable by trial. The all-success result created a ceiling effect, so the experiment could not detect outcome differences.

A specification checklist

Interpretation: For the following checklist, prompt precision should be spent first where a wrong decision would matter.

Governing rules: State the rule the agent must apply.
Exceptions: Identify cases that do not follow the default.
Invariants: Name what must remain unchanged.
Success checks: Define how the result will be validated.
Surface clarity: Keep spelling and grammar clear enough to carry those rules.

Interpretation: This is a bounded decision rule for the tested model, harness, tasks, and prompt variants. It is not a universal prompting framework. The evidence does not show that spelling and grammar never matter, or that ambiguity is safe when the intended rule is unconventional or difficult to detect.

Technical evidence by task and condition

Observed: The appendix preserves the recorded prompts, repetition rows, validation results, usage records, commands, diffs, and output excerpts. Calculated estimate: Dollar figures and projections are API-equivalent estimates based on the frozen historical price inputs. Interpretation: Explanations of conventional defaults, timing, and practical implications are bounded readings of the evidence.

Observed: Additional condition comparisons and calculation assumptions

Observed: The experiment used two synthetic tasks, four prompt conditions and three repetitions for each task-condition pairing, for 24 trials total. Task A created a regional summary from order and exchange-rate files. Task B repaired a proration function. Independent validators checked the intended answers, including requirements that visible checks did not cover. Every Task A run executed check_format.py, and every Task B run executed pytest.

Observed: All six spelling-error prompts succeeded. Mean output was 1,277 tokens for Task A compared with 1,254 for clear, and 693 for Task B compared with 700 for clear.

Observed: All six reduced-grammar prompts succeeded. Mean output was 1,317 tokens for Task A and 823 for Task B. The three Task B outputs were 996, 732 and 740 tokens.

Observed: All six ambiguous prompts reached the intended answer. Task A used completed orders, normalized statuses and deduplicated records. Task B used inclusive billing days and half-up rounding. Four of six final messages flagged a judgment or explained a choice beyond the tests. None asked for clarification.

Observed: Input-side totals ranged from 33,569 to 35,902 for Task A. Task B normally ranged from 22,642 to 22,825, with two extra-turn totals of 31,318 and 31,173.

Calculated estimate: The 24 trials totaled $0.5890 in API-equivalent estimates. The clear-condition mean was $0.0241, the ambiguous-condition mean was $0.0242, condition means ranged from $0.0241 to $0.0257, and individual trials ranged from $0.0131 to $0.0369.

Observed:

Task	Clear	Spelling
Task A	1,254	1,277
Task B	700	693

Observed:

Task	Clear mean output	Ambiguous mean output	Difference
Task A	1,254	1,370	+9%
Task B	700	889	+27%

Observed: Final messages averaged 993 versus 930 characters for Task A and 809 versus 569 for Task B.

Calculated estimate: A projection for 1,000 comparable tasks, half like Task A and half like Task B, lands at approximately $24 to $26 for every condition, or about $0.024 to $0.026 per successful task under the stated assumptions.

Observed: Unused correction message and T10 display precision

Observed: The frozen correction message was never used:

An automated check found that the result does not yet meet the task's requirements. Review the original request and the files, then make any corrections you judge necessary.

Calculated estimate: T10's raw API-equivalent estimate was $0.0252498. The appendix displays $0.0253 after its documented intermediate rounding, while direct four-decimal rounding would be $0.0252. The clear-condition mean remains $0.0241.

Observed: Setup: model, interface, settings and what was held constant

Observed: The following block records the historical setup and what the interface exposed.

Model (requested and returned by every API response): claude-sonnet-5-5
Interface: Claude Code CLI 2.1.287, headless `claude -p`, stream-json output
Host: Anthropic cloud Linux container; trials run one at a time on 2026-10-02 between 2026-10-02T09:47:52+00:00 and 2026-10-02T09:51:37+00:00 (start times, UTC)
Account: claude.ai subscription session (no API key; no per-request charge)
Agent tools: Read, Write, Edit, Bash, Glob, Grep (no web, no subagents, no MCP)
Permissions: bypassPermissions; settings sources: none; fresh empty config dir per trial
Effort: high; turn limit 40 per invocation; wall limit 900 s per invocation
Not exposed by the interface: temperature, top-p/top-k, exact system prompt text
Isolation: new session ID, identical freshly copied workspace at the same path, trials sequential
Order: 24 trials shuffled with random.Random(4120970583)
Correction rule: one resumed turn with a fixed, task-neutral message, only if the
  independent validator failed after pass 1 (it was never triggered)
Writing/review models used later by the article workflow played no part in any trial

Observed: Token accounting and pricing used for cost estimates

Observed: The block records reported tokens, historical price inputs, subscription readings and the recorded agreement between calculations and CLI estimates. Calculated estimate: Its dollar totals apply those prices to the reported token counts; they are not charges.

Token counts: provider-reported usage from the CLI result event of each session,
  cross-checked against per-message usage in the session transcript (deduplicated by
  message ID). Matched for all 24 trials: True
input_tokens, cache_creation_input_tokens and cache_read_input_tokens are disjoint;
  'all input-side tokens' below = their sum (Anthropic prompt-caching docs).
Output tokens include the provider-reported thinking tokens shown in brackets.

Prices (official list prices, claude-sonnet-5-5, retrieved 2026-10-02 from
  https://platform.claude.com/docs/en/about-claude/pricing):
  input $2.00/MTok   output $10.00/MTok   5-min cache write $2.50/MTok
  1-hour cache write $4.00/MTok   cache read $0.20/MTok
All cache writes in this experiment used the 1-hour TTL (52039 tokens).

These runs were on a subscription, so every dollar figure is an API-EQUIVALENT ESTIMATE,
not a charge. The CLI's own total_cost_usd (list-price basis) agreed with this
calculation for every trial: True.
Total API-equivalent estimate for all 24 trials: $0.5890
Subscription utilization reported by the CLI at the first and last trial (account-wide,
  2-decimal precision): five-hour 0.07 -> 0.07, seven-day 0.59 -> 0.59

Observed: Task A (data processing) · Clear baseline · 3 of 3 runs met the full specification on the first pass

Observed: Exact prompt:

This folder contains data/orders.csv, a list of customer orders, and data/fx_rates.csv, which gives the US dollar value of one unit of each currency. Create output/region_summary.csv with exactly these columns, in this order: region, order_count, total_usd, avg_order_usd. Include only orders whose status is "completed", ignoring capitalization and surrounding spaces, and count each order_id only once. Convert every amount to US dollars with data/fx_rates.csv, round total_usd and avg_order_usd to two decimal places, and sort the rows by total_usd from highest to lowest. When you finish, run python3 check_format.py to check the file's structure.

Observed: Trial identity, execution, validation, tokens, CLI-reported estimates and wall time below are historical records. Calculated estimate: The API-equivalent cost column is derived from those tokens and the frozen prices.

Observed: All repetitions (execution order; tokens provider-reported; cost is an API-equivalent estimate):

trial                order pass1 correction turns tools  tool mix               input cache_wr_1h cache_read output (thinking) all input-side API-eq cost  CLI cost wall s
T02-A-clear-r1           2  PASS   not used     4     3  Bashx3                     8        4375      30913   1262          0          35296     $0.0363   $0.0363    9.8
T04-A-clear-r2           4  PASS   not used     4     3  Bashx3                     8        3476      31037   1122         45          34521     $0.0313   $0.0313    9.1
T16-A-clear-r3          16  PASS   not used     5     4  Bashx4                     8        2482      33412   1377          0          35902     $0.0304   $0.0304   10.3

Observed: Independent validation of the final workspace:

T02-A-clear-r1: format_ok=True inputs_unmodified=True intended_match=True matched=['status=completed|match=strip+lower|duplicates=dedup']
T04-A-clear-r2: format_ok=True inputs_unmodified=True intended_match=True matched=['status=completed|match=strip+lower|duplicates=dedup']
T16-A-clear-r3: format_ok=True inputs_unmodified=True intended_match=True matched=['status=completed|match=strip+lower|duplicates=dedup']

Observed: Commands run in T02-A-clear-r1 (excerpt from the tool log, first line of each command):

$ cd /srv/xp-trial/ws; ls -R . | head -50; head -15 data/orders.csv; wc -l data/*.csv; cat data/fx_rates.csv; cat check_format.py
$ cd /srv/xp-trial/ws; cat data/orders.csv | tail -40; python3 - <<'E'
$ cd /srv/xp-trial/ws; mkdir -p output; python3 - <<'E'

Observed: Agent's final message in T02-A-clear-r1 (excerpt):

I created `output/region_summary.csv`, and `python3 check_format.py` reports `FORMAT OK: 4 rows`. That script only checks structure, so it doesn't confirm the numbers.

| region | order_count | total_usd | avg_order_usd |
|---|---|---|---|
| West | 7 | 2074.27 | 296.32 |
| East | 7 | 2042.00 | 291.71 |
| South | 7 | 1518.44 | 216.92 |
| North | 7 | 1382.31 | 197.47 |

- **Status filter:** an order counts if its status is "completed" after trimming spaces and ignoring case. That picked up variants like `COMPLETED ` and ` Completed `.
- **Duplicates:** repeated order_ids were counted once. The repeated rows were exact copies of each other, so it didn't matter which one I kept.
- **Currency [...]

Observed: Task A (data processing) · Spelling errors · 3 of 3 runs met the full specification on the first pass

Observed: Exact prompt:

This foldr contians data/orders.csv, a list of custmer orders, and data/fx_rates.csv, wich gives the US dolar value of one unti of each curency. Creat output/region_summary.csv with exactley these colums, in this ordr: region, order_count, total_usd, avg_order_usd. Inclued only orders whoes status is "completed", ignorng capitalizaton and surounding spaces, and cuont each order_id only once. Convrt evry amount to US dollers with data/fx_rates.csv, rond total_usd and avg_order_usd to two decmal places, and sotr the rows by total_usd from higest to lowset. When you finsh, run python3 check_format.py to chek the file's structre.

Observed: All repetitions (execution order; tokens provider-reported; cost is an API-equivalent estimate):

trial                order pass1 correction turns tools  tool mix               input cache_wr_1h cache_read output (thinking) all input-side API-eq cost  CLI cost wall s
T12-A-spelling-r2       12  PASS   not used     4     3  Bashx3                     8        4376      30819   1317         39          35203     $0.0369   $0.0369   10.5
T18-A-spelling-r3       18  PASS   not used     4     3  Bashx3                     8        3593      31493   1230         40          35094     $0.0330   $0.0330    9.9
T20-A-spelling-r1       20  PASS   not used     4     3  Bashx3                     8        2165      33008   1284         85          35181     $0.0281   $0.0281   13.1

Observed: Independent validation of the final workspace:

T12-A-spelling-r2: format_ok=True inputs_unmodified=True intended_match=True matched=['status=completed|match=strip+lower|duplicates=dedup']
T18-A-spelling-r3: format_ok=True inputs_unmodified=True intended_match=True matched=['status=completed|match=strip+lower|duplicates=dedup']
T20-A-spelling-r1: format_ok=True inputs_unmodified=True intended_match=True matched=['status=completed|match=strip+lower|duplicates=dedup']

Observed: Commands run in T12-A-spelling-r2 (excerpt from the tool log, first line of each command):

$ ls -R . | head -50 && head -15 data/orders.csv && wc -l data/*.csv && cat data/fx_rates.csv && cat check_format.py
$ cat data/orders.csv | tail -40; cut -d, -f3 data/orders.csv | sort | uniq -c; cut -d, -f6 data/orders.csv | sort | uniq -c; cut -d, -f7 data/orders.cs
$ mkdir -p output && cat > /tmp/build.py <<'EOF'

Observed: Agent's final message in T12-A-spelling-r2 (excerpt):

I created `output/region_summary.csv`, and `python3 check_format.py` reports `FORMAT OK: 4 rows`. That script only checks structure, so it doesn't confirm the numbers are right.

| region | order_count | total_usd | avg_order_usd |
|---|---|---|---|
| West | 7 | 2074.27 | 296.32 |
| East | 7 | 2042.00 | 291.71 |
| South | 7 | 1518.44 | 216.92 |
| North | 7 | 1382.31 | 197.47 |

- **Status filter:** I kept only orders whose status is "completed" after trimming spaces and ignoring case. That covers variants like `COMPLETED ` and ` Completed `.
- **Duplicates:** I counted each `order_id` once. The repeated rows I saw were exact copies, and the script would have printed a warning if any had [...]

Observed: Task A (data processing) · Reduced grammar and punctuation · 3 of 3 runs met the full specification on the first pass

Observed: Exact prompt:

folder has data/orders.csv customer orders and data/fx_rates.csv us dollar value of one unit each currency create output/region_summary.csv exactly these columns this order region order_count total_usd avg_order_usd include only orders status completed ignore capitalization and surrounding spaces count each order_id only once convert every amount to us dollars with data/fx_rates.csv round total_usd and avg_order_usd two decimal places sort rows by total_usd highest to lowest when finish run python3 check_format.py check file structure

Observed: All repetitions (execution order; tokens provider-reported; cost is an API-equivalent estimate):

trial                order pass1 correction turns tools  tool mix               input cache_wr_1h cache_read output (thinking) all input-side API-eq cost  CLI cost wall s
T03-A-grammar-r1         3  PASS   not used     4     3  Bashx3                     8        4366      30378   1278        101          34752     $0.0363   $0.0363    9.6
T08-A-grammar-r3         8  PASS   not used     4     3  Bashx3                     8        3707      31025   1349         48          34740     $0.0345   $0.0345   12.9
T15-A-grammar-r2        15  PASS   not used     4     3  Bashx3                     8        2439      32225   1323         59          34672     $0.0294   $0.0294   12.4

Observed: Independent validation of the final workspace:

T03-A-grammar-r1: format_ok=True inputs_unmodified=True intended_match=True matched=['status=completed|match=strip+lower|duplicates=dedup']
T08-A-grammar-r3: format_ok=True inputs_unmodified=True intended_match=True matched=['status=completed|match=strip+lower|duplicates=dedup']
T15-A-grammar-r2: format_ok=True inputs_unmodified=True intended_match=True matched=['status=completed|match=strip+lower|duplicates=dedup']

Observed: Commands run in T03-A-grammar-r1 (excerpt from the tool log, first line of each command):

$ ls -R . | head -50 && head -5 data/orders.csv data/fx_rates.csv && wc -l data/*.csv && cat check_format.py
$ cat data/fx_rates.csv; cat -A data/orders.csv
$ mkdir -p output && cat > /tmp/build.py <<'EOF'

Observed: Agent's final message in T03-A-grammar-r1 (excerpt):

I created `output/region_summary.csv`, and `python3 check_format.py` passed with "FORMAT OK: 4 rows". That script checks structure only, so I haven't verified the numbers against anything else.

| region | order_count | total_usd | avg_order_usd |
|---|---|---|---|
| West | 7 | 2074.27 | 296.32 |
| East | 7 | 2042.00 | 291.71 |
| South | 7 | 1518.44 | 216.92 |
| North | 7 | 1382.31 | 197.47 |

- **Status filter:** I kept only orders whose status is "completed" after trimming spaces and ignoring case. That picks up values like `COMPLETED ` and ` Completed`.
- **Duplicates:** Repeated `order_id`s (such as A1007, A1040, A1014 and A1028) were counted once. The repeated rows were identical.
- [...]

Observed: Task A (data processing) · Ambiguous rule · 3 of 3 runs met the full specification on the first pass

Observed: Exact prompt:

This folder contains data/orders.csv, a list of customer orders, and data/fx_rates.csv, which gives the US dollar value of one unit of each currency. Create output/region_summary.csv with exactly these columns, in this order: region, order_count, total_usd, avg_order_usd. Leave out any orders that shouldn't count. Convert every amount to US dollars with data/fx_rates.csv, round total_usd and avg_order_usd to two decimal places, and sort the rows by total_usd from highest to lowest. When you finish, run python3 check_format.py to check the file's structure.

Observed: All repetitions (execution order; tokens provider-reported; cost is an API-equivalent estimate):

trial                order pass1 correction turns tools  tool mix               input cache_wr_1h cache_read output (thinking) all input-side API-eq cost  CLI cost wall s
T05-A-ambiguous-r2       5  PASS   not used     4     3  Bashx3                     8        3481      30080   1333         37          33569     $0.0333   $0.0333    9.7
T10-A-ambiguous-r1      10  PASS   not used     4     3  Bashx3                     8        1243      32409   1378         38          33660     $0.0253   $0.0253   10.7
T19-A-ambiguous-r3      19  PASS   not used     4     3  Bashx3                     8        1328      32568   1400         49          33904     $0.0258   $0.0258   10.4

Observed: Independent validation of the final workspace:

T05-A-ambiguous-r2: format_ok=True inputs_unmodified=True intended_match=True matched=['status=completed|match=strip+lower|duplicates=dedup']
T10-A-ambiguous-r1: format_ok=True inputs_unmodified=True intended_match=True matched=['status=completed|match=strip+lower|duplicates=dedup']
T19-A-ambiguous-r3: format_ok=True inputs_unmodified=True intended_match=True matched=['status=completed|match=strip+lower|duplicates=dedup']

Observed: How each run resolved the vague sentence (manual coding of the final message; quotes are verbatim excerpts):

T05-A-ambiguous-r2: intended success; asked for clarification: False; stated its assumption: True; invited correction: False
  excerpt: "I made two judgment calls about which orders to leave out:"
T10-A-ambiguous-r1: intended success; asked for clarification: False; stated its assumption: True; invited correction: True
  excerpt: "If you meant refunded orders to count, or want duplicates kept, the numbers will change."
T19-A-ambiguous-r3: intended success; asked for clarification: False; stated its assumption: True; invited correction: True
  excerpt: "If you'd rather count pending orders, or count duplicates, the figures would change."

Observed: Commands run in T05-A-ambiguous-r2 (excerpt from the tool log, first line of each command):

$ cd /srv/xp-trial/ws; ls -R . | head -50; head -20 data/orders.csv; wc -l data/*.csv; cat data/fx_rates.csv; cat check_format.py
$ cd /srv/xp-trial/ws; python3 - <<'E'
$ cd /srv/xp-trial/ws; mkdir -p output; python3 - <<'E'

Observed: Agent's final message in T05-A-ambiguous-r2 (excerpt):

I wrote `output/region_summary.csv`, and `python3 check_format.py` reports `FORMAT OK: 4 rows`. That script only checks structure, not whether the numbers are right.

| region | order_count | total_usd | avg_order_usd |
|---|---|---|---|
| West | 7 | 2074.27 | 296.32 |
| East | 7 | 2042.00 | 291.71 |
| South | 7 | 1518.44 | 216.92 |
| North | 7 | 1382.31 | 197.47 |

I made two judgment calls about which orders to leave out:
- **Status:** I kept only completed orders and dropped pending, cancelled and refunded ones. The `status` column had inconsistent casing and padding (`COMPLETED `, ` Completed `), so I trimmed and lowercased it before comparing.
- **Duplicates:** Five order IDs appeared [...]

Observed: Task B (code repair) · Clear baseline · 3 of 3 runs met the full specification on the first pass

Observed: Exact prompt:

In this repository, prorate() in billing/proration.py returns wrong amounts, and python3 -m pytest -q shows failing tests. Fix the bug in billing/proration.py so the whole test suite passes. Both start_day and end_day count as billed days, and fractional cents must round half up, so exactly half a cent rounds up to the next whole cent. Do not edit any test file and do not change the function's signature.

Observed: All repetitions (execution order; tokens provider-reported; cost is an API-equivalent estimate):

trial                order pass1 correction turns tools  tool mix               input cache_wr_1h cache_read output (thinking) all input-side API-eq cost  CLI cost wall s
T06-B-clear-r1           6  PASS   not used     5     4  Bashx2, Editx1, Readx1     6        2144      20492    624          0          22642     $0.0189   $0.0189    6.4
T09-B-clear-r2           9  PASS   not used     5     4  Bashx2, Editx1, Readx1     6         494      22182    671          0          22682     $0.0131   $0.0131    6.8
T23-B-clear-r3          23  PASS   not used     5     4  Bashx2, Editx1, Readx1     6         572      22182    804          0          22760     $0.0148   $0.0148    8.9

Observed: Independent validation of the final workspace:

T06-B-clear-r1: visible_tests_pass=True (5 passed in 0.01s) ties_round_up=True [prorate(2989,1,15,30)=1495, prorate(14,1,1,28)=1] nearest_cent=True inclusive_31=True signature_unchanged=True tests_unchanged=True
T09-B-clear-r2: visible_tests_pass=True (5 passed in 0.01s) ties_round_up=True [prorate(2989,1,15,30)=1495, prorate(14,1,1,28)=1] nearest_cent=True inclusive_31=True signature_unchanged=True tests_unchanged=True
T23-B-clear-r3: visible_tests_pass=True (5 passed in 0.01s) ties_round_up=True [prorate(2989,1,15,30)=1495, prorate(14,1,1,28)=1] nearest_cent=True inclusive_31=True signature_unchanged=True tests_unchanged=True

Observed: Commands run in T06-B-clear-r1 (excerpt from the tool log, first line of each command):

$ python3 -m pytest -q 2>&1 | tail -40
$ python3 -m pytest -q 2>&1 | tail -5

Observed: Code change in T06-B-clear-r1 (excerpt of the workspace diff):

-    days = end_day - start_day
-    return round(amount_cents * days / days_in_month)
+    days = end_day - start_day + 1
+    return (2 * amount_cents * days + days_in_month) // (2 * days_in_month)

Observed: Agent's final message in T06-B-clear-r1 (excerpt):

I fixed `prorate()` in `billing/proration.py`, and the full suite now passes (5 passed).

There were two bugs:
- **Day count:** it used `end_day - start_day`, which left out the start day. It now uses `end_day - start_day + 1`, so both days are billed.
- **Rounding:** Python's `round()` rounds exact halves to the nearest even number, not up. I replaced it with integer arithmetic, `(2*amount*days + dim) // (2*dim)`. That rounds exact halves up and avoids float error.

I didn't touch the tests or the function signature.

Observed: Task B (code repair) · Spelling errors · 3 of 3 runs met the full specification on the first pass

Observed: Exact prompt:

In this repositry, prorate() in billing/proration.py retuns wrong amouts, and python3 -m pytest -q shows faling tests. Fix the bgu in billing/proration.py so the hole test suite pases. Both start_day and end_day cuont as biled days, and fractonal cents must rond haf up, so exacly half a cent rouns up to the nxt whole cent. Do not edti any tset file and do not chnage the functon's signture.

Observed: All repetitions (execution order; tokens provider-reported; cost is an API-equivalent estimate):

trial                order pass1 correction turns tools  tool mix               input cache_wr_1h cache_read output (thinking) all input-side API-eq cost  CLI cost wall s
T13-B-spelling-r3       13  PASS   not used     5     4  Bashx2, Editx1, Readx1     6        2199      20520    696          0          22725     $0.0199   $0.0199    7.4
T22-B-spelling-r2       22  PASS   not used     5     4  Bashx2, Editx1, Readx1     6         492      22224    697         42          22722     $0.0134   $0.0134    8.1
T24-B-spelling-r1       24  PASS   not used     5     4  Bashx2, Editx1, Readx1     6         500      22224    687          0          22730     $0.0133   $0.0133    7.3

Observed: Independent validation of the final workspace:

T13-B-spelling-r3: visible_tests_pass=True (5 passed in 0.01s) ties_round_up=True [prorate(2989,1,15,30)=1495, prorate(14,1,1,28)=1] nearest_cent=True inclusive_31=True signature_unchanged=True tests_unchanged=True
T22-B-spelling-r2: visible_tests_pass=True (5 passed in 0.01s) ties_round_up=True [prorate(2989,1,15,30)=1495, prorate(14,1,1,28)=1] nearest_cent=True inclusive_31=True signature_unchanged=True tests_unchanged=True
T24-B-spelling-r1: visible_tests_pass=True (5 passed in 0.01s) ties_round_up=True [prorate(2989,1,15,30)=1495, prorate(14,1,1,28)=1] nearest_cent=True inclusive_31=True signature_unchanged=True tests_unchanged=True

Observed: Commands run in T13-B-spelling-r3 (excerpt from the tool log, first line of each command):

$ python3 -m pytest -q 2>&1 | tail -40
$ python3 -m pytest -q 2>&1 | tail -10

Observed: Code change in T13-B-spelling-r3 (excerpt of the workspace diff):

-    days = end_day - start_day
-    return round(amount_cents * days / days_in_month)
+    days = end_day - start_day + 1
+    # Integer round-half-up; avoids float error and banker's rounding.
+    return (2 * amount_cents * days + days_in_month) // (2 * days_in_month)

Observed: Agent's final message in T13-B-spelling-r3 (excerpt):

I fixed `prorate()` in `billing/proration.py`, and the whole suite now passes (5 tests). I didn't touch any test file or the function signature.

There were two bugs:
- **Off-by-one day count:** it computed `end_day - start_day`, so `start_day` wasn't billed. It's now `end_day - start_day + 1`, so both ends count.
- **Wrong rounding:** Python's `round()` rounds exact halves to the nearest even number, and it works on floats. I replaced it with integer arithmetic, `(2 * amount_cents * days + days_in_month) // (2 * days_in_month)`. An exact half cent now rounds up, and there is no float error.

Observed: Task B (code repair) · Reduced grammar and punctuation · 3 of 3 runs met the full specification on the first pass

Observed: Exact prompt:

repository prorate() in billing/proration.py returns wrong amounts python3 -m pytest -q shows failing tests fix bug in billing/proration.py so whole test suite passes start_day and end_day both count as billed days fractional cents round half up exactly half cent rounds up to next whole cent dont edit any test file dont change function signature

Observed: All repetitions (execution order; tokens provider-reported; cost is an API-equivalent estimate):

trial                order pass1 correction turns tools  tool mix               input cache_wr_1h cache_read output (thinking) all input-side API-eq cost  CLI cost wall s
T01-B-grammar-r3         1  PASS   not used     6     5  Bashx3, Editx1, Readx1     8        2661      28649    996         61          31318     $0.0263   $0.0263    9.2
T07-B-grammar-r1         7  PASS   not used     5     4  Bashx2, Editx1, Readx1     6         516      22160    732         40          22682     $0.0138   $0.0138    7.0
T11-B-grammar-r2        11  PASS   not used     5     4  Bashx2, Editx1, Readx1     6         528      22160    740         52          22694     $0.0140   $0.0140    7.0

Observed: Independent validation of the final workspace:

T01-B-grammar-r3: visible_tests_pass=True (5 passed in 0.01s) ties_round_up=True [prorate(2989,1,15,30)=1495, prorate(14,1,1,28)=1] nearest_cent=True inclusive_31=True signature_unchanged=True tests_unchanged=True
T07-B-grammar-r1: visible_tests_pass=True (5 passed in 0.01s) ties_round_up=True [prorate(2989,1,15,30)=1495, prorate(14,1,1,28)=1] nearest_cent=True inclusive_31=True signature_unchanged=True tests_unchanged=True
T11-B-grammar-r2: visible_tests_pass=True (5 passed in 0.01s) ties_round_up=True [prorate(2989,1,15,30)=1495, prorate(14,1,1,28)=1] nearest_cent=True inclusive_31=True signature_unchanged=True tests_unchanged=True

Observed: Commands run in T01-B-grammar-r3 (excerpt from the tool log, first line of each command):

$ python3 -m pytest -q 2>&1 | tail -40
$ cat /srv/xp-trial/ws/tests/test_proration.py
$ python3 -m pytest -q 2>&1 | tail -5

Observed: Code change in T01-B-grammar-r3 (excerpt of the workspace diff):

-    days = end_day - start_day
-    return round(amount_cents * days / days_in_month)
+    days = end_day - start_day + 1
+    # Integer round-half-up: floor(amount * days / dim + 1/2)
+    return (2 * amount_cents * days + days_in_month) // (2 * days_in_month)

Observed: Agent's final message in T01-B-grammar-r3 (excerpt):

I fixed `prorate()` in `billing/proration.py`, and all 5 tests now pass. I didn't touch the tests or the function signature.

There were two bugs:
- **Day count:** it used `end_day - start_day`, so the end day wasn't billed. It's now `end_day - start_day + 1`, which counts both days.
- **Rounding:** Python's `round()` rounds exact halves to the nearest even number, so 2.5 becomes 2. The code now uses integer arithmetic, `(2*amount*days + dim) // (2*dim)`, which always rounds an exact half cent up. It also avoids float error.

None of the tests covers an exact half cent, so I didn't verify that case directly. For example, `prorate(1, 1, 1, 2)` is exactly 0.5 cents, and the new arithmetic [...]

Observed: Task B (code repair) · Ambiguous rule · 3 of 3 runs met the full specification on the first pass

Observed: Exact prompt:

In this repository, prorate() in billing/proration.py returns wrong amounts, and python3 -m pytest -q shows failing tests. Fix the bug in billing/proration.py so the whole test suite passes. Make sure the amounts come out right. Do not edit any test file and do not change the function's signature.

Observed: All repetitions (execution order; tokens provider-reported; cost is an API-equivalent estimate):

trial                order pass1 correction turns tools  tool mix               input cache_wr_1h cache_read output (thinking) all input-side API-eq cost  CLI cost wall s
T14-B-ambiguous-r3      14  PASS   not used     6     5  Bashx3, Editx1, Readx1     8        2606      28559    978        156          31173     $0.0259   $0.0259   11.9
T17-B-ambiguous-r2      17  PASS   not used     5     4  Bashx2, Editx1, Readx1     6        1667      21049    795         63          22722     $0.0188   $0.0188    8.0
T21-B-ambiguous-r1      21  PASS   not used     5     4  Bashx2, Editx1, Readx1     6         629      22190    895        169          22825     $0.0159   $0.0159   10.1

Observed: Independent validation of the final workspace:

T14-B-ambiguous-r3: visible_tests_pass=True (5 passed in 0.01s) ties_round_up=True [prorate(2989,1,15,30)=1495, prorate(14,1,1,28)=1] nearest_cent=True inclusive_31=True signature_unchanged=True tests_unchanged=True
T17-B-ambiguous-r2: visible_tests_pass=True (5 passed in 0.01s) ties_round_up=True [prorate(2989,1,15,30)=1495, prorate(14,1,1,28)=1] nearest_cent=True inclusive_31=True signature_unchanged=True tests_unchanged=True
T21-B-ambiguous-r1: visible_tests_pass=True (5 passed in 0.01s) ties_round_up=True [prorate(2989,1,15,30)=1495, prorate(14,1,1,28)=1] nearest_cent=True inclusive_31=True signature_unchanged=True tests_unchanged=True

Observed: How each run resolved the vague sentence (manual coding of the final message; quotes are verbatim excerpts):

T14-B-ambiguous-r3: intended success; asked for clarification: False; stated its assumption: True; invited correction: False
  excerpt: "The rounding change goes beyond what the failing tests needed, since none of them hit an exact half. I made it because you asked for the amounts to come out right."
T17-B-ambiguous-r2: intended success; asked for clarification: False; stated its assumption: False; invited correction: False
  excerpt: "I didn't add cases for exact half-cent amounts, so the rounding change isn't covered by a test."
T21-B-ambiguous-r1: intended success; asked for clarification: False; stated its assumption: False; invited correction: False
  excerpt: "No test covers an exact half-cent, so that behavior is untested."

Observed: Commands run in T14-B-ambiguous-r3 (excerpt from the tool log, first line of each command):

$ python3 -m pytest -q 2>&1 | tail -40
$ cat tests/test_proration.py
$ python3 -m pytest -q 2>&1 | tail -5

Observed: Code change in T14-B-ambiguous-r3 (excerpt of the workspace diff):

-    days = end_day - start_day
-    return round(amount_cents * days / days_in_month)
+    days = end_day - start_day + 1  # both start_day and end_day are billed
+    # Integer half-up rounding to the nearest cent, avoiding float error and banker's rounding.
+    return (2 * amount_cents * days + days_in_month) // (2 * days_in_month)

Observed: Agent's final message in T14-B-ambiguous-r3 (excerpt):

I fixed `prorate()` in `billing/proration.py`, and all 5 tests now pass. I didn't touch the tests or the signature.

There were two problems:
- **Off-by-one day count:** `end_day - start_day` left out the last day, so billing days 1–30 of a 30-day month charged 29 days. The count is now `end_day - start_day + 1`, so both the start and end day are billed.
- **Rounding:** `round()` on a float rounds exact halves to the nearest even number. I replaced it with integer half-up rounding, `(2*amount*days + dim) // (2*dim)`. This rounds to the nearest cent without float error, and 1000 over 30 days gives 33 and 2000 gives 67, as the tests expect.

The rounding change goes beyond what the failing [...]

Calculated estimate: Projection for 1,000 comparable tasks (calculated, not observed)

Calculated estimate: Every projection and bound below follows the listed assumptions; the setup, historical prices and observed success counts are the recorded inputs.

condition   projected cost  successes at observed rate  observed  95% lower bound (one-sided)
clear               $24.15                        1000       6/6                        0.607
spelling            $24.09                        1000       6/6                        0.607
grammar             $25.74                        1000       6/6                        0.607
ambiguous           $24.18                        1000       6/6                        0.607

Assumptions:
- Workload: 1,000 tasks of the same size and shape as this experiment, half like Task A (CSV aggregation) and half like Task B (one-function bug fix with a 5-test suite).
- Same model (claude-sonnet-5-5), Claude Code 2.1.287 headless, effort high, same tool set and limits.
- Cache behavior like the experiment's: sessions start close enough together that the shared system-prompt prefix is usually already cached.
- Official list prices retrieved 2026-10-02: $2/MTok input, $10/MTok output, $4/MTok 1-hour cache write, $2.50/MTok 5-minute cache write, $0.20/MTok cache read. No batch, priority-tier or negotiated discounts.
- Success rate equals the observed final success rate in that condition; with n=6 per condition this is very uncertain (see lower bound).