# Results

Both runs through OpenRouter `/api/v1/systemone`, from one machine in Europe. Tables are generated by `bench/analyze.mjs` and `bench/cost.mjs` over `bench/combine.mjs` output.

| Arm | Run (UTC) | Model id served | applet-skeleton version |
| --- | --- | --- | --- |
| jev-1002 | 2026-10-02 09:40 to 09:58 | typesafe/jev-1.13-20260917 | 1 |
| jev-1006 | 2026-10-06 22:24 to 22:28 | typesafe/jev-1.13-20260917 | 3 |
| luna | 2026-10-06 22:24 to 22:28 | openai/gpt-6-luna-decisions-20261006 | 3 |

All other sets were at version 1 in both runs. Per run: one rep of every set, a second rep of chat-route, chat-sections and tag-pick, arms alternating which goes first. applet-skeleton quality is scored on the 63 items of the blind adjudication only.

## Non-200 calls

| At | Set | Item | Status | Message |
| --- | --- | --- | --- | --- |
| 2026-10-06T22:26:00.133Z | applet-skeleton | skel-h6-07 | 503 | HTTP 503: {"error":{"message":"Decision inference is unavailable.","type":"server_error","param":null,"code":null}} |
| 2026-10-06T22:26:33.871Z | chat-route | chat-route-30 | 502 | OpenAI refused to answer question "shape" |
| 2026-10-06T22:26:34.726Z | chat-route | chat-route-31 | 503 | HTTP 503: {"error":{"message":"Decision inference is unavailable.","type":"server_error","param":null,"code":null}} |
| 2026-10-06T22:27:41.641Z | chat-route | chat-route-30 | 502 | OpenAI refused to answer question "shape" |

No other arm returned a non-200.

## Tables

### Latency (HTTP 200 calls, wall clock from this machine)

| set | budget | arm | n | errors | p50 | p90 | p95 | max | mean | in budget |
|---|---|---|---|---|---|---|---|---|---|---|
| applet-skeleton | 4000 | jev-1002 | 120 | 0 | 302 | 371 | 384 | 469 | 315 | 100.0% |
| applet-skeleton | 4000 | jev-1006 | 120 | 0 | 386 | 517 | 627 | 896 | 415 | 100.0% |
| applet-skeleton | 4000 | luna | 120 | 1 | 418 | 818 | 917 | 4522 | 547 | 97.5% |
| chat-route | 1000 | jev-1002 | 80 | 0 | 292 | 362 | 400 | 488 | 310 | 100.0% |
| chat-route | 1000 | jev-1006 | 80 | 0 | 284 | 331 | 414 | 652 | 298 | 100.0% |
| chat-route | 1000 | luna | 80 | 3 | 297 | 383 | 712 | 3946 | 399 | 93.8% |
| chat-chart | 4000 | jev-1002 | 30 | 0 | 290 | 360 | 457 | 498 | 313 | 100.0% |
| chat-chart | 4000 | jev-1006 | 30 | 0 | 278 | 318 | 408 | 455 | 292 | 100.0% |
| chat-chart | 4000 | luna | 30 | 0 | 276 | 346 | 418 | 3189 | 381 | 100.0% |
| chat-sections | 1500 | jev-1002 | 16 | 0 | 303 | 363 | 378 | 378 | 315 | 100.0% |
| chat-sections | 1500 | jev-1006 | 16 | 0 | 342 | 449 | 558 | 558 | 355 | 100.0% |
| chat-sections | 1500 | luna | 16 | 0 | 379 | 479 | 950 | 950 | 410 | 100.0% |
| name-pick | 4000 | jev-1002 | 12 | 0 | 313 | 375 | 386 | 386 | 326 | 100.0% |
| name-pick | 4000 | jev-1006 | 12 | 0 | 268 | 322 | 337 | 337 | 278 | 100.0% |
| name-pick | 4000 | luna | 12 | 0 | 277 | 357 | 470 | 470 | 294 | 100.0% |
| tag-pick | 4000 | jev-1002 | 28 | 0 | 316 | 365 | 369 | 420 | 321 | 100.0% |
| tag-pick | 4000 | jev-1006 | 28 | 0 | 311 | 369 | 382 | 401 | 314 | 100.0% |
| tag-pick | 4000 | luna | 28 | 0 | 388 | 438 | 467 | 488 | 390 | 100.0% |

jev-1002 overall: n=286, p50 302 ms, p95 394 ms, over 4 s 0, over 9 s (relay timeout) 0

jev-1006 overall: n=286, p50 311 ms, p95 517 ms, over 4 s 0, over 9 s (relay timeout) 0

luna overall: n=282, p50 346 ms, p95 860 ms, over 4 s 2, over 9 s (relay timeout) 0
luna non-200: [[503,null,"HTTP 503: {\"error\":{\"message\":\"Decision inference is unavailable.\",\"type\":\"server_error\",\"param\":null,\"code\":null}}"],[502,null,"OpenAI refused to answer question \"shape\""],[503,null,"HTTP 503: {\"error\":{\"message\":\"Decision inference is unavailable.\",\"type\":\"server_error\",\"param\":null,\"code\":null}}"],[502,null,"OpenAI refused to answer question \"shape\""]]

### Validity (answers the relay would pass on)

| set | arm | questions | valid | invalid reasons |
|---|---|---|---|---|
| applet-skeleton | jev-1002 | 1080 | 99.3% | {"probabilities":7,"choice-not-argmax":1} |
| applet-skeleton | jev-1006 | 1080 | 99.3% | {"probabilities":8} |
| applet-skeleton | luna | 1071 | 100.0% | {} |
| chat-route | jev-1002 | 160 | 99.4% | {"probabilities":1} |
| chat-route | jev-1006 | 160 | 100.0% | {} |
| chat-route | luna | 154 | 100.0% | {} |
| chat-chart | jev-1002 | 30 | 100.0% | {} |
| chat-chart | jev-1006 | 30 | 100.0% | {} |
| chat-chart | luna | 30 | 100.0% | {} |
| chat-sections | jev-1002 | 384 | 100.0% | {} |
| chat-sections | jev-1006 | 384 | 100.0% | {} |
| chat-sections | luna | 384 | 100.0% | {} |
| name-pick | jev-1002 | 36 | 100.0% | {} |
| name-pick | jev-1006 | 36 | 100.0% | {} |
| name-pick | luna | 36 | 100.0% | {} |
| tag-pick | jev-1002 | 840 | 100.0% | {} |
| tag-pick | jev-1006 | 840 | 100.0% | {} |
| tag-pick | luna | 840 | 100.0% | {} |

### Quality against labels, with the shipped gates

Choice: acted = confidence at or above the gate; precision = right when acted; argmax = right ignoring the gate. Noul: true when noul at or above the gate.

| set | question kind | arm | labelled | acted / predicted true | precision | recall | argmax or accuracy | Brier |
|---|---|---|---|---|---|---|---|---|
| applet-skeleton | choice | jev-1002 | 63 | 57.1% | 86.1% | - | 79.4% | - |
| applet-skeleton | choice | jev-1006 | 63 | 57.1% | 94.4% | - | 73.0% | - |
| applet-skeleton | choice | luna | 62 | 67.7% | 85.7% | - | 74.2% | - |
| chat-route | choice | jev-1002 | 60 | 95.0% | 100.0% | - | 96.7% | - |
| chat-route | choice | jev-1006 | 60 | 96.7% | 100.0% | - | 100.0% | - |
| chat-route | choice | luna | 57 | 68.4% | 100.0% | - | 100.0% | - |
| chat-route | noul | jev-1002 | 78 | 25.6% | 100.0% | 100.0% | 100.0% | 0.027 |
| chat-route | noul | jev-1006 | 78 | 25.6% | 100.0% | 100.0% | 100.0% | 0.027 |
| chat-route | noul | luna | 75 | 24.0% | 100.0% | 90.0% | 97.3% | 0.019 |
| chat-chart | noul | jev-1002 | 29 | 44.8% | 100.0% | 100.0% | 100.0% | 0.034 |
| chat-chart | noul | jev-1006 | 29 | 44.8% | 100.0% | 100.0% | 100.0% | 0.036 |
| chat-chart | noul | luna | 29 | 44.8% | 100.0% | 100.0% | 100.0% | 0.020 |
| chat-sections | noul | jev-1002 | 212 | 19.8% | 100.0% | 84.0% | 96.2% | 0.033 |
| chat-sections | noul | jev-1006 | 212 | 20.3% | 97.7% | 84.0% | 95.8% | 0.034 |
| chat-sections | noul | luna | 212 | 17.9% | 100.0% | 76.0% | 94.3% | 0.050 |
| name-pick | choice | jev-1002 | 10 | 80.0% | 100.0% | - | 100.0% | - |
| name-pick | choice | jev-1006 | 10 | 80.0% | 100.0% | - | 100.0% | - |
| name-pick | choice | luna | 10 | 30.0% | 100.0% | - | 100.0% | - |
| name-pick | noul | jev-1002 | 16 | 75.0% | 100.0% | 100.0% | 100.0% | 0.020 |
| name-pick | noul | jev-1006 | 16 | 75.0% | 100.0% | 100.0% | 100.0% | 0.021 |
| name-pick | noul | luna | 16 | 75.0% | 100.0% | 100.0% | 100.0% | 0.035 |
| tag-pick | noul | jev-1002 | 510 | 13.9% | 87.3% | 93.9% | 97.5% | 0.032 |
| tag-pick | noul | jev-1006 | 510 | 14.1% | 86.1% | 93.9% | 97.3% | 0.032 |
| tag-pick | noul | luna | 510 | 11.8% | 90.0% | 81.8% | 96.5% | 0.033 |

### Bursts (n concurrent requests)

- jev-1002 chat-route n=10: wall 500 ms, statuses [200,200,200,200,200,200,200,200,200,200], p50 444 max 485
- jev-1002 chat-sections n=8: wall 721 ms, statuses [200,200,200,200,200,200,200,200], p50 650 max 709
- jev-1006 chat-route n=10: wall 521 ms, statuses [200,200,200,200,200,200,200,200,200,200], p50 470 max 521
- jev-1006 chat-sections n=8: wall 723 ms, statuses [200,200,200,200,200,200,200,200], p50 510 max 695
- luna chat-route n=10: wall 1553 ms, statuses [200,200,200,200,200,200,200,200,200,200], p50 456 max 1530
- luna chat-sections n=8: wall 873 ms, statuses [200,200,200,200,200,200,200,200], p50 550 max 846

### Cost

| set | arm | calls | mean input tokens | mean output tokens | mean cost per call (USD) | per 1,000 calls |
|---|---|---|---|---|---|---|
| applet-skeleton | jev-1002 | 120 | 1389 | 463 | 5.84e-5 | 0.058 |
| applet-skeleton | jev-1006 | 120 | 2436 | 530 | 1.02e-4 | 0.102 |
| applet-skeleton | luna | 119 | 2507 | 0 | 2.51e-4 | 0.251 |
| chat-chart | jev-1002 | 30 | 483 | 21 | 2.03e-5 | 0.020 |
| chat-chart | jev-1006 | 30 | 483 | 21 | 2.03e-5 | 0.020 |
| chat-chart | luna | 30 | 348 | 0 | 3.48e-5 | 0.035 |
| chat-route | jev-1002 | 80 | 528 | 72 | 2.22e-5 | 0.022 |
| chat-route | jev-1006 | 80 | 528 | 72 | 2.22e-5 | 0.022 |
| chat-route | luna | 77 | 446 | 0 | 4.46e-5 | 0.045 |
| chat-sections | jev-1002 | 16 | 2489 | 436 | 1.05e-4 | 0.105 |
| chat-sections | jev-1006 | 16 | 2489 | 436 | 1.05e-4 | 0.105 |
| chat-sections | luna | 16 | 5317 | 0 | 5.32e-4 | 0.532 |
| name-pick | jev-1002 | 12 | 885 | 126 | 3.72e-5 | 0.037 |
| name-pick | jev-1006 | 12 | 885 | 126 | 3.72e-5 | 0.037 |
| name-pick | luna | 12 | 877 | 0 | 8.77e-5 | 0.088 |
| tag-pick | jev-1002 | 28 | 2051 | 544 | 8.62e-5 | 0.086 |
| tag-pick | jev-1006 | 28 | 2051 | 544 | 8.62e-5 | 0.086 |
| tag-pick | luna | 28 | 5768 | 0 | 5.77e-4 | 0.577 |

jev-1002: 286 calls, total USD 0.0139

jev-1006: 286 calls, total USD 0.0192

luna: 282 calls, total USD 0.0600

## Two tag-pick items over time (luna, t01)

| Item | Tile / tag | 2026-10-06 22:24 to 22:28 (2 reps) | 2026-10-07 ~11:10 to 11:15 (4 calls) |
| --- | --- | --- | --- |
| tag-pick-10 | Mortgage calculator: estimate your monthly payment (Bankrate) / house hunting | 0.21, 0.21 | 0, 0, 0, 0 |
| tag-pick-04 | S&P 500 index: live chart and news (Bloomberg) / investing | 0.56, 0.56 | 0.47, 0.47, 0.47, 0.47 |

Same served model id in both windows. On 2026-10-06 t01 was one of the 30 questions of the bench request; on 2026-10-07 it was asked alone with the same state and text. The 2026-10-07 calls are in `results/2026-10-07-framing-ablation.log` and `luna-tag-predicates.md` (raw responses).

## Framing variants for the same two items (2026-10-07 ~11:10 UTC)

One question per call. "Full tag list" is the bench state; "one tag" is `{"title", "tag"}`. "Bench wording" is the tag-pick question text; "short wording" is "Should the provided tag be used for the given title of a browser tile?".

| Item | Framing | luna | jev |
| --- | --- | --- | --- |
| tag-pick-10 | full tag list, bench wording | 0, 0 | 0.70, 0.69 |
| tag-pick-10 | full tag list, short wording | 0.99, 0.99 | 0.55, 0.56 |
| tag-pick-10 | one tag, bench wording | 0.94, 0.94 | 0.79, 0.78 |
| tag-pick-10 | one tag, short wording | 0.99, 0.99 | 0.75, 0.76 |
| tag-pick-04 | full tag list, bench wording | 0.47, 0.47 | 0.90, 0.91 |
| tag-pick-04 | full tag list, short wording | 0.86, 0.86 | 0.47, 0.50 |
| tag-pick-04 | one tag, bench wording | 1, 1 | 0.91, 0.90 |
| tag-pick-04 | one tag, short wording | 1, 1 | 0.88, 0.87 |

Labels for both items: true.