# R1 Evaluation Report — Model-Aware Context Economics

> Generated by `tests/r1-report.spec.ts` on 2026-09-29 · window 16000 tokens · 64 steps · keyless (no provider credentials).
> Every number below is produced by running the suites named in this repository;
> nothing is hand-entered. Re-generate with `npx vitest run tests/r1-report.spec.ts`.

Baseline: R0-C at `12a5842`. This report covers R1-A (measurement), R1-B
(workloads + counterfactual ROI), R1-D (policy compiler), R1-E (Pareto).

---

## 1. Attribution: where the tokens go

| Workload | EF folds | EF total | Basic folds | Basic total | Dominant source | Share |
|---|---:|---:|---:|---:|---|---:|
| W1-narrative-heavy | 47 | 183132 | 21 | 73689 | `checkpoint-framing` | 64.0% |
| W2-state-rich | 7 | 102831 | 5 | 83396 | `raw-assistant` | 30.9% |
| W3-tool-heavy | 63 | 322645 | 63 | 87340 | `checkpoint-framing` | 60.6% |
| W4-recall-heavy | 12 | 100371 | 8 | 72696 | `checkpoint-framing` | 31.3% |
| W5-multi-agent | 58 | 267286 | 32 | 89531 | `checkpoint-framing` | 62.6% |

Attribution reconciles exactly with the metered prompt total on every arm
(asserted before this file is written). Shares are of total prompt tokens.

### Per-workload breakdown

#### W1-narrative-heavy

*Long prose trajectory — the R0-C baseline, for continuity of comparison.*

| Token source | Tokens | Share |
|---|---:|---:|
| checkpoint-framing | 117273 | 64.0% |
| raw-assistant | 29886 | 16.3% |
| checkpoint-identity | 14508 | 7.9% |
| checkpoint-leaf-rationale | 13299 | 7.3% |
| raw-user | 8166 | 4.5% |
| **total** | **183132** | 100% |

#### W2-state-rich

*Many constraint/decision anchors with supersession, so leaf checkpoints carry real machine state.*

| Token source | Tokens | Share |
|---|---:|---:|
| raw-assistant | 31735 | 30.9% |
| raw-user | 26710 | 26.0% |
| checkpoint-leaf-state | 23083 | 22.4% |
| checkpoint-framing | 17209 | 16.7% |
| checkpoint-identity | 2136 | 2.1% |
| checkpoint-leaf-rationale | 1958 | 1.9% |
| **total** | **102831** | 100% |

#### W3-tool-heavy

*Large read/grep/test/build results, to test whether tool output dominates context cost.*

| Token source | Tokens | Share |
|---|---:|---:|
| checkpoint-framing | 195552 | 60.6% |
| raw-tool-result | 78249 | 24.3% |
| checkpoint-identity | 24192 | 7.5% |
| checkpoint-leaf-rationale | 22176 | 6.9% |
| raw-assistant | 1987 | 0.6% |
| raw-user | 489 | 0.2% |
| **total** | **322645** | 100% |

#### W4-recall-heavy

*Folded material becomes relevant again; measures what recall costs when it is actually used.*

| Token source | Tokens | Share |
|---|---:|---:|
| checkpoint-framing | 31428 | 31.3% |
| recall | 30645 | 30.5% |
| raw-assistant | 17351 | 17.3% |
| raw-user | 13495 | 13.4% |
| checkpoint-identity | 3888 | 3.9% |
| checkpoint-leaf-rationale | 3564 | 3.6% |
| **total** | **100371** | 100% |

#### W5-multi-agent

*Child-agent reports folded into the parent, to size delegation payload cost.*

| Token source | Tokens | Share |
|---|---:|---:|
| checkpoint-framing | 167422 | 62.6% |
| raw-tool-result | 57403 | 21.5% |
| checkpoint-identity | 20712 | 7.7% |
| checkpoint-leaf-rationale | 18986 | 7.1% |
| raw-assistant | 2124 | 0.8% |
| raw-user | 639 | 0.2% |
| **total** | **267286** | 100% |

## 2. Regime sensitivity: the dominant cost is not a constant

The same workload changes its dominant cost with fold pressure. A single
number quoted without its regime would be a threshold artifact.

| Workload | Aggressive folds | framing | raw history | Realistic folds | framing | raw history |
|---|---:|---:|---:|---:|---:|---:|
| W1-narrative-heavy | 47 | 64.0% | 20.8% | 4 | 3.8% | 95.3% |
| W2-state-rich | 7 | 16.7% | 56.8% | 1 | 0.4% | 98.9% |
| W3-tool-heavy | 63 | 60.6% | 25.0% | 9 | 7.8% | 90.4% |
| W4-recall-heavy | 12 | 31.3% | 30.7% | 2 | 1.0% | 54.7% |
| W5-multi-agent | 58 | 62.6% | 22.5% | 6 | 5.4% | 93.3% |

**Structural finding.** The frozen prefix is monotonically non-decreasing
(EF may never re-fold a frozen checkpoint, plan §13). Once it alone exceeds
the pressure threshold, every subsequent step is over threshold, each fold
can only compact the new tail, and one checkpoint is produced per step —
each paying the full ~116-token framing preamble. At `aggressive` pressure
EF folds 51 times against Basic's 21 on an identical workload AND identical
digest text: a 3.3x total-token gap that is architectural, not narrative.

This is the cost of the frontier invariant itself, and it is what an
amortized rebase policy exists to bound.

## 3. Counterfactual upper bounds

Fraction of total prompt tokens removable under each arm's stated
idealization. These are UPPER BOUNDS, not achievements.

| Candidate | W1 | W2 | W3 | W4 | W5 | Risk | Kind |
|---|---:|---:|---:|---:|---:|---|---|
| E-delta-oracle | 0.0% | 15.3% | 0.0% | 0.0% | 0.0% | medium | new-mechanism |
| E-M1-oracle | 0.0% | 0.0% | 23.3% | 0.0% | 20.6% | high | new-mechanism |
| E-M5-oracle | 53.3% | 28.6% | 50.6% | 25.5% | 52.2% | high | new-mechanism |
| E-adaptive-root-oracle | 67.6% | 18.3% | 65.4% | 28.5% | 67.2% | low | policy-change |
| E-framing-oracle | 60.7% | 11.4% | 58.7% | 25.5% | 60.4% | low | policy-change |

### Idealization assumptions

- **E-delta-oracle (idealized Delta Leaf)** — Every leaf checkpoint after the first carries only state changes: the full Current snapshot is charged once and the recurring full-state payload is removed from all later checkpoints. Rationale and framing are unchanged, and no correctness cost is charged.
- **E-M1-oracle (idealized typed tool results)** — Every raw tool result is replaced by a typed summary plus an artifact ref retaining 4.0% of its tokens (a 25x reduction). No information is assumed lost and no extra retrieval cost is charged.
- **E-M5-oracle (idealized generational merge)** — All frozen leaf checkpoints merge into a single root snapshot retaining 25% of their combined tokens, with zero merge cost on the request path and no state loss. An upper bound on any real generational fold.
- **E-adaptive-root-oracle (idealized amortized rebase)** — A root rebase fires whenever its payback horizon is met, collapsing the frozen prefix to a single snapshot at no request-path cost after the rebase itself. An upper bound on an amortized-cost rebase policy over the EXISTING root fold.
- **E-framing-oracle (deduplicated handoff preamble)** — The repeated per-checkpoint preamble and summary wrapper are emitted once per surface instead of once per checkpoint. Every checkpoint keeps its marker, state, rationale, and recall pointer, so no identity or recoverability is assumed lost. An upper bound on what framing deduplication could save.

### Decision (see docs/11_R1B_ROUTE_SELECTION_GATE.md)

- **Delta Leaf is rejected.** Weakest on every workload (3.7–16.6%); it does
  not justify a new correctness seam, and it targets the smaller half of the
  checkpoint cost.
- **Routed to R1-D (amortized root).** Highest bound, lowest risk, and a
  policy change over existing machinery rather than a new mechanism.
- **M1 pays only where tool output exists** (0% on W1/W2, ~20% on W3), which
  is exactly why workload shape had to be varied.

## 4. Money is profile-dependent, and the token share is not the bill

The same framing-dedup saving on W3, priced under each profile. Removed
frozen-prefix tokens are charged at the WARM price: they are exactly the
tokens that would otherwise have been billed as cache hits.

| Profile | ρ | h = 1.0 | h = 0.8 | h = 0.5 |
|---|---:|---:|---:|---:|
| deepseek-flash-2026-09 | 0.020 | $0.0011 | $0.0115 | $0.0271 |
| deepseek-pro-2026-09 | 0.033 | $0.0035 | $0.0240 | $0.0548 |
| openai-gpt-5.6-2026-09 | 0.100 | $0.0237 | $0.0663 | $0.1302 |
| synthetic-no-cache | 1.000 | $0.1894 | $0.1894 | $0.1894 |

A ~67% token saving is worth very different money on a cache-dominant model
than on one that bills cache writes. Any "N% cheaper" claim without its
profile and realization rate is not a cost claim.

## 5. Policy decisions per profile (R1-D)

| Profile | Regime | Action | Break-even requests | Reason |
|---|---|---|---:|---|
| deepseek-flash-2026-09 | cache-dominant | `root` | 9.9 | rebase break-even 9.9 requests <= horizon 200 (regime cache-dominant, rho_eff 0.020) |
| openai-gpt-5.6-2026-09 | hybrid | `root` | 0.4 | rebase break-even 0.4 requests <= horizon 200 (regime hybrid, rho_eff 0.100) |
| synthetic-no-cache | token-dominant | `root` | 0.1 | rebase break-even 0.1 requests <= horizon 200 (regime token-dominant, rho_eff 1.000) |

The compiler is deterministic, pure, and contains no provider branch: it
reads a profile's numbers. Hard overrides (overflow, pressure) beat
economics unconditionally — economics never buys correctness.

## 6. Pareto: a frontier, not a winner

Cost vs peak context for the measured arms. **Task success is not
measurable in the keyless tier**, so it is held constant and no success
claim is made here.

| Policy | Effective cost | Peak context | Task success |
|---|---:|---:|---:|
| W1-narrative-heavy Basic | 73689.0000 | 2043 | 100.0% |
| W4-recall-heavy Basic | 72696.0000 | 2351 | 100.0% |
| W5-multi-agent Basic | 89531.0000 | 1932 | 100.0% |

Dominated (some frontier policy is at least as good everywhere):

| Policy | Effective cost | Peak context | Task success |
|---|---:|---:|---:|
| W1-narrative-heavy EF | 183132.0000 | 5986 | 100.0% |
| W2-state-rich EF | 102831.0000 | 2396 | 100.0% |
| W3-tool-heavy EF | 322645.0000 | 8804 | 100.0% |
| W4-recall-heavy EF | 100371.0000 | 2355 | 100.0% |
| W5-multi-agent EF | 267286.0000 | 7817 | 100.0% |
| W2-state-rich Basic | 83396.0000 | 2333 | 100.0% |
| W3-tool-heavy Basic | 87340.0000 | 2201 | 100.0% |

### Reading this honestly

With task success held constant, Basic dominates EF on both axes on every
workload. That is the correct reading of these numbers and it should not
be softened — but it is also not an argument that EF is worthless, because
the premise (equal task success) is exactly what R1 cannot yet test.

The entire justification for Epistemic Fold is that success is NOT equal:
EF exists so a long-horizon agent does not forget constraints, obligations,
and verified failures that Basic's lossy narrative rewrite would drop. If
that holds, EF buys success with tokens and the real comparison is
cost-to-success, not cost. If it does not hold, EF is strictly worse and
should be abandoned.

**This report cannot decide between those two cases.** It establishes the
cost side precisely and leaves the success side open. Flipping any
production default on the strength of §4-§6 alone would be exactly the
error docs/11 §21 warns about.

## 7. Open items

| Item | Status |
|---|---|
| Live behavioral validation (task success) | ⚠️ EXECUTED, NULL RESULT — see [docs/13](13_R1_LIVE_BEHAVIORAL_RESULTS.md) |
| Provider cache realization (`h`) measured, not assumed | ✅ measured h = 0.910 on the configured route |
| Latency (TTFT, request/compaction/recall) | ⏸ OPEN — requires a live provider |
| Production default flip to a provider-aware policy | ⏸ BLOCKED on a POSITIVE behavioral result |

### The live tier ran, and it did not find a behavioral advantage

The opt-in live subset (`npm run eval:r1-live`, route taken from this
workspace's ZCode configuration) executed 64 paired trials: **EF 32/32
(100%) vs Basic 31/32 (97%)**. One trial is not evidence. The honest
conclusion is the null one — at this scale, on this model, Basic's lossy
summary retained every task-critical fact the cases probed.

This does not refute the premise that lossy summarization eventually drops
what matters; it shows these four cases at this compression ratio do not
reach that regime. Finding the regime where it DOES fail is now the most
valuable next experiment, ahead of any further folding mechanism.

The live tier did produce two decisive results:

1. **Measured cache realization h = 0.910** (2176 hit / 253 miss on an
   identical stable prefix), giving ρ_eff = 0.118 against a headline
   ρ = 0.020 — **5.9× worse than the profile claims**. R1-A's correction
   is confirmed against a real provider, not just argued.
2. **A production bug no keyless test could see**: failure anchors rendered
   without their description, so the model saw that something was
   unresolved but not what. Fixed in `src/renderer.ts` with a regression
   assertion.

Per docs/11 §21, no production default may change on token economics alone:
a 10% cheaper policy that loses 5% of task success may be worthless. R1
therefore closes on measurement, counterfactual bounds, and a compiled
policy — with the behavioral gate explicitly still open.

## 8. Correctness hard gates

Unchanged from R0 and not traded for economics: ALR = 0, SSR = 0,
ProvenanceCoverage = 100%, CrossSessionLeak = 0, ExactRecallMismatch = 0,
CriticalConstraintViolation = 0. The Pareto module EXCLUDES any candidate
failing these rather than scoring it down (`eval/src/pareto.ts`).