# R2 Evaluation Report — Baseline-Dominant Context Compression > Generated by `tests/r2e-dominance.spec.ts` (matrix) and the R2 stage suites. > Window 16,000 tokens · 64 steps · keyless · realization h = 0.910 (measured > live in R1, not assumed). Baseline: R1 merged at `78c347a`. R2 commits: R2-A `8ad757b`, R2-B `ce8a1c4`, R2-C `919cbd0`, R2-D `349d93f`. --- ## 1. The objective, and the verdict R2's exit condition is a combination: ``` Q_EF >= Q_Basic - ε AND C_EF < C_Basic ``` **Verdict: the first half holds, the second does not.** | Gate | Result | |---|---| | No worse in quality (BQR ≥ 1) | ✅ 40/40 cells (see §5 for what this can and cannot mean) | | Cheaper than Basic (BCR < 1) | ❌ **0/40 cells** | | **Gate P (parity)** | ❌ **FAIL** | | **Gate D (dominance)** | ❌ **FAIL** | **Mean BCR = 1.451** across the full matrix. On the recommended DeepSeek Flash profile specifically: | Policy | Mean BCR | Per-workload | |---|---:|---| | E0 legacy | 1.276 | 1.242, 1.186, 1.357, 1.164, 1.431 | | **E3 full (R2)** | **1.149** | 1.090, 1.174, 1.123, 1.207, 1.151 | So R2 delivered a large improvement — **BCR 1.276 → 1.149, a 38% reduction in the excess cost over Basic** — and still did not reach the goal. EF remains about 15% more expensive than Basic on the profile where it looks best. --- ## 2. What R2 fixed, and by how much Three mechanisms, each measured against the arm before it: | Stage | Mechanism | Measured effect (W1, 64 steps) | |---|---|---| | R2-A | Pressure attribution | diagnostic only; exposed the loop | | R2-B | Economic leaf admission | folds 50 → 16, but **total 236K → 428K (regression)** | | R2-C | Rebase handoff | **total → 103K**, peak 7,896 → 2,547, frozen 7,550 → 1,057 | | R2-D | Checkpoint surface diet | checkpoint body 41 → 23 tokens (−44%) | R2-C is the load-bearing fix: it made EF **2.3× cheaper than its own legacy mode** and cut peak context by 68%. R2-B alone was a 1.8× regression, and that negative result is pinned as a test so it cannot be forgotten. --- ## 3. Why EF is still not cheaper — the structural cause Decomposing each arm's measured warm/cold split on the Flash profile: | Workload | Basic cold | EF cold | cold Δ | Basic warm | EF warm | warm Δ | |---|---:|---:|---:|---:|---:|---:| | W1 | 38,983 | 39,888 | +905 | 34,643 | 61,561 | **+26,918** | | W2 | 10,619 | 11,932 | +1,313 | 72,720 | 90,283 | +17,563 | | W3 | 86,317 | 88,120 | +1,803 | 960 | 81,533 | **+80,573** | | W4 | 18,417 | 20,012 | +1,595 | 54,219 | 80,943 | +26,724 | | W5 | 58,653 | 63,123 | +4,470 | 30,815 | 75,655 | +44,840 | **EF's cold-token overhead is nearly zero (+905 to +4,470). Its entire disadvantage is warm tokens: it carries 1.5× to 85× more of them.** The cause is the frontier invariant. Basic re-folds its single checkpoint, so its frozen prefix stays small and constant. EF may never re-fold a frozen checkpoint, so each fold appends to a chain that only a rebase can collapse — and a rebase makes everything cold again (measured: greedy rebasing drove warm tokens to 1,376 and was *worse*). This is a genuine tension with no free resolution: - **Few rebases** → the chain grows → more warm tokens than Basic. - **Many rebases** → the prefix keeps changing → warm tokens stop being warm. A rebase cadence sweep on W1 confirmed the interior optimum is flat and still above 1: | Rebase every N steps | 0 (never) | 4 | 8 | 16 | 32 | |---|---:|---:|---:|---:|---:| | BCR | 1.607 | 1.198 | 1.137 | 1.101 | 1.096 | Even the best cadence lands at 1.096 — above 1. **Cadence tuning cannot reach the objective.** --- ## 4. What would actually be required Holding everything else fixed, the warm-token gap alone accounts for the shortfall: pricing EF as if it carried Basic's warm-token count yields **BCR = 1.021** instead of 1.089 on W1. So the remaining work is not "compress more" — it is **stop carrying a growing chain of checkpoint bodies**. Three candidates, none of which R2 implemented, in the order the evidence supports them: 1. **Framing deduplication (L3 in the R1 directive).** Each checkpoint still repeats a ~116-token preamble. R2-D removed the *empty sections* but not the per-checkpoint preamble, which lives in DSH Basic's `frameSummary` and would need a core seam. Measured upper bound from R1: 66.9–70.8% of tokens on the fold-heavy workloads. 2. **Delta leaves.** R1 measured this as the weakest candidate (3.7–16.6%) and R2's decomposition does not overturn that: the state payload is a small part of the warm excess. It remains rejected. 3. **Bounded checkpoint chain.** Cap the number of frozen checkpoints and force a merge rather than a rebase, so the prefix stops growing without becoming cold. This is the M5 generational fold the R1 directive deferred — and R2's data is the first evidence that it targets the actual bottleneck. --- ## 5. What "BQR = 1" does and does not mean Quality is **not measurable in the keyless tier**, so BQR is held at 1.0 throughout this matrix. That is not a measurement of EF's quality; it is the statement that this tier cannot produce one. ### The live re-validation that was run (added after §5 was first written) The gap flagged below was partially closed with a live suite (`tests/r2-live-behavior.spec.ts`, opt-in, `npm run eval:r2-live`). It compares three arms — Basic, EF legacy, and the full R2 policy — on identical scenarios with machine-scored probes, at a far deeper compression ratio than R1 tested: **~50K tokens of history folded into a single checkpoint of 220–666 tokens.** | Arm | Probes passed | Checkpoint tokens | |---|---:|---:| | B1 Basic | 15/15 (100%) | 572 | | E0 EF legacy | 15/15 (100%) | 574 | | **E2 EF R2 policy** | **15/15 (100%)** | **220** | **Result: no behavioral regression from the R2 policy.** It answered every probe correctly while producing a checkpoint 2.6× smaller than legacy EF's (220 vs 574 tokens) and 2.6× smaller than Basic's — the R2-D diet and the admission policy visible in real data. Two caveats, both material: 1. **This is one deep fold, not a chain of folds and rebases.** The harness builds a session and drives the pressure path over it, so the first call folds everything foldable and later calls find nothing to do. A genuine multi-fold/multi-rebase test needs incremental growth with a fold between turns. **The rebase path specifically remains behaviorally unvalidated.** 2. **The sample is small** (5 replicates × 3 facts × 3 arms). It rules out gross regression; it cannot establish parity with a confidence interval. The suite carries a vacuity guard that FAILS when no checkpoint lands. That guard earned its place immediately: the first version of this suite reported 100% for all three arms while folding nothing at all, because the adapter's own reported window (131K) overrode the harness setting and the pressure threshold was never reached. ### The remaining honest reading The only other behavioral evidence in the project is R1's live tier: **EF 32/32 vs Basic 31/32** over 64 paired trials — also a null result. So the quality half of R2's gate is: *no evidence of degradation, at two compression depths*, not *evidence of parity*. The largest open risk is now narrower than it was — the R2 surface format and admission policy have been exercised against a real model — but the rebase policy has not. --- ## 6. Correctness Unchanged and not traded: ALR = 0, SSR = 0, ProvenanceCoverage = 100%, CrossSessionLeak = 0, ExactRecallMismatch = 0, CriticalConstraintViolation = 0. Every R2 change is fold-policy only; none touches the bundle, state, authority or recall layers. Attribution still reconciles exactly to the metered total on every arm, and pressure still decomposes exactly, both asserted in the stage suites. Defaults remain `leafAdmission: legacy` and `rootPolicy: legacy`. **No production default was flipped**, because the exit gate did not pass. --- ## 7. Status | Item | Status | |---|---| | Fold-every-step regime eliminated | ✅ folds 50 → 24, loop broken | | Peak context reduced | ✅ 7,896 → 2,489 (−68%) | | Checkpoint body dieted | ✅ −44% per checkpoint (220 vs 574 tokens live) | | R2 policy behaviorally re-validated | ✅ no regression at one deep fold (15/15 vs 15/15) | | **BCR < 1 (the R2 objective)** | ❌ **not achieved (1.149 on the best profile)** | | Rebase path behaviorally validated | ❌ **not done — needs incremental multi-fold harness** | | Production default flip | ❌ blocked on BCR < 1 | R2 is **not complete**. The fold loop is fixed and the cost is nearly halved relative to its own starting point, but EF still does not beat Basic on price, and the reason is now precisely identified rather than suspected.