# OCRR — Online Correction Recovery Rate

[![arXiv](https://img.shields.io/badge/arXiv-2605.03153-b31b1b.svg)](https://arxiv.org/abs/2605.03153)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**A benchmark for measuring how fast classification systems recover from
distribution shift via online correction.**

📄 **Paper:** [arXiv:2605.03153](https://arxiv.org/abs/2605.03153)

## TL;DR

Imagine an AI assistant that needs to keep learning new skills without
forgetting the old ones. The world changes, users correct it, the system
has to catch up — fast. **How well does any given system actually do
that?**

OCRR is a benchmark that answers this. It streams a sequence of
classification tasks where the input distribution has shifted away from
training, lets each system update on every wrong prediction, and tracks
both how fast the system *learns the new* and how much it *forgets the
old*. We score 13 systems — EWC, A-GEM, LwF, kNN-LM, river-LogReg,
LoRA-on-DeBERTa, and a substrate (append-only ledger + retrieval-vote) —
across two NLP datasets, three correction policies, and three seeds.

**The substrate sits alone on the Pareto frontier in all 6 cells.** No
alternative system simultaneously matches it on novel-class recovery and
original-distribution retention.

📄 **Read the full draft paper (~6 000 words):**
[`paper.pdf`](paper/paper.pdf) (rendered)
or [`paper.md`](paper/paper.md) (Markdown source).
Methodology, all 13 systems with hyperparameters, ablations, limitations.

```
   stream  ──►  predict  ──►  if wrong, correct(text, label)
                  │                 │
                  └────► measure ◄──┘
                  ┌─────────────┐
                  │ "novel" acc │   how fast does it learn the new?
                  │  "orig" acc │   how much does it forget the old?
                  └─────────────┘
```

## The headline finding

> **At 1 000 stored entries — equal memory to A-GEM's 1 000-item replay
> buffer — the bounded substrate variant beats A-GEM by +32.6 pp on
> novel-class accuracy (0.807 vs 0.484) while losing only 4 pp on the
> original distribution.**
>
> The substrate's advantage is not "more memory." It's the *primitive* —
> append-only retrieval + margin-band majority vote — at any storage
> budget.

## Headline table — Banking77, oracle correction policy, 3 seeds

| System | Buffer | Final novel | Final orig | →70 % novel |
|---|---:|---:|---:|---:|
| **substrate** | ∞ | **0.905 ± 0.027** | **0.950 ± 0.007** | **103** |
| bounded reservoir 5000 | 5000 | 0.883 ± 0.029 | 0.943 ± 0.003 | 135 |
| bounded reservoir 1000 | 1000 | 0.807 ± 0.016 | 0.897 ± 0.003 | 351 |
| river_logreg | params | 0.867 ± 0.106 | **0.000** | 134 |
| knn_lm | ∞ | 0.823 ± 0.045 | 0.963 ± 0.005 | 271 |
| lora_deberta_v3_large | params + LoRA | 0.771 ± 0.086 | 0.108 ± 0.008 | 297 |
| online_linear | params | 0.544 ± 0.081 | 0.928 ± 0.012 | never |
| a_gem | params + 1000 | 0.484 ± 0.065 | 0.938 ± 0.014 | never |
| ewc | params | 0.405 ± 0.069 | 0.946 ± 0.007 | never |
| lwf | params | 0.118 ± 0.025 | 0.949 ± 0.004 | never |
| static_knn | (seed) | 0.000 | 0.957 | never |
| static_linear | params | 0.000 | 0.952 | never |

**Substrate sits alone on the storage-vs-recovery Pareto frontier across all
6 (dataset × policy) cells.** No alternative system simultaneously matches it
on novel-class recovery and original-distribution retention.

See [`paper/paper.md`](paper/paper.md) for the full draft and
[`results/`](results/) for per-seed CSVs, logs, and figures.

## What OCRR measures

A classification system is presented with a stream of `(text, label)` pairs
drawn from a distribution that has shifted away from its initial training
set. After each prediction:

- If wrong, a correction policy decides whether to call
  `system.correct(text, label)`.
- The system updates its state in real time.
- We track accuracy on **both** the held-out novel distribution AND the
  original distribution (forgetting check) over the correction-count axis.

Reported metrics: final novel accuracy, final original accuracy,
corrections-to-N % thresholds, and per-system storage footprint.

### Correction policies

Real-world feedback is rarely a perfect oracle — sometimes the user
corrects, sometimes they don't, sometimes they only correct when they
notice. OCRR runs each system under three policies to characterise this:

| Policy | What it models | Behaviour |
|---|---|---|
| `oracle` | A diligent annotator who corrects every error | Every wrong prediction gets the true label revealed |
| `random_50` | Half-attentive feedback (50 % chance per error) | Corrections arrive on a Bernoulli(0.5) draw when wrong |
| `random_10` | Sparse feedback (10 % chance per error) | Stress test for sparse-supervision regimes |

Right predictions never trigger a correction call under any policy — the
benchmark only measures recovery from observed mistakes. Across all three
policies the substrate's Pareto dominance is preserved; sparser feedback
just means recovery takes more total stream items to accumulate the same
number of corrections.

## Streaming-learning constraints

| Property | Required by OCRR? |
|---|---|
| Data arrives sequentially | **Yes** |
| Model updates in real time on each correction | **Yes** |
| Memory bounded (no historical-data storage) | Optional — reported per system |

The third constraint is what classical online-ML libraries (`river`)
require. OCRR does not enforce it but **reports each system's storage
footprint** so the comparison is honest about the trade-off. The
`bounded_reservoir_*` and `bounded_fifo_*` variants probe the entire
storage-vs-recovery Pareto.

## Repository layout

```
ocrr-benchmark/
├── ocrr_benchmark/         # importable Python package
│   ├── eval/               # harness, systems, baselines, ablations
│   ├── memory/             # ImmutableLedger (append-only + Merkle hash chain)
│   └── datasets/           # Banking77 / CLINC150 loaders
├── scripts/                # run_ocrr*.py — one per result cell
├── results/                # CSVs, logs, figures from the paper
├── paper/                  # paper draft + figures
├── REPRODUCING.md          # step-by-step reproduction
├── pyproject.toml          # dependencies
└── LICENSE                 # MIT (code) — paper is CC BY 4.0
```

## Quick start

```bash
# Install
pip install -e .

# Run the v1 single-cell sanity check (Banking77, oracle, 4 systems, 1 seed)
python scripts/run_ocrr.py --output results/_sanity.csv

# Reproduce the headline 9-system × 18-cell sweep
python scripts/run_ocrr_full_sweep.py --output results/_repro_full.csv
```

See [REPRODUCING.md](REPRODUCING.md) for the full reproduction playbook.

## Systems benchmarked (13 total)

**Static strawmen:** `static_knn`, `static_linear` — zero learning, lower
bound on novel-class accuracy.

**Naive online:** `online_linear` — frozen encoder + per-correction SGD on
the classifier head.

**Continual-learning baselines:** `ewc` (Kirkpatrick 2017), `a_gem`
(Chaudhry 2019), `lwf` (Li & Hoiem 2017).

**Retrieval/parametric hybrids:** `knn_lm` (Khandelwal 2020).

**Online-ML libraries:** `river_logreg` (LogisticRegression).

**Parameter-efficient fine-tune:** `lora_deberta_v3_large` (LoRA rank 8 on
DeBERTa-v3-large query/value projections).

**Substrate:** `substrate` (unbounded), plus `bounded_reservoir_{1000, 5000}`
and `bounded_fifo_{1000, 5000}` storage-Pareto variants.

**Ablations** (not in main table): `substrate_k1`, `substrate_sumsim`,
`substrate_count_only`, `substrate_no_recency`. Vote-rule details barely
matter in the dense-substrate regime; margin-band gating is the only
load-bearing piece.

## Datasets

- **Banking77** (Casanueva et al. 2020) — 77 fine-grained banking intents,
  ~10 k train / ~3 k test. CC-BY-4.0.
- **CLINC150** (Larson et al. 2019) — 150-class cross-domain intents, ~15 k
  train / ~5 k test. CC-BY-3.0.

## Limitations

What this benchmark and these results do **not** claim, kept honest:

- **Single language.** Both datasets are English-only. Recovery dynamics
  in a multilingual or cross-script setting are open. The MASSIVE-style
  multilingual extension is sketched in the paper as future work.
- **Categorical shift only.** The current shift scenario holds out 10
  full classes per seed. A within-class drift scenario (paraphrase /
  topic creep) would let static systems score > 0 and reframe the
  comparison as recovery *speed* vs. recovery *possibility*. Open as
  Phase 10.4.
- **Scale.** Headline numbers were validated up to ~10 k stored entries.
  At that scale brute-force comparison touches every entry and the
  never-forget property is mechanically guaranteed. Beyond that, the
  guarantee depends on HNSW retrieval recall (which is approximate). The
  substrate ships a `force_brute=True` mode for 100%-recall mode at any
  scale (O(N) latency tradeoff), and a `verify_hnsw_recall()` helper to
  measure the gap. `scripts/run_substrate_scale_study.py` characterises
  the gap on synthetic data at user-supplied scales (10 k, 100 k, 1 M).
- **Encoder choice fixed.** All retrieval-style systems use
  `BAAI/bge-large-en-v1.5`. An encoder-swap ablation (CLIP, CLAP,
  code-bge, or DeBERTa as encoder) is open work.
- **LLM-ICL row partial.** Local Ollama qwen2.5:14b on CPU was too slow
  to complete the full sweep (~60 s per inference). Frontier-API
  replication is open as Phase 10.1f.
- **No human-in-the-loop study.** All correction policies are
  programmatic. Real users may behave differently — partially,
  inconsistently, or with errors of their own. Out of scope here.
- **Reproducibility caveat.** LoRA-DeBERTa numbers shift slightly with
  PyTorch / transformers minor versions due to dtype handling. The
  substrate, kNN-LM, online-linear, and continual-learning baselines
  are version-stable across the pinned range.

## Citation

```bibtex
@misc{grassi2026ocrr,
  title  = {OCRR: Online Correction Recovery Rate — A Benchmark for
            Classification Systems Under Distribution Shift},
  author = {Adrian Grassi},
  year   = {2026},
  note   = {arXiv preprint, NeurIPS Datasets & Benchmarks 2026 submission}
}
```

The arXiv ID will be inserted here once the submission is live.

## License

- **Code** (`ocrr_benchmark/`, `scripts/`): MIT. See [LICENSE](LICENSE).
- **Paper** (`paper/`): CC BY 4.0 — distributed via arXiv under that licence.
- **Data**: Banking77 (CC-BY-4.0) and CLINC150 (CC-BY-3.0) are upstream
  datasets distributed under their original licences.

## Status

- v0.1.0 — initial public release of paper draft + reproducibility package.
- See [open issues](https://github.com/adriangrassi/ocrr-benchmark/issues) for
  follow-up work (LLM-ICL with frontier-API spot check, cross-modal
  encoder-swap study, convergence theory).

## Lineage — what comes next

This repository hosts the OCRR research program. The OCRR v1 paper above
introduced retention as a measurable property of memory systems, evaluated
across one axis (correction-stream retention) on two NLP datasets. Two
follow-ups are in progress:

| Version | Scope | Status |
|---|---|---|
| **OCRR v1** (this paper) | Retention only — single axis, two NLP datasets, 13 systems | Public ([arXiv:2605.03153](https://arxiv.org/abs/2605.03153)) |
| **OCRR v2** | Retention + cross-modal + adversarial corrections + 10-stage chain | In progress, mid-2026 release |
| **OCRR v3 / AMTB** — Agent Memory Transfer Benchmark | Six-axis benchmark unifying retention + recall + auditability + cross-modal + scale + adversarial revision | **Pre-registered 2026-05-07** — see [`v3-amtb/`](./v3-amtb) |

**OCRR v3 / AMTB is pre-registered.** The pre-registration locks the
experimental design (axes, datasets, hypotheses, metrics) before any
measurements begin. See [`v3-amtb/README.md`](./v3-amtb/README.md) and
[`v3-amtb/PRE-REGISTRATION.md`](./v3-amtb/PRE-REGISTRATION.md) for full
details.