# Retrieval benchmark — L1-267 (2026-10-27)

**Status:** one-off retrieval evaluation, not a leaderboard. This is the first published run of the
hosted endpoint's retrieval path on more than the 50-query pilot. It is **not** the paused recitation
benchmark, and it is **not** a full 650-lesson run.

**Question:** when a failure is phrased differently from the lesson that records it, does the hosted
endpoint retrieve that lesson — and how much does the optional semantic fallback change the answer?

## Method

- **Queries:** 300 lessons from the start of `data/lessons.json`; Workers AI
  `@cf/meta/llama-3.3-70b-instruct-fp8-fast` generated one synthetic failure description per lesson.
  After de-duplicating to one query per lesson, **267 queries** remained.
- **Arms:** the same `misakanet-eval` worker and corpus for both arms, with only
  `MISAKANET_SEMANTIC` toggled:
  - `0` — BM25 only;
  - `1` — BM25 + the hosted semantic fallback (`bge-m3` + Vectorize, minimum score `0.62`).
- **Scoring:** top-5 retrieval, target lesson counted at rank 1/3/5; a query with no result is
  `no_match`. Rate: 30 queries/minute.
- **Target:** `R@1 >= 0.46` **or** `no_match <= 0.50`. This is the target used by the 50-query pilot;
  it is retained here for continuity, not because 267 queries make it a universal threshold.
- **Cost:** the query generation used roughly **7.3k Workers AI neurons** (included daily allowance:
  10k); the semantic evaluation adds only query embeddings. No overage was purchased.

## Results

### Overall

| arm | R@1 | R@3 | R@5 | MRR | no_match | errors |
|---|---|---|---|---|---|---|
| BM25 | 0.210 | 0.225 | 0.225 | 0.217 | 0.712 | 0 |
| Semantic fallback | **0.446** | **0.468** | **0.472** | **0.458** | **0.453** | 2 |
| Δ | +23.6 pp | +24.3 pp | +24.7 pp | +24.1 pp | -25.9 pp | |

The semantic arm had **2 connection errors** out of 267 (0.7%); they are scored as misses. Even if both
had been hits, R@1 would rise by at most 0.7 points to about 0.453 — still below 0.46.

### By overlap

| bucket | n | BM25 R@1 | fallback R@1 | BM25 no_match | fallback no_match |
|---|---|---|---|---|---|
| high | 115 | 0.296 | 0.530 | 0.635 | 0.409 |
| low | 152 | 0.145 | 0.382 | 0.770 | 0.487 |

### By language

| lang | n | BM25 R@1 | fallback R@1 | BM25 no_match | fallback no_match |
|---|---|---|---|---|---|
| en | 202 | 0.233 | 0.480 | 0.713 | 0.441 |
| zh | 64 | 0.141 | 0.344 | 0.703 | 0.500 |
| ko | 1 | 0.000 | 0.000 | 1.000 | 0.000 |

## Verdict

The fallback is a real improvement, not noise: **+23.6 points R@1** and **-25.9 points `no_match`**
on the same corpus. It is not enough to call the R@1 target met:

| view | R@1 | no_match | target |
|---|---|---|---|
| 113-query subset | 0.460 | 0.460 | both clauses pass |
| 250-query subset | 0.404 | 0.504 | both clauses miss narrowly |
| **267-query run** | **0.446** | **0.453** | **no_match passes; R@1 misses** |

The subset variance is the honest finding: the 50-query pilot's 0.660 hybrid result (and the 113-query
0.460) are not stable estimates for the broader set. On 267 queries the fallback clears the `no_match`
half of the target and misses the `R@1` half by 1.4 points. Low-overlap queries remain the weak spot
(0.382 R@1, 0.487 `no_match`), and Chinese queries are weaker than English (0.344 vs 0.480 R@1).

## What this does not show

- It is **267 of 300 lessons**, not the full 650-lesson corpus. 33 lessons produced no query
  (model empty output / one Workers AI failure), so the sample is not complete.
- The eval worker's corpus was **not refreshed** immediately before this run. Both arms share it, so the
  A/B comparison is internally consistent, but the absolute numbers may change after a corpus sync.
- The first 300 lessons are not a random sample; they over-represent some domains (`devops`, `fanuc`,
  `development`) and contain both English and Chinese lessons.
- The semantic arm is the **fallback** path (invoked only after BM25 returns no match), not a full hybrid
  reranker. The 50-query pilot's "Hybrid" arm was a different retrieval mode and is not comparable.
- Two connection errors remain in the semantic arm; they are counted as misses, so the result is
  conservative rather than inflated.

## Artifacts

- [Query set](retrieval-l1-267-queries.jsonl) — 267 one-per-lesson queries, with `lesson_id`,
  `target_url`, language, domain and overlap bucket.
- [Machine-readable summary](retrieval-l1-267-2026-10-27.json) — aggregates and limitations.

## Reproduction notes

The run used the internal `bench.py` harness and the `misakanet-eval` worker in the working tree, not a
published package. The query set is included so the retrieval side can be replayed against any endpoint;
the published package does not claim to ship this harness. A future full run should refresh the eval
corpus, retry the 33 empty lessons, and publish the same two-arm comparison on all 650 lessons.