# Retrieval eval

The measurement behind DD-070 §8's "measured retrieval win" claim
(P-023 §7 step 7). Compares docdog's hybrid search against an honest
ripgrep baseline on this repo's own corpus, using the frozen query set
in `queries.yaml`.

## Run

```
docdog index                                # fresh cache first
npx tsx tests/eval/run-retrieval-eval.ts    # ~1 min, writes report.md
```

Needs the real ONNX embedder and `rg` on PATH. Not part of `npm test`
(vitest only picks up `tests/**/*.test.ts`). Deterministic: same corpus
+ same query set → same report.

## Systems

| system | what it is |
|---|---|
| `hybrid` | the real `search()` from `src/storage/search.ts` — FTS5 BM25 + brute-force cosine, RRF fusion, top-10 |
| `fts` | the keyword leg alone (diagnostic — quantifies what fusion adds) |
| `vector` | the cosine leg alone (diagnostic) |
| `rg-rank` | grep, *generous* reading: mechanical keyword extraction from the query, an `rg --count-matches`-equivalent pass per term over the scan paths, files ranked by (distinct terms matched, total matches) — a poor-man's TF ranking an agent or script could plausibly build |
| `rg-walk` | grep, *honest transcript*: a single `rg -in` OR-run, output in path-sorted order — what an agent actually reads after one grep |

The isolated legs **mirror** `search.ts`'s SQL rather than importing
internals — the eval must not grow the prod surface (session-7 handoff
guardrail). If the legs change, update the mirrors.

The grep engine is an in-harness reimplementation of ripgrep's
`--ignore-case --fixed-strings --line-number --sort path` semantics
(case-insensitive literal matching over the scan paths' `*.md` files),
so the eval needs no PATH-dependent binary. Validated against ripgrep
14.1.1: occurrence counts and matching line numbers are identical on
probe terms ("supersedes" ×10 in the DD-070 file, "arango" ×4 in the
OQ-30 file).

## Metrics

- **Hit@k** — the query's gold record set intersects the top k. Gold
  sets are 1–3 records; any counts (they were judged once, frozen in
  `queries.yaml` — DP-001: relevance judgment is agent+user work, done
  once, so runs stay mechanical).
- **MRR@10** — mean reciprocal rank of the first gold (0 when absent
  from the top 10).
- **tokens-to-locate** — approximate tokens (chars/4) of tool output
  the consumer must scan, in the tool's own presentation order, until
  the first gold reference appears. Measured only on the two
  end-to-end consumer surfaces:
  - `hybrid`: the MCP text rendering (`src/mcp/tools/search.ts` shape —
    id, title, file, relevance, description/preview per entry);
  - `rg-walk`: the raw `rg -in` transcript up to the first gold-file
    line.

## Caveats (read before quoting numbers)

- **rg gets the same corpus docdog indexes** (the config `scan_paths`,
  `*.md`), mapped back to record ids by file. rg hits on files hosting
  no record stay in its ranking as noise — that is the honest agent
  experience.
- **rg-rank has no reading transcript** — its ranked list is a
  construct, so it gets Hit/MRR but no tokens metric. Its real cost is
  *higher* than any list suggests: a filename alone doesn't confirm an
  answer; the agent still opens files. The tokens comparison
  (hybrid vs rg-walk) is the honest end-to-end pair.
- **Keyword extraction is fixed and mechanical** (stopword list frozen
  in the harness). A human might pick better grep terms; an agent
  might pick worse. Reformulation loops are not modeled on either
  side.
- **Gold sets contain 1–3 near-equivalent records** (e.g. a DD and the
  EJ it mirrors). This helps every system equally.
- **Token counts are chars/4 approximations**, not a real tokenizer.
- The corpus is small (~250 records) and self-describing (titles and
  descriptions are agent-authored) — generalization to larger or
  messier corpora is untested.

## Files

- `queries.yaml` — frozen query set with gold answers (versioned; do
  not reword — append)
- `run-retrieval-eval.ts` — the harness
- `report.md` — generated report (committed as the run's evidence)