# Retrieval-accuracy eval

<!-- eval:stamp -->
> Generated by `npm run eval` (`eval/eval.mjs`) on 2026-08-04. Re-run to refresh.
<!-- /eval:stamp -->
>
> Fixtures are **pinned to exact commits** (see `FIXTURES` in `eval/eval.mjs`), so
> these numbers are re-derivable rather than a snapshot of whatever upstream looked
> like that day. A figure moving between runs therefore reflects a change in
> agentmap, not in the fixture. Re-pin deliberately with `--repin`.
> Complements `RESULTS.md` (token efficiency). This file answers the harder question:
> **when agentmap returns fewer tokens, are they the _right_ tokens?**

## Why this exists

`RESULTS.md` proves agentmap puts far fewer tokens in front of the agent. It does **not**
prove those tokens contain the correct answer — "fewer tokens" could mean "fewer correct
answers". This eval measures **retrieval correctness** against ground truth derived live
from real public repos, then shows accuracy and token cost together.

## Method

Ground truth is **derived at runtime** from each cloned repo (not hand-authored, so it
can't silently rot), using a **different mechanism than agentmap's ts-morph graph** — so the
comparison is a real cross-check, not circular:

- **Symbol definition** — "where is symbol `X` defined?" Ground truth = the single file
  whose source contains `export <kind> X` (regex over declaration sites). Only globally
  unique definitions are tested (no ambiguity). Compared: `agentmap --find X` (exact-name
  matches, in returned order) vs naive `git grep -n X` (every occurrence).
  Metric: **top-1 / top-3 hit rate** (is the definition file the 1st / among the first 3
  results?).
- **Dependents / blast radius** — "which files import module `M`?" Ground truth = files
  whose relative-import specifiers **resolve** to `M` (a real resolver: handles TS
  `./x.js`→`x.ts`, `.tsx/.mts`, `index` barrels, `require`/dynamic `import`,
  re-export edges). Compared: `agentmap --relates M` `.dependents` vs naive
  `git grep -l` for the module's name in import lines (for `index.*` modules the parent
  dir name, so the baseline isn't strawmanned into matching every barrel). Metric:
  **precision / recall / F1** against the resolved set.

**Scope alignment (so the comparison is fair both ways):** test files (`*.test.*`,
`runtime-tests/`, etc.) are excluded from ground truth **and** from both tools' outputs
before scoring — otherwise agentmap's legitimate test-file importers would score as false
positives. **Type-only edges** (`import type` / `export type`) are excluded from ground
truth, because agentmap's ts-morph graph drops them by design — counting them would penalize
recall for a documented behaviour rather than a defect. Each fixture is scoped to a
`sourceRoot`. Token cost = chars/4 of the **full default (human) output** each tool puts in
context (same heuristic as `RESULTS.md`, both sides).

> Caveats — read these before quoting numbers. (1) The ground-truth resolver is regex-based,
> not a TypeScript type-checker; it is the *reference*, and a handful of exotic edges
> (`tsconfig` `paths` aliases — none in these fixtures) may differ from a true compiler.
> (2) Definitions tested are uniquely-declared only — the easy, unambiguous cases. (3)
> `agentmap --find` lists barrel **re-export** sites alongside the real declaration, but
> ranks the declaration site above them, so top-1 no longer trails top-3. Until 0.22.0 it
> did not, and a barrel — imported by everything, therefore high PageRank — could take the
> top slot on a repo whose `index.ts` forwards many modules. That is what moved top-1 from
> 53.3% to 100% here; the barrel rows are still returned, just lower. (4)
> Dependents recall reflects agentmap's **value-import** graph only (type-only edges are
> excluded from truth to match it); a separate "type-aware" mode would be needed to retrieve
> type-only importers. (5) Feature-level retrieval (`--feature`) is **not** scored — these
> are libraries with no `app/` routes, so the route-based feature detector is empty; that
> needs a Next.js-style app fixture (TODO). (6) Numbers move with upstream repos; resolved
> SHAs are recorded below.

## Results

### Overall (pooled across fixtures)

<!-- eval:overall -->
| Task | n | agentmap | naive grep |
|---|---|---|---|
| Symbol definition — top-1 / top-3 hit | 75 | **100% / 100%** | 32% / 80% |
| Dependents — recall / precision | 42 | **99.2% / 100%** | 100% / 59.9% |

Symbol-definition lookups cost a median **~1.9× fewer tokens** than dumping `git grep`
output, while landing the definition in the top 3 more often. For dependents the story is a
**precision** win, not a token win: agentmap returns a clean importer list (high precision)
where naive grep returns a noisy superset (high recall, low precision) — and `--relates`
actually costs **more** tokens than `grep -l` because it returns the full blast radius
(exports + imports + dependents + related), not just the file list.
<!-- /eval:overall -->

> **Reconciling this with `RESULTS.md`'s 99.2% blast-radius row.** Both numbers are real;
> they price different baselines. `RESULTS.md` scenario D compares against an agent that
> `cat`s every dependent file — agentmap wins by ~99%. This eval compares against `grep -l`,
> which returns a file list and nothing else — agentmap loses on tokens. The list is cheaper
> because it is *less* correct: at **59.9%** precision, roughly 4 of every 10 paths on it are
> not dependents, and the agent pays for them on the next turn when it opens them. Neither
> file is the whole picture on its own; quote them together.

### Per fixture

<!-- eval:fixtures -->
| Repo | commit | def n | agentmap top1/top3 | grep top1/top3 | deps n | agentmap recall/prec | grep recall/prec |
|---|---|---|---|---|---|---|---|
| zod | `912f0f51b0` | 25 | 100% / 100% | 40% / 84% | 20 | 98.3% / 100% | 100% / 49.9% |
| zustand | `beca84e600` | 25 | 100% / 100% | 24% / 80% | 2 | 100% / 100% | 100% / 65% |
| hono | `26d8e42bff` | 25 | 100% / 100% | 32% / 76% | 20 | 100% / 100% | 100% / 69.3% |
<!-- /eval:fixtures -->

## Reproduce

```bash
npm run eval                 # all fixtures
node eval/eval.mjs --repo zod --sample 40
node eval/eval.mjs --refresh # re-clone upstreams
```

Clones land in `tmp/eval/` (gitignored). Network required; not part of CI.