# Measured retrieval quality exomem ships a golden-set eval harness, so retrieval quality is a number you can re-derive, not a claim. This page documents the methodology and the measured results. It is deliberately transparent about what the numbers are and are not: a small, single-vault, self-graded benchmark — strictly more evidence than "no published results" or "no harness at all", but not an independent third-party benchmark. ## Relation lifecycle evidence The graph-value lifecycle harness is synthetic and is run with `uv run python scripts/graph_value_benchmark.py`. It measures an explicit caller decision separately from resolver evidence: a resolver candidate does not count as selection, registry promotion, or edge authoring. Relation-review scale and concurrency checks likewise measure graph-native queue work bounds separately from latency; queue responses distinguish `available` from typed `warming`, `pending`, and `unavailable` results. The relation census is not a benchmark score. Its aggregate `relation_counts`, `page_counts`, and `denominators` compare caller-supplied origin-date and page-type cohorts without paths, bodies, source snippets, or examples. In particular, `zero_authored_relation_rows` and `zero_body_connections` are distinct counts, not a debt score or a backfill target. > The results table below was measured with `scripts/eval_retrieval.py` against > the **bundled fixture vault** (`tests/fixtures/`) — a deterministic, tiny, > public corpus, so the exact run reproduces from a clean checkout with no > private data (it needs the downloaded embedding + reranker models, which the > lean CI matrix does not have; the dedicated `retrieval-eval` job does). The > same regression floors are asserted in `tests/test_retrieval_golden.py`. See > [Reproduction](#reproduction) to re-run it or point it at your own vault. ## Pinned local-core comparison The retrieval eval below measures Exomem against a golden set. A separate cross-product harness, `scripts/graph_value_benchmark.py`, executes Exomem and a pinned Basic Memory checkout over one neutral manifest rendered into each product's native Markdown. It reconciles runtime MCP/CLI inventories, isolates all state, uses persistent public MCP sessions, preserves raw envelopes, and reports shared core, lifecycle integrity, explanation truth, performance, and Exomem extensions independently—never as a weighted score. The corrected lean direct run on 2026-07-16 passed all 22 required Exomem probes and kept shared core, lifecycle integrity, explanation truth, and Exomem extensions green. Operation-level preflight separately proved all 21 active Exomem MCP operations and all 16 MCP plus 5 CLI operations assigned to Basic Memory's lean probes; renderer facts and per-probe state transitions are recorded independently of probe-ID coverage. The fresh run retained 242 raw request/response artifacts for 121 public calls and exercised strict lifecycle generation/removal plus reconcile/full-maintenance equivalence. It emitted no full local-core-advantage claim because learned models, paired performance, and media are outside the lean profile. The current WSL2 full-profile attempt is invalid at native model setup (`SIGABRT` while SentenceTransformers loads BGE on either CPU or CUDA), so it is recorded as an environment failure rather than a product result. See [Exomem vs Basic Memory: graph-value comparison](comparison-basic-memory-graph.md) for exact revisions, gates, limitations, and reproduction commands — **withdrawn 2026-08-09** (adversarial re-examination: authored-competitor bias in both directions plus an unreproducible recorded run; see that page's banner). Its comparative claims are not citable. A third, falsification-oriented system — the four-track **memory-proof benchmark** (`benchmarks/membench`: deterministic seeded epistemic corpus, capability-declaring adapters, deterministic gates a blinded judge cannot override, harness-activation and workflow tracks) — is specified in [memory-proof-benchmark.md](memory-proof-benchmark.md) with its first weaknesses-first baselines in [memory-proof-benchmark-v01-findings.md](memory-proof-benchmark-v01-findings.md) (headline: in the lexical-degraded profile, natural-language questions retrieve nothing — confirmed by two independent harnesses, root-caused to the hybrid retention seam). ## Methodology - **Harness.** `scripts/eval_retrieval.py --report markdown` runs the golden set once per retrieval mode and emits an aggregate-only markdown report: per-mode ranking-quality metrics, per-mode `find()` latency percentiles, and rounded corpus counts. The report contains no query text, no vault-relative paths, no excerpts, and no per-query rows — only aggregates. - **Modes.** Three, at the shipped default ranking config: - `keyword` — lexical (BM25) only. - `hybrid` — lexical + vector fusion (the default). - `hybrid+rerank` — `hybrid` with the cross-encoder reranker on (`rerank=True`; reranking is orthogonal to mode, not a fourth mode). - **Metrics.** The four the harness already computes per query and mean-aggregates (`src/exomem/eval_metrics.py`): **NDCG@5**, **NDCG@10**, **MRR**, **recall@10**. - **Latency.** Wall-clock `find()` time is sampled by repeating the golden set `--repeat N` times (default 3) per mode and timing each call with `time.perf_counter()`. The report shows the **median** and **p90** (nearest-rank percentile) over that flat sample. - **Golden set.** `tests/golden/queries.yaml` — **26** hand-authored natural-language queries, each with one or more relevant target pages (`expect_any_of`, or `graded` 0–3 for NDCG). Several entries each PIN one ranking behavior so a regression points at what it broke: compiled-over-source (`prefer_compiled`), supersession demotion (`prefer_active`), a graph-hop-only target (reachable only through a wikilink), a temporal-recency query, an exact/quoted lookup, and a `scope="kb"` query whose answer lives OUTSIDE `Knowledge Base/` (exercising the auto-widen reserve). The set is designed to grow: `scripts/derive_relevance_pairs.py` proposes additions mined from real (query → cited path) usage, which a human confirms. - **Vault scale (rounded).** Reported as counts rounded DOWN to the nearest 10 (privacy floor): markdown files (whole vault), KB notes (the pages `find()` indexes), and media artifacts (audio/video/image/pdf). - **Embedding sidecar required.** The eval reads a prebuilt embedding sidecar (a real vault's running server keeps it fresh). The bundled fixture ships without one, so the run below builds it first with `embeddings.get_embedding_index(vault).rebuild_all()`. Without that step the vector lane is empty and `hybrid` collapses to `keyword` — which is exactly the silent-degradation state the doctor sidecar probe and the `find` `degraded` marker now surface. - **exomem version / models.** exomem `0.4.1`, embedding model `BAAI/bge-base-en-v1.5`, reranker `BAAI/bge-reranker-base`. - **Hardware.** AMD Ryzen 7 5800X3D (8-core) / NVIDIA RTX 5080 / 32 GB RAM / Windows 11. ## Results Measured 2026-07-03 against the bundled fixture vault (sidecar built first). - Corpus scale: 30 markdown files, 20 KB notes, 0 media artifacts (rounded down to the nearest 10); 198 chunk vectors. - Golden set: 26 queries. | Mode | NDCG@5 | NDCG@10 | MRR | recall@10 | latency median (ms) | latency p90 (ms) | |---|---|---|---|---|---|---| | keyword | 0.3430 | 0.3430 | 0.3462 | 0.3269 | 6.4 | 7.2 | | hybrid | 0.9142 | 0.9270 | 0.9154 | 0.9615 | 6.5 | 7.3 | | hybrid+rerank | 0.9286 | 0.9397 | 0.9397 | 1.0000 | 6.6 | 7.6 | The gap is the whole point: lexical-only (`keyword`) recalls the intended page for only about a third of the queries (recall@10 0.3269, NDCG@10 0.3430), while `hybrid` finds nearly every target in the top-10 (recall@10 0.9615) and ranks it at or near the top (NDCG@10 0.9270, MRR 0.9154); the cross-encoder reranker then tightens the order and lifts recall to 1.0 (NDCG@10 0.9397). `hybrid`'s recall is below 1.0 by design: two queries deliberately grade a marginal page that `prefer_compiled` / `prefer_active` demotes out of the top-10 (the compiled-over-source and supersession pins), so their grade-3 ideal is found but the marginal is not. Latency here is dominated by fixed per-call overhead — the corpus is 198 chunk vectors — so treat the absolute ms as a floor, not a representative production figure; the per-lane curve below shows how cost grows with corpus size. ## Per-lane latency vs. corpus scale The aggregate latency in the results table is measured on the ~200-vector fixture and is dominated by fixed per-call overhead — it says nothing about how cost grows with a real vault. That blind spot has bitten before: a 10-file fixture benchmark once reported a whole `find()` at ~5ms and hid a ~14s graph-lane cost on a ~1700-note vault, because an aggregate over a toy corpus cannot show a single lane blowing up. So this section measures latency PER LANE at realistic scale. `scripts/latency_curve.py` generates a synthetic, densely-wikilinked vault (25 outbound links per note) at increasing sizes, warms every lane, then times a fixed query set with `find(include_timings=True)` and reports median / p90 per stage. The default run is **model-free** (the vector/CLIP/rerank lanes are switched off) so it needs no GPU or embedding sidecar and reproduces anywhere; it reports the lanes that both scale with the corpus and need no model — exactly where the graph regression lived. Model-free, measured 2026-07-03 on the reference host (AMD Ryzen 7 5800X3D / RTX 5080 / 32 GB / Windows 11), per-lane **median / p90 in ms**: | Notes | vector | bm25 | keyword | graph | fusion | rerank | total | |---|---|---|---|---|---|---|---| | 100 | — | 13.4 / 14.1 | 14.2 / 15.7 | 15.6 / 16.7 | 4.5 / 5.1 | — | 53.0 / 59.0 | | 500 | — | 63.2 / 75.8 | 73.9 / 82.7 | 63.3 / 70.8 | 19.3 / 32.3 | — | 231.5 / 289.1 | | 1000 | — | 128.4 / 139.6 | 140.6 / 160.9 | 124.8 / 133.2 | 24.3 / 60.5 | — | 428.5 / 537.9 | | 2000 | — | 243.2 / 270.1 | 267.7 / 303.8 | 225.9 / 238.6 | 27.5 / 122.6 | — | 805.2 / 1041.4 | | 5000 | — | 594.5 / 625.8 | 642.2 / 777.9 | 559.5 / 598.8 | 31.8 / 311.9 | — | 1840.1 / 2635.2 | Reading it: - Every model-free lane scales roughly linearly with corpus size on this dense corpus. `graph`, `bm25`, and `keyword` are all O(N) full-corpus lanes; `fusion` stays cheap. A dash means the lane was switched off for the model-free run. - The **graph** lane is the one that regressed historically. Even warm — with the wikilink resolver NOT rebuilt — it costs ~226ms at 2000 notes here, because the synthetic corpus is deliberately dense (25 links/note), heavier than a typical vault. The regression this guards against — the resolver reverting to a per-query full rebuild (read + YAML-parse every note) — pushes the graph lane to ~1.9s at 2000 notes and ~4.7s at 5000: the class of ~14s event a fixture-scale benchmark hid. - The synthetic dense corpus is a **stress shape, not a representative vault**: a real vault has fewer links per note and lighter graph cost. These numbers show scaling and relative lane cost, not an absolute production latency. With a real embedding sidecar and the reranker on (`--embeddings --rerank`), the `vector` and `rerank` lanes appear too — the per-call query embed and the cross-encoder over the fused candidates. Measured at smaller sizes (sidecar build dominates wall-time at large N): | Notes | vector | bm25 | keyword | graph | fusion | rerank | total | |---|---|---|---|---|---|---|---| | 100 | 14.0 / 19.2 | 15.2 / 17.5 | 15.4 / 18.4 | 20.2 / 25.0 | 4.4 / 4.9 | 194.2 / 219.1 | 289.3 / 311.1 | | 500 | 13.4 / 16.4 | 66.5 / 70.3 | 72.2 / 82.4 | 70.7 / 72.7 | 24.7 / 29.0 | 146.8 / 152.3 | 419.4 / 463.5 | | 1000 | 14.0 / 18.7 | 137.6 / 176.0 | 145.5 / 185.6 | 135.8 / 145.7 | 39.2 / 60.7 | 147.8 / 157.2 | 660.8 / 766.9 | | 2000 | 13.6 / 17.1 | 250.8 / 260.3 | 279.3 / 352.6 | 244.3 / 285.8 | 50.8 / 129.9 | 146.0 / 149.9 | 1030.5 / 1291.1 | The vector query-embed and the reranker's per-candidate cost are roughly corpus-size-independent (they scale with the query and the fixed candidate count, not N), so they add a near-constant offset on top of the O(N) lexical/graph lanes. ### Regression gate `tests/test_latency_gate.py` turns this curve into a CI gate: it generates the same dense 2000-note vault, warms the lanes, and asserts NO lane exceeds a ceiling (`graph < 1000ms`, `total < 5000ms`). The ceilings sit in the wide gap between the warm baseline (graph ~226ms, total ~805ms) and a resolver-rebuild regression (graph ~1.9s), so a 14s-style regression fails CI while ordinary CI-speed variance does not. It is model-free, so it runs in the lean matrix on every PR and is pinned in the `retrieval-eval` job next to the golden quality gate. Reproduce the curve: ``` uv run python scripts/latency_curve.py --sizes 100,500,1000,2000,5000 uv run --extra embeddings python scripts/latency_curve.py --embeddings --rerank --sizes 100,500,1000,2000 ``` ## Vector backend at scale (10k–50k notes, sqlite-vec vs in-memory scan) The `add-sqlite-vec-backend` change moved the vector lane's default onto vec0 virtual tables inside the embedding sidecar (`EXOMEM_VEC_BACKEND=auto`), with the in-memory numpy scan as kill switch and fallback. That swap was gated on measured evidence, per the find-recall-efficiency spec. This section is that evidence and the decision record. Method: same dense synthetic generator as the curve above, with a REAL bge sidecar per size (`--embeddings`), measured once per backend over the same built corpus via `--vec-backend numpy,sqlite-vec,binary` and reused across runs via `--corpus-cache` (embed once, measure many). `binary` is `sqlite-vec` with `EXOMEM_VEC_QUANT=binary`. Measured 2026-07-03 on the reference host (AMD Ryzen 7 5800X3D / RTX 5080 / 32 GB / Windows 11); 10k tier `--repeat 3`, 50k tier `--repeat 2`. Vector lane only (ms, median / p90); `overlap@10` is the mean top-10 overlap of end-to-end `find()` results against the numpy pass; RSS is the process resident set after the pass (psutil; not captured for the 10k run). | Notes | chunk vectors | vector numpy | vector vec0-f32 | overlap@10 f32 | vector vec0-binary | overlap@10 binary | RSS numpy / f32 / binary (MB) | |---|---|---|---|---|---|---|---| | 10000 | 40,000 | 17.5 / 21.8 | 130.5 / 137.2 | 1.00 | 59.7 / 74.3 | 0.86 | — | | 50000 | 200,000 | 31.2 / 36.8 | 544.5 / 550.0 | 1.00 | 253.3 / 357.6 | 0.68 | 3491 / 1707 / 1234 | Reading it honestly: - **The warm lane race goes to numpy, 7–17×.** BLAS over a RAM-resident contiguous matrix runs at memory bandwidth; the vec0 f32 scan re-reads every vector byte per query through SQLite's page layer (~614 MB at 200k chunks → ~545 ms). Both are brute force — neither is sub-linear — so the gap grows with N. - **What numpy pays for that speed is the point of the swap:** at 200k chunks the numpy pass holds ~1.8 GB more resident (the float32 matrix plus every chunk's text in the metadata cache), and any out-of-process sidecar change costs it a full O(N) reload. The vec0 backend holds essentially nothing between queries. - **vec0-f32 is exact.** overlap@10 = 1.00 at both tiers, end-to-end through `find()` — matching the unit-level parity tests. The default-backend swap changes no ranking. - **Binary quantization diverges at scale.** overlap@10 fell from 0.86 (10k) to 0.68 (50k) on this corpus. It stays strictly opt-in (`EXOMEM_VEC_QUANT=binary`, default off); the golden retrieval floors are its promotion gate (it clears them on the fixture corpus — both parametrized passes of `tests/test_retrieval_golden.py`). - **The loudest finding is about a different lane entirely:** at 50k notes the whole `find()` costs ~25 s under EVERY backend, because bm25 / keyword / graph are each ~8 s of O(N) work. The vector lane is under 2.2% of the total in its slowest configuration. "Sub-second search at 100k notes" is gated on the lexical/graph lanes, not on vector search — that is the next scaling build. **Decision record.** (Superseded 2026-07-05 — the default is now `numpy`; see the "Default-backend flip" note at the end of this section.) `auto` defaults to vec0-f32: the lane-latency delta is invisible end-to-end today, while the residency and reload wins are large and grow with corpus size. `EXOMEM_VEC_BACKEND=numpy` restores the previous behavior wholesale. If a future build makes the vector lane the visible bottleneck (e.g. after the lexical lanes are fixed), the recorded next step is a real ANN index — hnswlib, or sqlite-vec's ANN once it leaves alpha — behind the same `vecstore` seam, recall-gated by the same golden floors. **ANN decision record (2026-07-04).** The "recorded next step" above was evaluated and is deliberately **not shipping**. Candidates: **hnswlib** 0.8.0 publishes no binary wheels (source-only install, no release since 2023; the chroma-hnswlib fork's wheels stop at cp312 while exomem supports 3.13) — disqualified on packaging before recall was ever measured. **sqlite-vec's ANN** exists only in 0.1.10 alphas whose DiskANN DELETEs are documented as expensive — fatal for a write path that deletes + reinserts a file's rows on every note write (stable PyPI 0.1.9 remains brute-force-only). **usearch** 2.25 passes the packaging and incremental-upsert bars (tombstone remove/add keyed by blob rowid; save/restore to bytes, so the index could persist as chunked blobs inside the sidecar), but shipping it means a new dependency plus several hundred lines of snapshot+journal persistence machinery. What decided it is scale evidence, not candidate quality: on a real ~1,700-note production vault the vector lane is 16–19ms of a ~125ms `find()` (see the real-vault section below); at the measured 50k stress tier the shipped vec0 default still holds `find()` sub-second (864ms); and the only regime where ANN clearly wins — 100k+ — is far beyond any observed real vault, where the numpy kill switch already buys a ~60ms lane at a RAM cost that a ~50-line "numpy-lite" fix (drop `chunk_text` from the cached tuple and join metadata by rowid like the vec0 path already does; optionally hold the matrix bf16) would cut to well under 1GB with zero new dependencies. Verdict: **brute force is fine at realistic scale.** Revisit when sqlite-vec's ANN reaches a stable release with cheap deletes (it keeps the zero-residency, in-sidecar model for free), or if real vaults approach the 50k–100k tier. Any future ANN backend stays gated by the golden floors PLUS an at-scale recall-vs-exact-scan test with a mistuned negative control — an undersized HNSW search parameter caps recall silently while search still "works" (that exact failure mode is on record from a sibling project at ef_search=40). The 100k-note tier (400k chunk vectors) stays generated, embedded, and cached. A full 6-pass measurement (3 vector × 2 lexical backends) was attempted 2026-07-04 and killed at the 72-minute mark by owner decision — the `--lexical-backend python` passes dominate wall-time at 100k (each rebuilds the in-process token corpus over 100k notes per pass). If re-attempted, run it overnight, or drop the python passes: ``` uv run python scripts/latency_curve.py --embeddings --sizes 100000 --max-embeddings-size 100000 --repeat 1 --vec-backend numpy,sqlite-vec,binary --lexical-backend fts5 --corpus-cache ``` **Default-backend flip (2026-07-05): numpy is now the default; sqlite-vec is opt-in.** The `add-sqlite-vec-backend` decision above defaulted `auto` to vec0-f32 on the *synthetic* evidence, where its brute f32 scan lost the warm race to numpy but won on residency. Real-vault measurement reversed the call. On the owner's production vault (41k chunks on a spinning D: drive), the vec0 f32 KNN measured 123–134ms only while its ~127MB of vec tables stayed warm in the OS page cache; as soon as the cache evicted them (routine on a working machine) the same query cost 1–12s. The RAM-resident numpy scan with the generation-keyed matrix cache (#125) measures 24–65ms and is write-independent — no page-cache cliff. Both backends are exact and rank-identical (the golden floors and the f32 parity tests still hold), so recall is unaffected; this is purely a latency/robustness decision. The trigger was also operational: the old probe-and-activate `auto` silently flipped production onto vec0 the moment a `uv sync` installed the extension — a behavior change no one requested. `EXOMEM_VEC_BACKEND` now defaults to `numpy`, and vec0 activates ONLY on the explicit value `sqlite-vec`; there is no `auto`. The at-scale synthetic numbers in the table above are unchanged and still measured by naming both backends explicitly (`--vec-backend numpy,sqlite-vec,binary`). ## Lexical backend at scale (10k–50k notes, FTS5 vs in-process) The `add-fts5-lexical-backend` change moved the bm25 and keyword lanes onto an FTS5 inverted index + trigram table in a per-vault `.lexical.sqlite` sidecar (`EXOMEM_LEXICAL_BACKEND=auto`, kill switch `=python`), built from the same pre-stemmed tokens the in-process scorer uses. This section is the measured before/after — and a correction to how the earlier curve's numbers should be read. **Correction: the registry-cold walk artifact.** The per-lane curve above (and the 10k/50k lane numbers quoted from it in the FTS5 change's proposal) was measured with the event-maintained freshness registry accidentally WIPED between seeding and measurement (`latency_curve` seeded, then `clear_cache()` cleared the registry too — fixed in this change). Registry-cold, every request re-derives its corpus freshness triples by a full stat walk, and with the hot cache disabled that walk lands inside the first lane that asks: the graph lane's `graph.resolver` sub-span (vault triple) and the bm25 lane (scope triple). Measured at 10k notes, python backends: registry-cold graph = 1130.7ms of which `graph.resolver` = 1126.7ms; registry-live graph = 4.6ms. That is why bm25/keyword/graph tracked each other so closely in the historical curve (each ≈ the walk): **the "graph wall" was the walk, not graph work** — the resolver itself was event-maintained and warm all along, as the new `graph.seeds` / `graph.resolver` / `graph.expand` sub-spans in FindTimings now show directly. Production servers run registry-live (the watcher maintains it); one-shot CLI processes still pay one honest walk per request — the freshness contract, not a regression. Method: same cached dense corpora and query set as the curve above, model-free, registry-live, hot cache off, `--lexical-backend python,fts5` (median / p90 ms; repeat=2 at 10k, repeat=1 at 50k so the p90 column at 50k is effectively the first, cache-cold query of a pass): | Notes | backend | bm25 | keyword | graph | total | |---|---|---|---|---|---| | 10000 | python | 15.4 / 18.0 | 1322.7 / 1519.3 | 4.0 / 4.4 | 1389.1 / 2798.6 | | 10000 | **fts5** | 17.6 / 20.8 | **3.7** / 73.9 | 4.3 / 4.7 | **82.1** / 1360.4 | | 50000 | python | 96.9 / 113.8 | 6552.9 / 7473.4 | 4.4 / 4.5 | 6760.3 / 13929.0 | | 50000 | **fts5** | 74.0 / 78.6 | **4.8** / 376.3 | 4.3 / 5.6 | **198.1** / 6700.0 | With the full stack live (real bge sidecar, vec0-f32 vector lane, rerank off, hot cache off, registry-live; measured via the graph-lane profile over the same 50k cached corpus): end-to-end `find()` median **864ms at 50k notes** — vector 592.9, bm25 81.4, keyword 8.7, graph 8.9, fusion 61.3. The whole call cost ~25s under every backend when this change was specced. Reading it: - **The keyword lane stops being O(N):** 6.55s → 4.8ms at 50k (trigram postings instead of a per-page stat + substring scan), at EXACT parity with the reference scan (the parity suite asserts identical match sets, including mid-word and sub-trigram needles). - **bm25 python was never the 8s monster the registry-cold curve suggested** (~97ms warm at 50k once the walk artifact is removed), but the FTS5 rung still wins (74ms at 50k), needs no per-process token-list residency, and keeps no rebuild cliff on corpus change. Its `bm25()` scorer differs from BM25Okapi, so promotion was gated on the golden floors + per-query pins (including the new morphological-variant stemming pin) — all green under both backends. - **The warm graph lane is flat** — 3.8ms @ 2k → 8.9ms @ 50k (live vector lane) — and `tests/test_latency_gate.py` now pins that shape with a 2k→8k scaling-ratio bound (1.5× + 25ms noise floor) plus the absolute ceilings, so a linear warm cost cannot return silently whatever its mechanism. - **Sub-second at 50k is a measurement** (864ms full-stack, dominated by the vector lane's query embed + KNN); the 100k story rides the same sub-linear lanes. The dedicated 100k measurement pass was descoped 2026-07-04 (see the vector section's ANN decision record for what was measured instead and why). Decision record: `auto` defaults to FTS5 for both lexical lanes. `EXOMEM_LEXICAL_BACKEND=python` restores the in-process paths wholesale; every FTS5 unavailability (SQLite without FTS5/trigram, unreadable sidecar, runtime error) soft-fails to them silently. The sidecar rebuilds from markdown on count/mtime drift (deleting it is always safe), writer/watcher/reconcile keep it in lockstep through the shared `index_sync` dispatch, and lean installs maintain it — FTS5 ships in CPython's bundled SQLite, no extra, no extension. Reproduce: ``` uv run python scripts/latency_curve.py --sizes 10000,50000 --repeat 2 --lexical-backend python,fts5 --corpus-cache ``` ## Real-vault check: the same lanes on a production vault (2026-07-04) Synthetic curves are a relative-grade instrument; this section is the absolute check against a real, media-heavy ~1,700-note production vault (the same class of vault as the historical ~14s graph event). Method: fresh, unique natural-language queries — cache misses — through the running server's `find(include_timings=True)`, reading the server-side per-lane milliseconds. Four varied queries; per this page's no-query-text rule they are not reproduced here. The "before" side was measured against a service process started 2026-07-03, BEFORE the FTS5 lexical backend (#113) had landed on its checkout; the "after" side minutes after restarting that service onto 0.7.0, same vault, same four queries (the two sides were probed by different sessions at different result limits, so read orders of magnitude, not small deltas). Ranges are min–max across the four queries: | Lane (ms) | before: stale pre-#113 process | after: 0.7.0 | |---|---|---| | keyword | 565 – 1,016 | 5.1 – 7.3 | | graph | 13 – 2,550 | 21 – 30 | | outside-KB widen | 550 – 599 | 3.0 – 15.8 | | bm25 | 8 – 806 | 3.0 – 3.2 | | vector | 28 – 68 | 16 – 19 | | clip | 38 – 88 | 21 – 24 | | **find() total** | **1,700 – 4,600** | **94 – 159** | Reading it: - **A ~25× total win from a restart, not a code change.** Every millisecond of it had already shipped; the running process simply predated its own checkout's fixes. Benchmark-vs-production gaps have TWO failure modes — instrument mismatch and deployment staleness — and this one was staleness: the synthetic curve had correctly caught the O(N) keyword scan (that is why FTS5 exists), and the production service wasn't running it. - **The graph "wall" collapses to 21–30ms flat**, with `graph.resolver` at ~0.01ms. The wild 13→2,550ms variance on the stale build matches missed- filesystem-event freshness re-walks landing inside the request that trips them (the same walk artifact documented in the correction above), not uncapped fan-out — seed expansion is capped (`graph_seed_cap`). - **The outside-KB auto-widen was the one genuinely invisible lane**: the synthetic corpus puts every note inside `Knowledge Base/` (the lane costs ~0 by construction) and the harness didn't report its stage. The `outside_kb` column is reported as of 2026-07-04; corpus-shape realism (outside-KB siblings, hub-degree, note-size distribution) is recorded follow-up work. - **Vector is a footnote at real scale** — 16–19ms under the vec0 default — which is the load-bearing evidence in the ANN decision record above. ### Whole-tool latency (server call log, real usage) Per-lane `find()` numbers miss the rest of the tool surface. From the server's own call log (`event=tool_success … duration_ms=…`) over the stale build's final ~36h of real usage, plus post-restart probe samples on 0.7.0: | tool | n | median ms | p90 ms | max ms | 0.7.0 probe | |---|---|---|---|---|---| | replace | 1 | 19,792 | — | 19,792 | — | | note | 51 | 7,743 | 21,877 | 55,753 | 15,237 cold → **2,578** after the resolver fix | | find | 69 | 3,704 | 16,876 | 50,974 | 94 – 159 | | edit | 46 | 2,872 | 18,943 | 65,059 | — | | link | 3 | 2,606 | 2,692 | 2,692 | — | | delete | 1 | 1,493 | — | 1,493 | 1,078 – 1,563 | | suggest_links | 5 | 762 | 3,886 | 3,886 | — | | append_to_file | 4 | 482 | 1,080 | 1,080 | 429 | | get | 75 | 4 | 8 | 64 | — | The read path (`get`, 4ms) was never a problem, and the raw write/upsert path is fine (`append_to_file` ~0.4s includes re-embed + index sync). The compiled- write family's cost was root-caused the same day: **every write op rebuilt the `WikilinkResolver` from scratch per call** — a full vault read + YAML parse of every .md file, cProfiled at ~2.1s of a 4.6s `note()` on a ~1,900-file copy of this vault. This is the same failure class as the embedding-matrix cache bug ("holder rebuilt per call"): find()'s graph lane got the freshness-checked shared-resolver cache long ago, and the write path never did. Fixed by routing all seven writer sites through `find.shared_resolver` with post-write re-sync via `index_sync` (also closing the async watcher window that invalidated the cache after every write) — pinned by `tests/test_write_resolver_reuse.py`. Measured: harness `note()` 4.6s → 3.2s (registry-cold; the residual is the tool's *designed* inline work — a link-suggestion find pass + draft near-dup/contradiction embedding); live cold-probe `note` **15.2s → 2.6s**. `edit`/`replace` ride the same fix; fresh live distributions accumulate in the call log. ## Reproduction The report reproduces in the same shape against any vault + golden set that follow the harness contract. It needs live vectors (the embedding/reranker models), so the full `--report markdown` is a desk-side command; CI asserts the lighter regression floors (mean NDCG@10 / MRR / recall@10, plus a per-query "nothing dropped to recall@10 = 0" guard) in the dedicated `retrieval-eval` job via `tests/test_retrieval_golden.py`. Against your own vault and golden queries: ``` EXOMEM_VAULT_PATH=/path/to/your/Obsidian \ python scripts/eval_retrieval.py --report markdown --golden tests/golden/queries.yaml ``` As a deterministic, smoke-scale run against the bundled fixture vault (no private vault required — same report shape, tiny corpus). Because the fixture ships without an embedding sidecar, build one first, in a throwaway copy so the repo tree stays clean (this is exactly what `tests/test_retrieval_golden.py` does): ``` cp -r tests/fixtures /tmp/exomem-fixture EXOMEM_VAULT_PATH=/tmp/exomem-fixture python -c \ "from pathlib import Path; from exomem import embeddings; \ print('chunks:', embeddings.get_embedding_index(Path('/tmp/exomem-fixture')).rebuild_all())" EXOMEM_VAULT_PATH=/tmp/exomem-fixture python scripts/eval_retrieval.py --report markdown ``` Skip the build step and every mode reports the same `keyword` numbers — the vector lane has nothing to read. Set `EXOMEM_BENCH_HARDWARE` to record the host in the report's methodology line. ## Limitations - **Small.** n = 26 golden queries. Individual metric moves are noisy at this size. - **Tiny corpus.** The quality numbers are measured on the bundled fixture vault — ~20 KB notes, 198 chunk vectors. They show the ranking *works* (hybrid ≫ keyword, nearly every target recalled), not production-scale quality, and do not generalize to larger or different corpora. Point the harness at your own vault for a representative read. The per-lane latency curve above uses a *separate*, larger synthetic corpus precisely because this fixture is too small to expose latency scaling. - **Self-graded.** Relevance labels are hand-authored (by the maintainer, not independent annotators) — a self-measurement, not a third-party benchmark. - **Hardware-dependent latency.** The median/p90 numbers reflect one host's CPU/GPU. A third party's latency will differ; the published p90 is not a portable guarantee. - **Synthetic corpus shape.** The dense synthetic vault matches real vaults on corpus SIZE but not on every cost-driving axis: link degree is uniform (25/note, no hub notes), note bodies are uniform (no transcript-sized pages), and every file lives inside `Knowledge Base/` — so the `outside_kb` auto-widen lane costs ~0 by construction (its column exists in the harness as of 2026-07-04 so the omission is at least visible). A synthetic ranking transfers only when a shape mismatch hits both arms of a comparison equally; the real-vault section above is the absolute check. Opt-in corpus axes (outside-KB sibling folders, hub-degree, heavy-tailed note sizes) are the recorded follow-up — as NEW cached tiers, because the existing corpora and published curves depend on byte-identical regeneration.