# haiku.rag Project Guide

Agentic RAG system built on LanceDB with hybrid search, multiple embedding providers, reranking, and a native Pydantic AI RAG capability with sandboxed code execution.

## Quick Reference

```bash
uv sync                                    # Install dependencies
pytest                                     # Run tests
ty check                                   # Type checking
ruff check && ruff format                  # Lint and format Python
cd app/frontend && biome check             # Lint and format frontend (TypeScript/JSX)
haiku-rag <command> --db /tmp/test.lancedb # CLI with temp DB (ALWAYS use temp DB for testing)
haiku-rag mcp --stdio                      # MCP server for Claude Desktop
```

## Project Structure

**Two PyPI packages:**
- `haiku.rag` - `haiku.rag-slim` with the docling, voyageai, cohere, zeroentropy, tui, cross-encoder and jina extras
- `haiku.rag-slim` - Core with optional extras

**Layout:**
```
haiku_rag_slim/haiku/rag/   # Source code
├── store/                  # LanceDB persistence
│   ├── engine.py           # Store class: connection, locks, migrations, vacuum, tags
│   ├── schema.py           # Table records (DocumentRecord, ChunkRecord, SettingsRecord), Arrow schemas, index_specs/ensure_indexes
│   ├── compression.py      # docling blob compression
│   ├── info.py             # gather_database_info, get_database_stats, DatabaseInfo
│   ├── models/             # Domain models: chunk (Chunk, SearchResult), document, document_item, citation
│   ├── repositories/       # CRUD: DocumentRepository, ChunkRepository, DocumentItemRepository, SettingsRepository
│   ├── upgrades/           # Version migrations (v0_75_0 … v0_91_0)
│   └── exceptions.py       # ReadOnlyError, MigrationRequiredError, UpgradeRequiredError, AmbiguousDatabaseError,
│                           # UnknownDatabaseError, AmbiguousCitationError, SourceUnavailableError, ConfigMismatchError, TagError
├── embeddings/             # VoyageAI, Cohere, vLLM, OpenRouter (ollama/openai via pydantic-ai)
├── reranking/              # cross-encoder, Cohere, Zero Entropy, Jina, Jina-local, vLLM, OpenRouter
├── sandbox/                # pydantic-monty sandbox used by RAGCapability and MCP execute_code
│   ├── sandbox.py          # Sandbox, SandboxResult
│   └── dependencies.py     # AnalysisContext (per-invocation filter)
├── providers/              # docling_serve.py (docling-serve client), picture_description.py
├── capabilities/           # Native Pydantic AI capabilities
│   ├── _tools.py           # Shared capability primitives
│   ├── ledger.py           # CapabilityEvidenceRecord, citation_status
│   ├── evidence.py         # discover_evidence(), DiscoveredEvidence
│   ├── compaction.py       # EvidenceCompactionCapability, build_capsule, compact_history
│   ├── policy.py           # CitationPolicyCapability, CitationPolicyState
│   ├── rag.py              # RAGCapability, RAGState, create_capability
│   └── instructions/       # Model instructions (rag.md, rag_multiple_collections.md)
├── utils/                  # Shared helpers organized by purpose
│   ├── models.py           # Model construction and provider settings
│   ├── images.py           # Image and retrieved-picture helpers
│   ├── sql.py              # SQL escaping and document ID filters
│   ├── formatting.py       # Citation and byte formatting
│   ├── concurrency.py      # Concurrent awaitable handling
│   ├── http.py             # Pooled httpx client, retry of a dropped connection
│   ├── paths.py            # Database and data directory paths
│   ├── packages.py         # Package versions and update checks
│   └── dependencies.py     # Optional dependency errors
├── chunkers/               # docling-local, docling-serve
├── converters/             # docling-local, docling-serve, pdf_split.py, text_utils.py, exceptions.py
├── config/                 # models.py (all config classes), loader.py
├── chat/                   # Chat TUI (app.py, widgets/)
├── inspector/              # Inspector TUI (app.py, widgets/)
├── client/                 # HaikuRAG high-level API
│   ├── __init__.py         # Lazy facade: HaikuRAG, RebuildMode, DatabaseScope, DocumentImport, all_found
│   ├── client.py           # HaikuRAG class, RebuildMode
│   ├── scope.py            # DatabaseScope, DatabaseRef (which databases an operation covers)
│   ├── session.py          # SingleDatabaseSession, FederatedSession
│   ├── documents.py        # create/import/update_document(s), create_document_from_source, DocumentImport
│   ├── processing.py       # convert, chunk
│   ├── titles.py           # generate_title
│   ├── search.py           # search, expand_context, visualize_chunk
│   ├── agents.py           # ask
│   ├── rebuild.py          # rebuild_database
│   ├── downloads.py        # download_models
│   └── exceptions.py
├── ingester/               # haiku-ingester service (own CLI + [ingester] extra)
│   ├── cli.py, app.py      # haiku-ingester entry point + application layer
│   ├── batch.py            # run-batch, dry-run manifests
│   ├── reconcile.py        # Startup reconciliation of LanceDB and sync_state
│   ├── maintenance.py      # maintain(), DatabaseMaintenance: vacuum + missing vector index per database
│   ├── metadata.py         # Metadata provider loading
│   ├── queue/              # db.py, migrations.py, repository.py (JobRepo/SyncStateRepo), models.py
│   ├── workers/            # pool.py (WorkerPool), pipeline.py (run_job), retry.py
│   ├── pollers/            # base, periodic, fs, manager, factory
│   ├── api/                # FastAPI control plane (server.py, auth.py, schemas.py)
│   │   ├── routes/         # health, sources, jobs, dlq, stats, config, database, dashboard, providers, reload
│   │   └── static/index.html # Self-contained vanilla-JS dashboard served at /
│   └── exceptions.py       # PermanentError, TransientError
├── sources/                # base, fs, http, s3, webdav adapters + registry, plugins, filter, walk_files
│                           # (used by the ingester AND one-shot client ingestion)
├── app.py                  # HaikuRAGApp CLI application layer
├── cli.py                  # Typer CLI entry point
├── context.py              # Section-bounded context expansion, build_toc
├── mcp.py                  # FastMCP server (create_mcp_server)
├── doctor.py               # Database health checks, provider probes, near-duplicate detection
├── circuit_breaker.py      # CircuitBreaker (ingester pollers and workers, docling-serve provider)
├── hooks.py                # Hook, IngestRequest, IngestEvent, DeleteEvent, SearchRequest, build_hooks, notify
├── s3.py                   # Object-store helpers
├── telemetry.py            # Logfire configuration
├── logging.py              # Logging configuration
└── uri.py                  # is_local_uri, uri_to_path
tests/                      # Mirrors source structure
├── conftest.py             # Fixtures (see Testing), capture_logs, for_path, writing
├── cassettes/              # VCR cassettes for the central suites (see Testing)
├── json_body_serializer.py # Custom VCR serializer for JSON bodies
├── docker/                 # docker-compose.yml for integration services
├── client/ chunkers/ converters/ embeddings/ interfaces/ providers/ reranking/ store/
├── capabilities/ chat/ ingester/ multi_db/ sandbox/ sources/ utils/
└── data/                   # Test data files
evaluations/                # Benchmarking workspace
├── configs/                # Reference configs per dataset
├── scripts/
├── evaluations/
│   ├── benchmark.py        # Typer CLI: run, pair, download, upload
│   ├── population.py       # populate_db
│   ├── retrieval.py        # run_retrieval_benchmark
│   ├── qa.py               # run_qa_benchmark, run_live_qa_benchmark
│   ├── results.py          # Per-case result files
│   ├── pairing.py          # evaluations pair
│   ├── artifacts.py        # download_dataset_db, upload_dataset_db, HF_REPO_ID
│   ├── experiment.py       # build_experiment_metadata
│   ├── config.py           # DatasetSpec, DocumentPayload, RetrievalSample
│   ├── capability_runner.py # native capability evaluation runner
│   ├── datasets/           # enterprise_rag, frames, hotpotqa, mtrag, open_rag_bench, t2_ragbench
│   └── evaluators/         # judge, map, citation, retrieval, number_match, refusal, answer_facts, conversation, transcript
└── tests/                  # benchmark and capability-runner tests
docs/                       # Documentation (zensical, not mkdocs; nav lives in zensical.toml)
docker/                     # Dockerfile (full), Dockerfile.slim (published)
examples/                   # Working examples (docker/, custom agents)
plugins/haiku-rag/          # Claude Code / Codex plugin (see MCP Server)
scripts/                    # bump_version.py, build-docker-images.sh, run-integration-tests.sh
app/                        # Web application: chat, ingestion, inspector, doctor (see below)
```

## Architecture

**Key Classes:**

| Class | Location | Purpose |
|-------|----------|---------|
| `HaikuRAG` | client/client.py | High-level API (main entry point) |
| `HaikuRAGApp` | app.py | CLI application layer |
| `Store` | store/engine.py | Async LanceDB connection, tables, upgrades |
| `DatabaseScope` | client/scope.py | The databases an operation covers, resolved once |
| `SingleDatabaseSession` | client/session.py | One database: store, repositories, lifecycle. Reads and writes |
| `FederatedSession` | client/session.py | Several, composed lazily. Reads only |
| `DocumentRepository` | store/repositories/document.py | Document CRUD |
| `ChunkRepository` | store/repositories/chunk.py | Chunk CRUD + search |
| `DocumentItemRepository` | store/repositories/document_item.py | Document items, pictures |
| `SettingsRepository` | store/repositories/settings.py | The version and embedder a database records (`recorded_settings`) |

**Factory Functions:**
- `get_embedder(config)` → `EmbedderWrapper` (embeddings/__init__.py)
- `get_reranker(config)` → `RerankerBase | None` (reranking/__init__.py)
- `get_converter(config)` → `DocumentConverter` (converters/__init__.py)
- `get_chunker(config)` → `DocumentChunker` (chunkers/__init__.py)
- `create_capability(db_path?, config?, *, defer_loading=False, rag=None, request_limit=30, sources=None, vision=None)` → `RAGCapability` (capabilities/rag.py). Tools `search(query, limit?, sources?)`, `execute_code`, `cite`, state under `rag`.
  - `request_limit` counts model requests per agent run. At the limit the capability's tools are removed except `cite`, which survives `CITATION_GRACE_REQUESTS` (2) further requests that call one of the capability's tools. Requests spent on other tools do not count.
  - A spent search or code budget does not withdraw the tool: it keeps failing. Withdrawing a tool the model still calls costs the agent's unknown-tool retries and aborts the run.
  - `vision` gates image attachment on search results and must reflect the model the hosting agent runs. Default `qa.model.vision`.
  - `rag` lends an open client: it lands in `borrowed_rag`, which `_ensure_rag` prefers and `_close` never closes.
  - `from_spec` delegates here so agent specs can declare the capability. `db_path` accepts a `str`.
- `create_capability()` → `EvidenceCompactionCapability` (capabilities/compaction.py). Optional, registering it is the only switch. Replaces earlier questions' evidence on the *request* with a capsule of what was cited (pictures included, fetched through the owning capability). Other earlier evidence returns as a receipt. No config, no budget: it reduces a request without bounding it.
- `create_capability()` → `CitationPolicyCapability` (capabilities/policy.py). Optional. Requires every answer to declare its grounding, asks a plain-text answer once per question, never asks a structured answer, records failures in `CitationPolicyState.violations`, never asks the model to change its answer. Enforced when the question retrieved evidence or the conversation already cited something.

**Base Classes:**
- `EmbedderWrapper` (embeddings/__init__.py) — wraps pydantic-ai `Embedder`
- `RerankerBase` (reranking/base.py)
- `DocumentConverter` (converters/base.py)
- `DocumentChunker` (chunkers/base.py)

**Domain vs Persistence:**
- Domain: `Document`, `Chunk`, `SearchResult`, `Citation` (Pydantic BaseModel in store/models/)
- Persistence: `DocumentRecord`, `ChunkRecord`, `SettingsRecord` (LanceModel in store/schema.py)

**Hooks** (hooks.py, user docs in docs/hooks.md):
- A package registers a factory under the `haiku.rag.hooks` entry-point group. `config.hooks` activates it by name or as `{name, options}` (`HookConfig`), and the options are the factory's keyword arguments. `HaikuRAG.__init__` builds the hooks with `build_hooks`, which loads only the entry points the config names. An unknown name raises `ValueError` at construction.
- Hook points, run in configured order: `before_ingest`, `after_convert`, `after_ingest`, `after_delete`, `before_search`, `after_search`, `lifespan`, `ingester_lifespan`. `before_ingest` refuses a document with `DocumentRejected`, and any other exception fails the ingest as `HookError`. `after_convert` (`ConvertedDocument`) runs between haiku.rag's conversion and chunking wherever it converts: client/documents.py (source, `create_document`, `update_document(content=)`) and FULL rebuild's stored-content fallback, which then also writes `content`. Never on a caller-supplied docling document or a rechunk. It has no rejection. Search hook failures propagate. `after_ingest` and `after_delete` run through `notify`, which logs a failure and never raises. A FULL rebuild passes `notify_hooks=False`.
- One set of hook instances and one lifespan per client that owns its session. A borrowed client (`clients_for`) gets `_hooks = []`. The ingester opens one client per database, so each database has its own instances and lifespan.
- `ingester_lifespan` is entered only by `IngesterApp._apply` (`serve`, `run-batch`, manifest replay, reload), through `HaikuRAG._ingester_lifespans`: after reconcile, before the database's sources reach the pool. `_Database.close_client` exits it before the client closes, so every close path takes it, after the pool has stopped serving the database. Background work goes there, never in `lifespan`, which every process enters.
- A hook's own tables take the `hook_` prefix.
- The ingester's reload refuses a `hooks` change (`changed_sections`), so it needs a restart.
- How long metadata from the caller, provider, `before_ingest` and `after_ingest` lasts: docs/hooks.md, "Metadata lifetime".

## HaikuRAG Client API

Key methods on `HaikuRAG` (client/client.py):

```python
async with HaikuRAG(db_path, config, create=True) as rag:
    # Document operations
    doc = await rag.create_document(content, uri, title, metadata)
    doc = await rag.create_document_from_source(path_or_url, title=..., metadata=...)
    doc = await rag.import_document(docling_doc, chunks, uri, title, metadata)
    docs = await rag.import_documents([DocumentImport(docling_doc, chunks, uri, title, metadata), ...])  # one table version per table
    doc = await rag.get_document_by_id(id)
    doc = await rag.get_document_by_uri(uri)
    docs = await rag.list_documents(limit, offset, filter)
    count = await rag.count_documents(filter)
    await rag.update_document(document_id, content=..., metadata=..., chunks=..., title=..., docling_document=..., uri=...)
    await rag.delete_document(id)

    # Processing (no database access)
    docling_doc = await rag.convert(Path("file.pdf"))  # a str that is not a URL is converted as text
    chunks = await rag.chunk(docling_doc)
    title = await rag.generate_title(doc)

    # Search & QA
    results = await rag.search(query, limit, filter)    # not expanded
    expanded = await rag.expand_context(results)
    answer, citations = await rag.ask(question, filter=None, images=None)  # images: Sequence[bytes], needs vision: true

    # Lookups
    doc = await rag.resolve_document(id_or_title)
    chunk = await rag.get_chunk_by_id(chunk_id)
    png = await rag.get_picture_bytes(document_id, self_ref)

    # Maintenance
    async for doc_id in rag.rebuild_database(mode=RebuildMode.FULL): ...  # also RECHUNK, EMBED_ONLY, TITLE_ONLY, DESCRIPTIONS, SET_EMBEDDER, REINDEX
    await rag.vacuum()
    built = await rag.create_vector_index()  # False below 256 chunks and on Cloud; writes wait for the build

    # Tags (Store-level; TagInfo has tables/missing_tables/complete)
    await rag.store.create_tag("release-1")
    tags = await rag.store.list_tags()
    safety_tag = await rag.store.restore_tag("release-1")
    await rag.store.delete_tag("release-1")

    # Visualization
    images = await rag.visualize_chunk(chunk, source=result.source)  # source required over a set

    # Coverage (see "Multiple Databases")
    rag.covers_multiple      # more than one database
    rag.source_names         # configured names covered, in order
    rag.source               # the one name, or None while covering a set
    rag.location             # configured URI or path, None while covering a set
    owner = await rag.reader_for("papers")          # the client reading that database
    papers, wiki = await rag.clients_for(["papers", "wiki"])
    covering = await rag.clients_covering(sources)  # honours a per-query selection
```

`download_models(config)` (client/downloads.py) is a module function, not a method.

## Multiple Databases

`lancedb.databases` maps a name to a location. The name is the only identity that
leaves the configuration: it travels in `SearchResult.source`, `Citation.source`,
`Document.source` and errors, where a location must not.

- **The configuration places databases, arguments select.** `DatabaseScope.resolve(config, database_name=, database_path=)`
  is the single place selection happens, and it never mutates config. Every
  database has a name: the `lancedb.databases` key, or the path's stem. A name
  selects one configured entry. A path places a database only where the
  configuration places none, and beside `lancedb.databases` raises
  `AmbiguousDatabaseError`. Neither selector covers the configured set, a set of
  one included. Nothing configured is the one entry `haiku.rag` at
  `storage.data_dir / "haiku.rag.lancedb"`, selectable by name.
- **`--db` is the one human override:** `DatabaseScope.at(path)` consults no configuration.
  `DatabaseRef.given` marks a caller-given path whose errors may name it. A
  configured or default database's errors name the database and never its
  location (`SourceUnavailableError`, with the remedy for a missing one).
- **No derived configs.** `lancedb.uri` and `HAIKU_RAG_DB` do not exist, and a config carrying `uri` fails validation naming `databases`. The name always travels with the scope: `create_mcp_server` and the capabilities take or resolve one, and `ChatApp` sets `capability.scope`.
- **An entry may carry a tag** (`DatabaseEntry`), and entries may share a location. No check refuses covering two states of one database: shared documents return once per entry and `AmbiguousCitationError` guards citation.
- **Never manufacture a default path.** `HaikuRAG(None, config)` resolves the configuration. A manufactured `data_dir / "haiku.rag.lancedb"` beside a configured placement raises `AmbiguousDatabaseError`. Downstream hosts, the ingester included, build clients from config alone.
- **Two sessions, two search paths.** `SingleDatabaseSession` reads and writes.
  `FederatedSession` composes single sessions lazily and reads only. A selection of
  one runs the ordinary single-database search: fusion would replace the
  database's hybrid scores with ranks, and embedding up front would embed for a
  filter the repository can see matches nothing.
- **Fusion.** With a reranker, it scores the union directly. Without one, the
  union is ordered by cosine similarity to the query vector, which is comparable
  across databases because the selection shares an embedder. Full-text-only
  searches order by retrieval score. Ties resolve by within-database rank, then
  configured order. Fused results carry the ordering score, so context expansion
  preserves fused order. `Chunk.embedding` is populated only when cosine fusion
  will read it (`with_vectors` mirrors `_fuse`'s cosine condition; an image query
  with a reranker configured still takes cosine). Over-fetch (`limit * search.rerank_pool_factor`) only
  with a reranker. Image queries are vector-only and skip the reranker.
- **`_fuse` maps a reranked chunk back by `id(chunk)`**, since chunk ids repeat between copies. Every shipped reranker indexes into the list it was given. One that returns copies raises a named `ValueError`.
- **One reranker per set.** A client covering a database for another borrows the
  lender's (`_lender`, `_own_reranker`). Only the owner closes it.
- **Collisions.** Chunk ids repeat between copies of a database. In-memory
  identity uses `qualified_id(source, id)`. Serialized structures cannot
  represent it, so a cited id retrieved from, or previously cited from, more than
  one selected database raises `AmbiguousCitationError`. A copy the search never
  returned grounded nothing, so one retrieved result resolves normally.
- **`clients_covering(sources)` opens the databases; `_require_known_sources` only checks names.** Validating at an operation boundary must use the latter, or an unscoped question opens every configured database before the model runs.
- **`UnknownDatabaseError` subclasses `KeyError`** and overrides `__str__`. `pytest.raises(KeyError)` cannot tell the contract from a bare one, so assert the specific type.
- **Model context** names the database as a `Collection:` line whenever the run
  spans more than one (`RAGCapability.collections`: the question's `sources`, else
  the lent client's coverage, else the scope), including a search narrowed to one,
  and the instructions of such a run list its collections by name. Storage
  vocabulary stays "database", the model boundary says "collection".

## Configuration

**Config classes** (config/models.py):
- `AppConfig` - Root config
- `StorageConfig` - data_dir, auto_vacuum, vacuum_retention_seconds, compaction_target_bytes
- `EmbeddingsConfig` / `EmbeddingModelConfig` - provider, name, vector_dim, multimodal (gates image embedding), query_prefix, document_prefix (prepended to query and chunk text, ollama/openai/vllm/openrouter only), batch_size
- `RerankingConfig` - optional reranker model, multimodal (vllm and openrouter)
- `ModelConfig` - provider, name, base_url, api_key, thinking, temperature, max_tokens, vision, extra_body
- `QAConfig` - model, max_searches (default 5), max_executions (default 15), max_images (default None). The budgets are per question. `max_searches` counts search *units*: searches emitted in one model response (same `RunContext.run_step`) share a unit, up to `FREE_SIBLINGS_PER_ROUND` (3) per unit, so a run searches at most `max_searches × 3` times and a sequential searcher pays one unit per search. A budget-rejected round fails its remaining siblings
- `SandboxConfig` - code_timeout, max_output_chars: limits of one `execute_code` call, read by the capability and the MCP tool. In-code `search()` does not count against `qa.max_searches`, so code execution outlives a spent search budget
- `PictureDescriptionConfig` - picture description model settings
- `ProcessingConfig` - chunk_size, converter, chunker, chunker_type, conversion_options, pictures, split_pages, conversion_timeout, auto_title, title_model
- `SearchConfig` - limit, max_context_chars, rerank_pool_factor, vector_index_metric, vector_refine_factor, vector_nprobes
- `DoctorConfig` - duplicates (DuplicateDetectionConfig: similarity_threshold 0.97, min_chunks 3)
- `ProvidersConfig` - ollama, docling_serve (OllamaConfig, DoclingServeConfig)
- `PromptsConfig` - domain_preamble, picture_description
- `IngesterConfig` - databases (database name → `IngestedDatabase`: sources), queue (QueueConfig), workers (WorkerConfig), api (APIConfig), maintenance (MaintenanceConfig: enabled, quiet_s, max_wait_s)
- `EvaluationsConfig` - judge (ModelConfig | None), system_one (SystemOneConfig | None: provider, base_url, model; key from `TYPESAFE_API_KEY`)
- `LanceDBConfig` - databases (name → local path or URI), api_key, region, storage_options, read_consistency_interval_seconds, index_cache_size_bytes, metadata_cache_size_bytes

Every model inherits `ConfigModel` (`extra="forbid"`), so an unknown or misspelled key raises at load. converter, chunker and chunker_type are `Literal`s and numeric fields carry bounds. Provider fields stay unrestricted `str`: `get_model` passes an unknown provider to pydantic-ai by name, logging a warning when model settings are set, since none apply. Read config through `get_config()`: there is no module-level `Config` singleton, and capturing the config in a default argument freezes it before `set_config()` runs.

A `qa.model` block's unset fields take `ModelConfig` defaults (`vision=False`, `thinking=None`), not `QAConfig`'s default factory (`vision=True`, `thinking=True`, `temperature=0.3`). The same holds for `title_model` and `picture_description.model`.

Adding a field or sub-model to `AppConfig` needs no plumbing: the loader validates it, and `init-config` and `settings` serialize `AppConfig().model_dump()`.

**Search order:** `--config` / `HAIKU_RAG_CONFIG_PATH` → `./haiku.rag.yaml` → platform user directory

**Database fragments:** `load_yaml_config` pops the top-level `databases_include` and merges every `*.yaml` in that directory (`_merge_fragments`): only the two database maps, a name defined twice raises, `${HAIKU_RAG_CONFIG_DIR}` is the main file's directory. It is not an `AppConfig` field, so `AppConfig.model_validate` on a dict reads no fragments. Reload goes through the loader, so it re-reads them.

**Global CLI options:** `--db-name NAME` selects one entry from
`lancedb.databases` and is global, so it precedes the command. `--db PATH` is
per-subcommand and follows it. They are mutually exclusive. `search`, `ask`,
`chat` and `mcp` cover the configured set, everything else works on
one database. `settings`, `init-config` and `download-models` resolve no scope.
`--at TAG` is per-subcommand on the nine read commands (`search`, `ask`, `list`,
`get`, `visualize`, `chat`, `inspect`, `info`, `mcp`) and resolves to one
database, refusing a configured set. `resolve_scope(db, at=)` puts the tag on
that `DatabaseRef`. Write commands have no `--at`.

## CLI Commands

```
# Document management
add           Add document from text
add-src       Add from file/URL/directory/S3
get           Get document by ID
delete/rm     Delete document by ID
list          List documents (with optional filter)

# Search & QA
search        Hybrid search (vector + full-text), not context-expanded
ask           QA via the RAG capability (always shows citations; --image PATH repeatable)

# Maintenance
init          Initialize new database
rebuild       Re-convert, re-chunk, re-embed (--rechunk, --embed-only, --title-only, --descriptions, --set-embedder, --reindex)
vacuum        Optimize and clean up tables
migrate       Database migration
create-index  Create vector index for similarity search
info          Show database info (read-only)
history       Show version history for tables (tagged versions annotated)
tag           Manage database tags: tag create/list/delete/restore NAME
doctor        Check database health, probe provider connectivity, detect near-duplicate documents (exits 1 on failure; --duplicates-out exports groups to YAML)

# Configuration
settings      Display current config
init-config   Generate YAML config file

# Server
mcp           Run the MCP server (--stdio, --host, --port); read-only, covers the configured set

# Tools
visualize     Show visual grounding for a chunk
inspect       Launch TUI to inspect database
chat          Launch TUI for conversational RAG
download-models  Download Docling, HuggingFace and Ollama models
```

`search`, `ask`, `list`, `get`, `visualize`, `chat` and `inspect` always open read-only. `cli()` turns the domain errors and `FileNotFoundError` into `Error: …` and exit 1.

**haiku-ingester** (separate entry point, `[ingester]` extra):
```
serve       Run pollers + workers, and the HTTP API unless --no-api; blocks until SIGINT/SIGTERM; SIGHUP or POST /reload re-reads the databases
run-batch   One discover sweep over every source (--db-name NAME: one database's), drain the queue, exit; non-zero on dead-letter or incomplete sweep
queue       init | migrate the queue DB; --dburi URL (or HAIKU_RAG_QUEUE_DBURI) names a server queue's owning account
```

**Tags:** `haiku-rag tag create/list/delete NAME` name database states across all five tables. `tag restore NAME` brings the live database back to a tagged state: it creates a `before-restore-*` safety tag first, requires stopped writers, and never migrates. A tag that needs no migration restores without the live state's migration check and config validation (`Store.tag_haiku_version`), so the safety tag stays reachable from an older tag. Vacuum keeps every tagged version and the files it references. Reading at a tag (`--at`, `HaikuRAG(tag=)`, `Store(tag=)`, a `lancedb.databases` entry `{location, tag}`, all landing on `DatabaseRef.tag`) is read-only: `Store` checks the tag on every table (`TagError`), checks out `settings` before reading it so the embedder check and the migration gate see the tagged state, then checks out the rest. A tag needing a migration raises `MigrationRequiredError`. `HaikuRAG.is_read_only` derives read-only from the scope's tags. `DatabaseRef.at_tag` refuses `--at` / `tag=` on a ref that already carries one. CLI commands without `--at` open raw `Store`s on the live location, so `resolve_scope` refuses a tagged ref unless the command passes `reads_tags=True` (the nine `--at` commands), and `ingester.databases` refuses one at load. `info` and the inspector's info modal read tables without `Store`, so they pass the tag to `gather_database_info` / `get_database_stats(db, tag=)`, which apply `require_complete_tag` and `require_current_tag` as `Store` does: a new read path that opens tables by name must do the same, or it reports the live state. Without a tag, `info` still reports a database that needs migrating.

## MCP Server Tools

Exposed via FastMCP (mcp.py). `create_mcp_server(db_path?, config?)` resolves a
scope. `_covering(scope, config)` takes one already resolved. The server opens
one read-only client for its lifetime and covers the configured database set.
There are no write tools: ingest with `haiku-rag add`/`add-src`/`delete` or `haiku-ingester`.

- `search_documents(query, limit?, include_images?, filter?, sources?)` → `ToolResult`
- `search_documents_by_image(image_base64, limit?, include_images?, filter?, sources?)` → `ToolResult` (registered only when the embedder `supports_images`)
- `get_document(document_id, source?)` → Document
- `get_document_outline(document_id, source?)` → list[OutlineNode]
- `get_document_section(document_id, section_id, source?)` → DocumentSection
- `list_documents(limit?, offset?, filter?, sources?)` → list[DocumentInfo]
- `execute_code(code, filter?, sources?)` → the program's stdout

**Search results are text, not JSON.** Both search tools expand through
`HaikuRAG.expand_context` and return the `format_for_agent` rendering (rank,
`Document ID`, `Collection` over several databases, matched chunk metadata,
passage) plus one `ImageContent` per distinct picture. They carry no structured
content, see the Claude Code gotcha under MCP. Every other tool returns a
pydantic model, whose JSON is the payload.

**`execute_code`** runs one program per call in the same Monty sandbox the
RAG capability uses, over the documents `filter` and `sources` select, and
returns what it printed. One sandbox per call: a session outliving the call
would never see documents ingested after its first mount. The server's value to a client that is already a model is
the sandbox, so there is no `ask_question` or `analyze` tool and
`haiku.rag.utils.formatting` has no `format_citations` (`format_citations_rich`
and the `_citation_*` helpers stay, for the CLI).

**One error contract.** `mask_error_details=False`, passed explicitly since the
setting is also read from the environment: every failure reaches the client as
an MCP error carrying its message, host errors inside a program included. A bad
filter fails in the read itself with the query engine's message and an unknown
collection with `UnknownDatabaseError`'s, so there is no pre-check query and no
`ToolError` translation. Explicit domain errors stay (`No document with id`,
`No section`, `Invalid base64 image`).

**Plugin bundle.** `plugins/haiku-rag/` holds one `.mcp.json` (`haiku-rag mcp
--stdio`) and one Agent Skill, `skills/haiku-rag/SKILL.md`, under two manifests:
`.claude-plugin/plugin.json` and `.codex-plugin/plugin.json`. Marketplaces are
`.claude-plugin/marketplace.json` (Claude Code) and `.agents/plugins/marketplace.json`
(Codex). Both manifests carry the package version: `scripts/bump_version.py`
rewrites both, and `tests/test_bump_version.py` pins them to
`haiku_rag_slim/pyproject.toml`. A test asserts the skill's `allowed-tools`,
prefix stripped, equal the tools the server registers. The shared `.mcp.json`
uses Claude's `mcpServers` key, which Codex's loader accepts although its
documentation does not list it: don't "fix" it to `mcp_servers`. `allowed-tools`
is Claude Code's pre-approval and other Agent Skills clients ignore it.

## Testing

**Key fixtures** (tests/conftest.py):
- `temp_db_path` - Isolated temp database (use this!)
- `qa_corpus` - Q&A pairs from `tests/data/qa_corpus.json`
- `temp_yaml_config` - Temp config file, pointed at via `HAIKU_RAG_CONFIG_PATH`
- `allow_model_requests` - Enables pydantic-ai model calls (disabled by default)
- `allow_expected_model_requests` - autouse: enables model calls for `vcr` and `integration` tests
- `set_mock_api_keys` - autouse: mock provider keys for playback
- `postgres_dburi`, `docling_serve_url` - integration services, skip when unreachable
- `capture_logs(logger, level)` - context manager collecting log records (see gotcha)

**Markers:**
- `@pytest.mark.integration` - External services (postgres, docling-serve, seaweedfs). CI excludes them. Start them with `docker compose -f tests/docker/docker-compose.yml up -d`. Each test skips when its service is unreachable.
- `@pytest.mark.slow` - Deterministic end-to-end tests. CI runs them separately with `pytest -m "slow and not integration"`.
- `@pytest.mark.vcr()` - HTTP call recording/replay

`pytest` runs the whole core suite, slow and integration included. CI's fast lane is `pytest -m "not slow and not integration" --cov`. `test-windows` runs it on Windows without `--cov`, as a check that is not required. `test-py314` runs it on Python 3.14 without `--cov`. `cd evaluations && uv run pytest --cov` runs the evaluations suite and its coverage gate. Tests run under xdist (`-n auto` in addopts); pass `-n0` when debugging races or container lifecycle. Async tests are detected through pytest-asyncio's auto mode, and asyncio fixtures default to **session** loop scope.

**Core coverage is enforced at 100%, branches included** (`branch = true`, `fail_under = 100`, `source = ["haiku_rag_slim"]`); the evaluations workspace enforces 85%. New code needs a test that takes every branch, or a `# pragma: no cover - <short reason>` or `# pragma: no branch - <short reason>` on one line. CI prints term-missing, so a gate failure names the line. Run coverage with `--cov` and scope the report by grepping it: a deep-dotted `--cov=haiku.rag.<module>` crashes beartype, and a file-path `--cov=<path>.py` collects nothing.

**VCR Recording:**
Tests use pytest-recording (VCR.py). Cassettes are committed: the suites in `_CENTRAL_CASSETTE_SUITES` (client, chunkers, converters, embeddings, interfaces, providers, reranking, store) record under `tests/cassettes/<module>/`, every other suite under `<suite>/cassettes/<module>/`.

```bash
# Run with recorded cassettes (default)
pytest

# Record new cassettes (exact test, serial, real services available)
pytest tests/embeddings/test_embedder.py::test_ollama_embedder -n0 --record-mode=rewrite

# Run against live services (disable VCR)
pytest --disable-recording

# Record with a real API key (each SDK reads its own variable, e.g. CO_API_KEY)
CO_API_KEY=... pytest tests/reranking/test_reranker.py::test_cohere_reranker -n0 --record-mode=rewrite
```

To add VCR to a new test, add `@pytest.mark.vcr()` and record with `pytest <path> -n0 --record-mode=once` against the real service. Without `--record-mode`, pytest-recording plays back in `none` mode, so an unrecorded call errors with a connection-style failure. Recording needs network.

Docling-serve polling and retry delays are skipped only during cassette playback. Recording and `--disable-recording` runs keep them.

## Code Conventions

- Python 3.12+ native typing (no `from __future__ import annotations`)
- Absolute imports (the `store` package `__init__` files are the only relative ones)
- Async/await for all I/O operations
- LanceDB is opened via `lancedb.connect_async`; all table reads/writes are awaited. `connect_async` requires absolute paths, so `connect_lancedb` makes a local location absolute. `Store(location, config)` takes a path or a URI; `Store.db_path` is `None` behind a URI.
- No module docstrings at top of .py files
- Type hints everywhere, Pydantic for validation
- Minimal comments (only explain WHY when not obvious)
- Every critical dependency carries an upper bound, and `pydantic-ai-slim` is capped below its next minor release. pydantic-ai changes capability hook semantics in minor releases: 2.54 moved `before_model_request` inside the `wrap_model_request` handler. Raise a cap only in a change that runs every suite and, when capability behaviour can move, the evals.

## Comments and Docstrings

**This section overrides "match the surrounding style".** Much of the existing
prose in this repo is longer than these rules allow; do not take it as the
standard to match.

- A docstring is one line by default. More than three lines needs a contract a
  caller cannot see from the signature — an ordering requirement, a value that
  looks wrong and is not, an invariant the caller must hold.
- A comment states what is true, never what used to be true, what would break
  without it, or what alternative was rejected. "would otherwise", "instead of",
  "rather than", "so that X does not" are the tells.
- Rationale, measurements and alternatives go in the commit message and the PR
  description. Not in the code.
- Test docstrings follow the same rules. Name the behaviour, not the bug.

Before proposing a commit, read the added comment lines on their own:
`git --no-pager diff -U0 --no-color | grep -E '^\+' | grep -E '#|"""'`

## Critical Rules

- ALWAYS use `--db /tmp/test.lancedb` when testing CLI commands
- Run `ruff check` and `ty check` after Python changes
- Run `biome check` in `app/frontend/` after frontend changes
- Update docs/ and README.md when changing functionality
- Online docs: https://ggozad.github.io/haiku.rag/

## Common Gotchas

Each entry is the trap and what to do. The evidence behind them is in the commit history.

### Store and LanceDB

- **One write transaction.** Multi-table writes go through `async with store.write_transaction():`. It takes `_write_lock`, snapshots `current_table_versions()`, and on any exception restores every table in `RESTORE_TABLE_ORDER` under a shield before re-raising (an absorbed `CancelledError` is re-delivered). Client code never restores tables itself.
- **Lock order** is `_rebuild_lock → _write_lock` (tag ops, restore, `create_vector_index`) and `_vacuum_lock → _write_lock` (vacuum). Rebuild holds `_rebuild_lock` for its whole run, and tag operations fail fast against it. All in-process: cross-process writers need a quiescent boundary for tag create and restore.
- **One version per write.** Each `create_document*` / `import_document` writes a version of `documents`, `chunks` and `document_items`. Bulk ingestion in a loop fills disk with versions. Use `import_documents([DocumentImport(...)])`, or `DocumentRepository.create([...])` / `DocumentItemRepository.create_all(...)`. `_flush_rebuild_batch` writes a batch's items with `create_all`, split whenever the whole documents gathered reach `_ITEMS_WRITE_BYTES` (16 MiB) of pictures, a threshold an add can exceed: lance holds about an add's pictures past the write, and one add per document leaves a manifest per document, each listing every fragment.
- **Opening a Store never writes to it.** Stored embedding settings change only via `init`/`rebuild`, and the version is a forward-only marker written by `init` and `migrate`. `init` and every `migrate` also write `schema_version`, the newest upgrade step (`latest_schema_version()`), and `recorded_settings` keeps it. A newer stored `schema_version` (`newer_schema`) raises `UpgradeRequiredError` on a writable open and warns on a read-only one. `migrate` and the `tag restore` path open with `skip_migration_check`, so `migrate()` and `restore_tag()` check for themselves, `restore_tag` against the tag's stamp. On a newer stamp no open creates a missing table. `Store.stored_settings` and `Store.stored_embedding` are read at open and never follow a later write. Refresh them together through `_remember_settings`.
- **Embedding drift on open.** A `vector_dim` mismatch always raises `ConfigMismatchError`. Provider, name or prefix drift at the same dimension warns read-only and raises writable (`EmbedderIdentity` and `embedder_drift` in engine.py: a missing prefix key reads as empty, and an empty prefix is not recorded). It is either a config naming another model or an interrupted rebuild, so the advice (`EMBEDDER_DRIFT_ADVICE`, shared with doctor) is `rebuild --embed-only` or rerunning the rebuild, and `rebuild --set-embedder` only for an embedder that produces the same vectors: it adopts a `query_prefix` change and refuses a `document_prefix` one. Maintenance commands, `list`, `get`, `visualize` and CLI deletion pass `skip_validation=True`. The MCP server does not: it opens one validating read-only client in its lifespan, so a `vector_dim` mismatch fails startup.
- **lance 8 `merge_insert`** rejects partial-schema sources upfront when an insert branch needs non-nullable columns the source lacks. Declare update-only merges (drop `when_not_matched_insert_all`) when matches are guaranteed.
- **An indexed `merge_insert` holds its source in lance's query memory pool**, 150 MB unless `LANCE_MEM_POOL_SIZE` is set, and cannot spill. lance takes that plan when every join column has a BTree, the target rows are under it, and there is no `when_not_matched_by_source_delete`. Fresh tables have no rows under their BTrees until vacuum or `rebuild_indexes`, so a small-table test passes either way. The `documents` merges pass `use_index(False)`. A test of a blob-row merge runs `rebuild_indexes()` first and pins the pool through `run_with_lance_memory_pool` (tests/conftest.py): lance reads the variable once per process.
- **Index sets** live in `index_specs()` / `ensure_indexes()` (schema.py), and every table-creating path routes through them. `ensure_indexes` matches on (column, index type) and never drops an index it did not declare. It skips FTS on an empty table (an index born over zero rows never catches up on `add`) and rebuilds an FTS index covering zero rows. The chunk write paths call it after writing. So a test needing the zero-coverage state writes via `chunks_table.add` directly, and the first chunk write into a fresh table writes one extra version. Rows written past an index stay outside it until vacuum. `rebuild_indexes()` rebuilds every declared index from scratch under its existing name, and every rebuild mode but TITLE_ONLY and SET_EMBEDDER ends with it (`Store.rebuild_indexes`), so a rebuild writes an index-only version of every table. The vector index is opt-in (`create_vector_index`, `create-index`) and not in `index_specs`. `create_vector_index` holds `_write_lock` for the build: a failed write's rollback erases an index committed during it, and vacuum's compaction preempts a running build. The rebuild calls `_ensure_vector_index` directly, under its own `_rebuild_lock`. EMBED_ONLY, and FULL, RECHUNK and DESCRIPTIONS when the vector dimension changes, recreate the chunks table, so `_rebuild_locked` notes whether one existed and retrains it with `_ensure_vector_index` afterwards. REINDEX leaves it alone. FULL, RECHUNK and DESCRIPTIONS at the same dimension never drop the table, so an interrupted one keeps the index and the run that finishes it retrains it. The 0.91.0 migration rebuilds a table's indexes when its FTS index reports `IndexConfig.index_version` 1 (lancedb's `list_indices()`; lance's `ds.list_indices()` `version` is the dataset version, not the format). pylance still writes v1 on request (`create_scalar_index(..., format_version=1)`), which is how its test builds one. A rebuild that recreates the chunks table first writes `chunks_rebuild_incomplete` (`IncompleteRebuildRecord`: `mode`, `vector_index`), outside the tagged tables, and drops it after the index rebuilds. `_ensure_vector_index` raises a failed build, so the record stays. A resumed run ORs `vector_index` with the live table. FULL, RECHUNK and DESCRIPTIONS write their own mode over an existing record before `_resolve_rebuild_recovery` discards embed-only staging. EMBED_ONLY refuses another mode's record. TITLE_ONLY keeps staging. `Store._initialize` warns when the table exists.
- **A zero-coverage or missing FTS index** returns every match unsorted, in insertion order, so `limit` slices arbitrarily and hybrid fuses that order as a ranking. `ensure_indexes`' rebuild is load-bearing. Single-term probes on small tables look healthy.
- **`create_index(replace=True)` rebuilds an identical index** (new uuid, new version, old files kept until vacuum), and matches by index *name* (`{column}_idx` by default), so a custom-named index on the same column gets a sibling. On lancedb 0.38+ an unnamed replace builds a suffixed sibling beside a different-typed index. Check `list_indices()` first and pass `name=`.
- **A vacuum after writes commits compaction and index versions**, so an `auto_vacuum` pass touches `documents`. A test measuring version deltas sets `storage.auto_vacuum = False`.
- **The lancedb `Session`** is process-wide and holds the index and metadata caches, so on object storage the first vector query's index load is paid once per process. `read_consistency_interval_seconds` defaults to 30: a connection sees its own writes at once, another process's after the interval.
- **Vacuum.** `Store.vacuum()` retains `storage.vacuum_retention_seconds` (default 24h). Every table goes through lance directly via `_run_lance_maintenance` (`compact_files`, `optimize_indices`, `cleanup_old_versions`, wrapped in `_wait_protected`). Tables with `large_binary` payloads (`documents`, `document_items`, matched by `has_payload_columns`) compact with `target_rows_per_fragment=…`, which `AsyncTable.optimize` does not expose (lancedb#2325). To purge recent fragments, call the path with zero retention.
- **`compaction_target_bytes`** sizes fragments compaction *writes*. It cannot shrink a larger one, which lance rewrites whole once before splitting it. Peak memory is max(target, largest fragment), and large ingest batches make large fragments.
- **Tags** are per-table lance refs: a haiku.rag tag is the same name on all five tables from one `current_table_versions()` snapshot. Names are alphanumeric/`.`/`-`/`_` only. `AsyncTable.tags` mints a fresh `AsyncTags` per access, so fault-injection tests patch `AsyncTags` methods at class level with call counting.
- **Tags and vacuum.** Vacuum prunes with `cleanup_old_versions(error_if_tagged_old_versions=False)`, which keeps every tagged version and removes every other version past the retention with its files. That needs lance-format/lance#8708 (pylance 11+). Never call `AsyncTable.optimize`: its cleanup defaults to 7 days, hard-errors on a tagged version in the window, and lancedb does not expose `error_if_tagged_old_versions`. Vacuum suppresses `OSError` only. `test_vacuum_reclaims_versions_written_after_a_tag` asserts no unreferenced files remain, so a lance bump that regresses this fails it.
- **lancedb and pylance move in lockstep.** lancedb bundles lance as a Rust crate, so nothing ties `pylance==X` to a lancedb version. Pair them by release date, then verify by round trip: lancedb writes, pylance compacts, lancedb reads and writes again. Before a bump, check the PyPI wheels per platform and their upload times. After a bump, run `pytest tests/store/test_vacuum_bounded.py -m integration -n0`: CI never runs the vacuum memory tests. FTS BM25 scores change between lancedb versions, so retrieval benchmarks are pin-conditional.
- **Blobs.** No document read loads the docling blobs by default: `get_by_id`/`get_by_uri` project `id`+`content`, and `list_all` needs `include_content=True` for text. Load blobs with `get_docling_data`/`get_pages_data`, or `get_by_id(..., include_blobs=True)`, which any path writing the row back through `DocumentRepository.update` needs, or it persists `None` over the blobs.
- **`Document.get_docling_document()`** decompresses only the structure blob. `set_docling(...)` after a structure-only load writes `docling_pages=None`. Update `docling_document` directly via `compress_docling_split`.
- **`document_items` across documents** goes through the grouped accessors only (`resolve_refs_grouped`, `get_items_in_ranges`, `get_pictures_grouped`, `get_caption_picture_refs_grouped`, `get_all_picture_data_grouped`). Those reading by `self_ref` build a per-document predicate, `(document_id = 'a' AND col IN (…)) OR (…)`: `self_ref` values collide across documents, and `self_ref IN (union)` over-fetches and can hand one document another's picture. Tests assert the query counts.
- **`get_pictures_grouped`** defaults `with_text=False`. Only the enrichment path opts in.
- **`expand_with_items`** does not fetch: it takes positions and window items, and `expand_context` fetches once for every document. Assemble results in `document_groups` order, since the stable score sort uses arrival order as the tiebreak.
- **FULL, RECHUNK and DESCRIPTIONS write one `write_transaction` per batch of 50 documents** (`_flush_rebuild_batch`) and replace only that batch's chunks, so a failed batch rolls back and later documents keep their chunks. They yield a document's id after its batch is written, never inside the transaction: the lock is not reentrant, so `rebuild_indexes`, `_ensure_vector_index` and FULL's `create_document_from_source` run outside it. A rollback does not survive SIGKILL. A vector dimension change still drops the chunks table before the loop.
- **The settings record the embedder whose vectors the chunks table holds.** A rebuild that keeps the table saves them after its loop, so an interrupted run keeps the old embedder recorded and a writable open refuses the mixed vectors. `_recreate_chunks_table` saves them right after recreating the table (dimension change, embed-only phase 2), since no old vector is left. TITLE_ONLY never saves them. A stop between recreating and saving leaves the stored `vector_dim` behind the table's, so `_rebuild_locked` takes `store.ChunkRecord` from the table's width. `rebuild_database` checks writability first.
- **Embed-only rebuild recreates the chunks table**, so patching `store.chunks_table.add` on the instance is discarded. Patch `lancedb.AsyncTable.add` at class level and filter on `self.name`. Its phase 2 writes via `chunks_table.add`. A FULL rebuild refreshes source-backed documents in place, keeping the id (re-fetch with `force=True`, falling back to stored content). An attachment (`parent_uri` set) is never fetched itself: its URI resolves to the parent's file. Parents go first, `force` re-ingests their attachments from the PDF's bytes, and `written_ids` tells the rebuild which documents a refresh wrote, so the rest fall back.
- **`client.update_document` re-chunks and re-embeds**, so it needs `@pytest.mark.vcr()`. To rewrite stored content without the embedder, use `DocumentRepository.update`.
- **Context expansion and `visualize_chunk` without the embedder:** `import_document(docling_doc, [Chunk(..., embedding=[0.1] * vector_dim)])`. Build the `DoclingDocument` with `add_page`/`add_text` and `ProvenanceItem` bboxes for page-image tests.
- **Throwaway lancedb scripts:** `uv run python` from the repo root. The sync `LanceTable` has no `.query()`. `.to_lance()` works.

### Docling and conversion

- **docling and docling-serve are pinned as a pair**: `docling==2.142.1` + `docling-core==2.104.0`, and `quay.io/docling-project/docling-serve:v1.38.2` in `tests/docker/`, `examples/docker/` and both `app/` compose files. The coupling is `tests/chunkers/test_chunker.py::test_local_and_serve_chunkers_produce_same_output` (slow, vcr), which the `test-slow` CI job enforces. Take the client version from the serve release's bundled library set and move both together. Re-record the serve cassettes when the image moves: that test plus the slow vcr `TestDoclingServeConverterIntegration` tests. Never use `:latest` in a compose file: Docker never re-pulls it.
- **One OpenCV build.** `rapidocr` requires `opencv-python`, the GUI build, which writes the same `cv2/` files as our `opencv-python-headless` and needs X11 libraries. The root `pyproject.toml` excludes it (`exclude-dependencies`), and `tests/test_dependencies.py` checks the lock. A pip install still gets both. Uninstalling either build deletes the shared files: `uv sync --reinstall-package opencv-python-headless` restores `cv2`.
- **`pdf_backend` defaults to `threaded_docling_parse`**. `docling_parse` is a deprecated alias of the same threaded implementation, not a serialized parser. docling-serve v1.38.2 names the threaded backend `docling_parse` in its API; the serve converter translates the configuration value. Keep conversion timeouts and hold the backend constant when comparing versions.
- **`processing.conversion_timeout`** (600s) bounds one conversion. Every part is deliberate: the deadline starts when the conversion *holds* `_CONVERTER_LOCK`; the lock is owned by the conversion thread, never the coroutine; the deadline runs in a task the caller `asyncio.shield`s; conversions run on daemon threads through `contextvars.copy_context().run`, not `asyncio.to_thread`; a queued conversion polls `acquire(timeout=0.5)` and re-checks `_WEDGED` before and after acquiring. Do not simplify any of it.
- **`TestConversionTimeout`** needs a fresh `_CONVERTER_LOCK` per test (`monkeypatch.setattr(docling_local, "_CONVERTER_LOCK", threading.Lock())`), never a forced `release()`. Stall stubs stay bounded by a `threading.Event`.
- **`processing.split_pages`** does not reproduce single-pass conversion: a paragraph or caption crossing a slice boundary regroups or reorders, while the word multiset stays identical. Only no boundary guarantees equality, which is what the `concatenate` test uses.
- **Labels are platform-dependent.** A label survives only past docling's hard-coded 0.5 layout confidence, so a borderline element flips between machines, and docling-serve substitutes a synthetic text item. A local/serve parity test compares items, item text, page numbers, chunk count, chunk text and word multisets, never labels or `self_ref`.
- **PDF control characters** (a BEL for a symbol-font list marker, tabs in headings, `/tildelow` glyph names) reach chunk text. Deliberately not fixed (issue #613). If it needs fixing: `Cc` covers control characters, a tab is a normalization question, and a glyph name is indistinguishable from text. Don't strip `Cf` (soft hyphens). The seams are the chunkers' `content=` sites or a post-conversion pass, not `prepare_text_content`.
- **Inline groups.** `converters/base.py::flatten_inline_groups` normalizes `InlineGroup`s at the three points a `DoclingDocument` is built (both `docling_local` sync paths, `docling_serve._parse_zip_to_docling`) and nowhere else. `import_document(s)` store the caller's document as given, since their chunks' `doc_item_refs` index into it, and `extract_item_text` recovers an empty owner's text there. `rebuild --rechunk` re-chunks the stored blob unnormalized. Only html, md, msword, opendocument, boxnote, webvtt and jats emit inline groups. The owner is the item with empty `text`, never merely the parent. A group with non-text runs, several provenance records or an inbound ref is left alone. `TestInlineGroups` pins the contract. Upstream: docling-core #768, #769, #770.
- **Mutating a `DoclingDocument` is O(document) per call**: batch `delete_items`. `DocSerializer` caches excluded refs by `self_ref`, so serialize everything before mutating. A ref into a cascade-deleted descendant is renumbered wrongly, and `GraphCell.item_ref` is renumbered by nothing. Never give a merged item several `ProvenanceItem`s: the doclang serializer emits the text once per record.
- **`MarkdownParams` escapes by default** (`escape_underscores`, `escape_html`). A serializer whose output becomes stored text sets both False (`flatten_inline_groups` and `extract_items` do, and `extract_items` sets `image_placeholder=""`).
- **HTML images** load through `docling/backend/utils/image_resource_loader.py`. Monkeypatch `ImageResourceLoader.load_image_data` (`(self, src_loc, base_path)`), not the backend method. Fetching is gated by `enable_remote_fetch`, which our options set from `fetch_remote_images`.
- **HTML furniture.** docling's HTML backend files everything before the first heading as `ContentLayer.FURNITURE`, which every reader of the body layer drops silently. `processing.conversion_options.infer_furniture` (default False) turns it off on `docling-local`. docling-serve has no such option.
- **rapidocr and google-genai** ship invalid escapes in string literals that beartype's import hook compiles. `pyproject.toml` carries `"ignore:.*invalid escape sequence:SyntaxWarning"`: Python 3.14 starts the message with the escape.
- **`providers.docling_serve.timeout`** bounds each HTTP call, not the conversion: the status poll has no deadline.
- **One pdfium lock.** libpdfium is not thread-safe. docling's PDF backends lock their calls with `docling.utils.locks.pypdfium2_lock`, and `PDFIUM_LOCK` (`pdf_split.py`) is that lock when docling is installed. A second lock around a pdfium call races the conversion thread under `worker_count > 1`: `Failed to import pages` from slicing, `Failed to parse file` from docling, on arbitrary slices. `test_pdf_concurrency.py` pins it.

### Ingester

- **Two databases.** Ingester state is LanceDB (documents) and the queue (jobs, sync_state). `ingester/reconcile.py` reconciles them at every `serve`/`run-batch` start from one `document_meta` listing: restore sync_state rows for owned documents, clear revisions through `SyncStateRepo.invalidate` for URIs the index lost, attribute unowned ingested documents through `HaikuRAG.set_document_source`. The evidence of a successful write is `last_ingested_at` (`list_ingested_uris`), never a revision: the permanent-failure path writes a revision with `ingested=False`. `upsert` cannot clear a revision (`_upsert_stmt` coalesces it), hence `invalidate`. A URI two sources ingested stays unattributed. Each repair is one transaction.
- **Queue path** defaults to `get_default_data_dir() / "ingester.db"`, not `storage.data_dir`. `ingester.queue.dburi` points it at Postgres, where `IngesterApp` requires a non-blank `namespace`: every repository statement carries it, and `Job` never does.
- **Ingester test doubles** use `AsyncMock(spec=HaikuRAG)`. They stub `_ingest_observed`, not `create_document_from_source`, and must give `list_documents` the documents they stored: an empty MagicMock reads as "the index lost everything" and invalidates every revision. Function-style doubles use explicit signatures, so a new kwarg on the real call raises `TypeError`, which `_classify` turns into a transient failure. Update every double when the signature changes.
- **`_ingest_observed(observed_revision=)`** is the ingester's entry point, passing `job.revision`. It is not on the public `create_document_from_source`. With it the revision short-circuit skips its `head()`. It decides only whether to skip: a mismatch falls through to the fetch, and `force` never reaches it. A source changing after discovery waits for the next sweep.
- **`create_document_from_source` metadata merges** differ by path: full ingest does `{**user_metadata, **source_metadata}`, the revision short-circuit's `_refresh_doc_metadata` does `{**doc.metadata, **user_metadata}` before `source_metadata`. `_caller_metadata` and `_provider_metadata` strip every `_INGESTION_OWNED_KEYS` entry (`content_type`, `md5`, `source_revision`, `source_id`, `parent_uri`) from caller and provider metadata. `update_document` restores all five from the stored record. A create or import over a stored URI and a source re-ingest restore only `source_id` and `parent_uri` (`_keep_ingestion_keys`), since the stored `md5` and `source_revision` describe the source's bytes. `source_id` is the ingesting source's id, or the stored one when the caller names none, so an ad-hoc re-ingest cannot detach a document. A different source id takes ownership with a WARNING. PDF attachment children get `user_metadata={}` and no `source_id`. `SingleDatabaseSession.delete_document` refuses an attachment whose parent is stored and outside the subtree (`AttachmentDeletionError`). `_reconcile_pdf_attachments` passes `allow_attachment=True`.
- **Stalled conversions.** `PermanentError` carries `conversion_stalled` (tombstone this document) and `fatal_to_process` (exit) separately, since an HTML/Markdown stall strands no converter. The worker records the job dead before exiting, and exits from a `finally`: `reap_stale` refunds the attempt of an owner that vanished, so exiting first requeues the poison document forever.
- **Queue schema 3** adds `jobs.conversion_stalled` and `uq_jobs_blocking_op`, a partial unique index over `(source_id, uri, op, coalesce(revision, ''))` for live and stalled jobs. Enforced by the index because no read is atomic against a concurrent `mark_dead`. `coalesce` because unique indexes treat NULLs as distinct. `prune_terminal` skips these rows. A new revision, a DELETE, `retry` and `prune_dead` free the slot. The migration emits the index from the `Index` object `create_all` uses.
- **Workers claim only the sources they serve** (`claim_next(source_ids=)`, the pool's served set). At startup `serve`, `run-batch` and manifest replay delete queued and claimed jobs of sources absent from `ingester.databases` (`cancel_unconfigured`), never `--dry-run`. That relies on one process per namespace. The batch commands refuse an empty `ingester.databases` (`NothingToIngestError`). `serve` starts on one and deletes every live job of its namespace.
- **Reload is remove-then-add per database.** `IngesterApp.add_database` prepares (opens the client, builds adapters, writes no queue row) then applies (reconciles, then serves). `remove_database` unserves, stops the pollers, waits for in-flight jobs, deletes live jobs and `sync_state`, then closes: each step idempotent, and a database stays `removing` until the close lands. Claims, `unserve` and API retries hold `WorkerPool.claim_lock`, and a claim reads the served set inside it, so no claim names an unserved source once `unserve` returns. A sweep holds its poller's lock, so `stop` waits for a refresh request's sweep. Reload never adds or removes one source of a running database: reconcile reads in two steps and is unsafe beside workers writing that database. `_reload` refuses any change outside the two database maps (`changed_sections`), so every open client shares one configuration. Shutdown cancels reload tasks before awaiting the API, or a reload waiting on a drain holds the pool stop.
- **`uq_jobs_live(source_id, uri)`** allows one live job per (source, uri), so a DELETE cannot run after a sibling UPSERT. Rapid (deleted, added) events squash to delete-only, which `FSPoller._handle_watch_change` mitigates with a `path.exists()` recheck. Keep it.
- **Queue migrations are versioned** (`ingester/queue/migrations.py`, `SCHEMA_VERSION`). `create_all` never adds a column to an existing table: a new column needs the model change, an explicit `ALTER TABLE` (+ backfill) under `current < N`, and a `SCHEMA_VERSION` bump. On SQLite `apply_migrations` issues `BEGIN` itself: sqlite3 starts a transaction before DML only, so DDL inside an `engine.begin()` block commits as it runs without one. `open_queue` migrates a SQLite file and only verifies a server queue (`verify_schema`, a `QueueError` naming `queue init` or `queue migrate`, nothing written); the `queue` commands run `apply_migrations`, under `pg_advisory_xact_lock` on Postgres, and refuse a schema newer than the release before any DDL.
- **Queue engine.** SQLAlchemy Core async. SQLite is `pool_size=5, max_overflow=5` with PRAGMA listeners. Postgres (`postgresql+asyncpg://`) gets `pool_pre_ping=True` and claims with `FOR UPDATE SKIP LOCKED`. `claim_next` must stay one `UPDATE … WHERE id=(SELECT … LIMIT 1) RETURNING`. Build the SQLite URL with `URL.create("sqlite+aiosqlite", database=str(path))`, never an f-string: `?` and `#` in a path become query and fragment.
- **Postgres queue tests** build the engine inside the test with `NullPool` (asyncpg binds a connection to its loop) and isolate xdist workers with a per-test schema, since `--dist load` ignores `xdist_group`.
- **Coverage** uses `concurrency = ["greenlet", "thread"]` for the aiosqlite worker thread. Postgres-only construction branches are covered by build-without-connect tests (`make_engine(QueueConfig(dburi=...))`, `_insert(jobs, "postgresql")`).
- **Maintenance** (`ingester.maintenance`, off by default): `DatabaseMaintenance` per database, built in `_apply`, run as a task only under `serve` (`_maintaining`), once per written database in `run-batch` and manifest replay, after the drain and after `_maintain_written` exits every ingester lifespan with the clients still open. A pass holds claims with `WorkerPool.pause`, a set apart from `_served`, so `resume` in a `finally` never re-serves a source that a reload unserved. A pass does not wait for jobs in flight: their writes wait on `_write_lock`. The `quiet_s` trigger waits on `WorkerPool.wait_idle` until the `max_wait_s` deadline, then re-evaluates, since a finishing job's write restarts the quiet period. `close_client` cancels the task before the ingester lifespans exit. Dirty state is `SingleDatabaseSession.unmaintained_since`, set by `record_write()` at every client write. While enabled, `_open` passes a config copy with `storage.auto_vacuum` off. The pass never retrains the vector index: pylance's `optimize_indices` splits IVF partitions as the table grows, and a grown index measured equal recall to a fresh one.
- **Reaping is lease-based.** Workers renew `last_heartbeat_at` every `heartbeat_interval_s`. `reap_stale` resets a claim when `COALESCE(last_heartbeat_at, claimed_at)` is older than `lease_ttl_s`, and never cancels a live coroutine. `WorkerConfig` enforces `heartbeat_interval_s <= lease_ttl_s / 3`. The reaper passes the pool's in-flight job ids as `reap_stale(keep=)` under `claim_lock`, which `_claim` holds while it claims and tracks, so a process never reaps its own live claims, whatever its clock or a suspension does to their leases. `claim_timeout_s` is gone.
- **Worker ids** are `{pid}-{uuid}-{n}`. Boot-reap is scoped to `lease_ttl_s`, so a peer's live claims survive. `_untrack_inflight` drops only the pool's own entry for a job, or it stops renewing a sibling's re-claim.
- **`POST /sources/{id}/refresh`** runs `_sweep_once`, which answers `refreshed: false` for an open breaker, a failed discovery or a stopped poller. `serve`'s sweeps and refresh run whatever is queued: `uq_jobs_live` deduplicates a queued file, and an UNCHANGED event writes no revision, so a sweep never overwrites one a worker recorded meanwhile. Only `sweep_all` (`run-batch`) passes `skip_if_pending=True`, since a batch sweeps once and an event deduplicated against a live job would never be seen again. The sweep log counts enqueued jobs, not discovered events. Default FS `poll_interval_s` is 300.
- **obstore exceptions** subclass only `obstore.exceptions.BaseError`. `workers/pipeline._classify` maps `PermissionDeniedError`, `UnauthenticatedError`, `UnknownConfigurationKeyError` and `InvalidPathError` to `PermanentError`. A missing object is a builtin `FileNotFoundError`, already permanent.
- **An exception out of an FS periodic sweep dies silently**: `_sweep_loop` is a task `run()` never awaits, `live_pollers` counts the outer task, and shutdown's `gather(..., return_exceptions=True)` swallows it. `/health` stays green.
- **Poller discovery failures** are handled in three places with the same `record_failure` + `_record_error` + `logger.exception` body: `pollers/base.py` `_sweep_once` and `_dry_run_once`, and `pollers/fs.py` `_watch_loop`. Change one, change all three.
- **Plugins** use `importlib.metadata` entry points: `haiku.rag.metadata_providers` and `haiku.rag.sources`. Only referenced ones load.
- **Docker images.** Both Dockerfiles install `--extra ingester`, which does not pull `[docling]`. The full image gets docling from the `haiku.rag` package itself. The published slim image cannot run `converter: docling-local`: `DoclingLocalConverter` imports docling at construction and raises `ImportError` naming the extra, so `haiku-ingester` and `add-src` exit at startup. `[ingester]` pulls `[s3]`.
- **Dashboard** (`ingester/api/static/index.html`) is outside the biome hook's scope. Run `biome lint <file>` by hand, and `node --check` on the extracted script.

### Capabilities and sandbox

- **`RAGState` is flat** and is dumped and re-validated at every carry point: `client/agents.py`, `chat/app.py`, `evaluations/capability_runner.py`, and `app/backend/main.py`, where the AG-UI client returns the snapshot next turn. Nesting or renaming a field is a breaking wire change. Additions are safe. `begin_invocation()` drops the previous question's `citations`, `searches` and `executions`.
- **Agent specs construct capabilities through `from_spec`**, never `cls()`: `id` comes from `create_capability()`, `rag.py` filters tools by `tool.capability_id != self.id`, and pydantic-ai's duplicate-id rejection enforces one compaction and one policy capability per run. pydantic-ai passes raw parsed YAML (`db_path` a `str`, `config` a `dict`). Third-party capabilities are never auto-discovered: the caller passes `custom_capability_types=[...]`. A spec needs a model, in the spec or as a kwarg. A zero-argument `from_spec` override keeps per-run caches out of the schema.
- **`_cite` repairs a near-miss chunk id** to the nearest id the run retrieved, above `CHUNK_ID_MATCH_CUTOFF` (0.75), before the database fallback, and before `resolve_citations`, so `cited_map` order holds.
- **`qa.max_images`** bounds the number of images in one request. `RAGCapability.prepare_model_request` applies it through `limit_retrieved_images` (`_tools.py`). That hook runs after every `wrap_model_request` and `before_model_request`, so it sees the capsule pictures whatever the registration order, and it changes the request only. Every image counts, tool returns included, and only label-plus-picture pairs carrying `RETRIEVED_IMAGE_TAG` are dropped: search pictures outside the last message first, then capsule pictures (`CITED_EVIDENCE` in the label), then the last message's. `prepare_model_request` hooks run in registration order, so images added by a later capability's are not counted.
- **Repeated cite reminders in `capabilities/instructions/*.md` are load-bearing.** Treat any prompt-length reduction there as a behaviour change needing its own eval.
- **Per-search `sources`** on the capability's `search` tool selects within
  `RAGCapability.collections`, never the client: a search cannot reach a database
  outside `state.sources`, whatever a lent client covers. `[]` searches nothing. A name outside the
  run is a `ToolFailed` naming the run's collections, raised before anything opens.
  `prepare_tools` strips the parameter from the schema on a one-collection run, so
  single-database runs keep the `(query, limit)` contract. `evidence_signature`
  renders without the collection label, so evidence dedups across differently
  narrowed siblings. The qualified id keeps copies apart.
- **Repeated search queries accumulate**: `state.searches[query]` merges through `merge_results` keyed on `chunk_id`. Hand-built `SearchResult`s without a `chunk_id` collapse to the first. `search_corpus` returns `"No results found."`, never an empty string.
- **Sibling dedup.** Searches in one model response collapse evidence a sibling already showed to `Also matched, shown above: [id]`, and attach each picture once per response on `(source, document_id, self_ref)`. Equivalence is `evidence_signature` (`_tools.py`). `state.searches` keeps results whole. Search state commits only after formatting and image construction succeed. Dedup never crosses run_steps. In-code `search()` bypasses pricing, dedup and spans.
- **User-attached images** need the instructions' "Questions with attached images" section, or vision models refuse without searching.
- **`SearchResult.document_meta` / `Citation.document_meta`** carry the document's metadata for UIs, excluded from `format_for_agent`. `chunk_meta` is the verbatim `Chunk.metadata`, typed keys included: on an expanded result it is the anchor chunk's, and `format_for_agent` shows it only with `include_chunk_meta` (the MCP search tools set it).
- **Sandbox** (`sandbox/sandbox.py`): pydantic-monty in a subprocess worker from an `AsyncMonty` pool, with read-only `CallbackFile`s under `/documents/{id}/`. `metadata.json` is prebuilt, `content.txt` lazy per document, `items.jsonl`/`chunks.jsonl`/`toc.json` a lazy bulk cache. One session serves every `execute()` call of a capability run. `_run_on_loop` bridges sync reads to async queries. A host error keeps its message for every caller.
- **`sandbox.code_timeout` is a deadline**: `_check_deadline` runs before every host call (`_timed` wraps the five readers, in-code `search()` and `list_documents()` call it first, `_run_on_loop` keeps its own), because cached reads never touch the bridge. Monty's feed limit (`max_feed_duration_secs = code_timeout`, reset per `feed_run`) stops compute and never counts time suspended on the host. It raises the same `TimeoutError` type as the read deadline and leaves the heap undefined, so `_hit_time_limit` tells them apart by the read deadline's message and discards the session on the feed limit only. The pool's grace backstop kills a worker that overruns the feed limit, so no `request_timeout` is set. A refused read costs the worker no compute, so a program catching the refusal outlives the feed limit: `_stop_after_deadline` kills the worker `_DEADLINE_GRACE_SECONDS` after the deadline (the crash path then discards the session), and a call that saw a refusal returns `success=False` with the time-limit message even when the program ended normally. The kill bounds the worker's lifetime, not a host call already running: that call finishes before `execute()` returns.
- **Sandbox OS policy**: `time.sleep`/`asyncio.sleep` default to real waits of up to 10s that no duration limit counts, so the checkout sets `os_policy={"sleep": "zero"}`. `feed_run(cwd="/documents")` makes relative paths resolve under the mount.
- **Monty caps host callbacks** per checkout (`max_suspensions`, 1000 by default), with no way to disable it. `_session_limits` sets `_MAX_HOST_CALLS = 10_000_000`, so the time budgets govern.
- **Monty runs VFS callbacks on a thread coverage does not trace**: a reader's lines need a direct-read test (`tests/sandbox/test_sandbox_toc.py`).
- **Compaction and policy hooks.** `EvidenceCompactionCapability` discovers the evidence capability through `RunContext.capabilities` and rewrites `request_context.messages` in `wrap_model_request`, never the stored history, which keeps question identities and epochs (message counts) meaningful. A question's identity is the index of its opening request, taken by `RAGCapability.open_question` at the question's first model request: pydantic-ai normalizes the history after `for_run`, and AG-UI passes the question inside `message_history`. `before_model_request` calls it, and so does the compactor's `wrap_model_request` through `open_questions`, since pydantic-ai runs `before_model_request` inside the wrap handler. `open_questions` skips capabilities outside `RunContext.active_capability_ids`: pydantic-ai never runs an unloaded deferred capability's `after_run`, so a question opened on it would stay in progress. A deferred capability takes the index of the first request after it loads. That is harmless because pydantic-ai refuses a second `RAGCapability` in one agent, by id or by tool name. `CitationPolicyCapability` reads the same records in `after_model_request` / `after_run`. `cite([])` records a declaration with no refs (`ungrounded`), distinct from declaring nothing (`missing`).
- **Context expansion** (`context.py`) is section-bounded: a section within `max_context_chars` returns whole, a larger one expands outward inside its edges, one under 20% of the budget grows across boundaries (picture and table matches excepted). Only overlapping ranges merge, and the result is hard-capped at `max_context_chars`. `visualize_chunk` expands before resolving bounding boxes. `HaikuRAG.search` does not expand: the capability, MCP and in-code search call `expand_context`.
- **Picture-only chunks** (one `#/pictures/*` ref, produced only under a multimodal embedder) are embedded from the image, and their `content` is the description or caption followed by the text docling found inside the picture (OCR'd words, footnotes), which can be empty. Search never reads `vision`, so a non-vision model still gets them, expanded to their section's text, and `Type:` then shows the section's label. Rerankers score the unexpanded chunk: multimodal ones image plus that text, text ones the text only, dropping a chunk with no text (`_scoreable`). The text is `scored_text` (`reranking/base.py`): document title and headings, where present, before the content. Not a defect (#649, #654).
- **`toc.json`'s `item_range`** indexes the position-ordered item list (the line numbering of `items.jsonl`), not database positions. `get_document_section` slices the same way.

### MCP

- **`FastMCP.lifespan()`** combines provider lifespans only and never runs the constructor's `lifespan=`. Tests use `_lifespan_manager()`. The server's lifespan resets the cached client in a `finally`, since it can be re-entered.
- **Claude Code and the Agent SDK drop a tool result's text when `structuredContent` is set**, forwarding the JSON and images only. Desktop and claude.ai forward both. So a tool whose text is a rendering sends one channel, which is why the search tools carry no structured content and their tests read the text.
- **Claude Desktop ignores server `instructions`**, and only Claude Code honours `allowed-tools`, so every tool description stands alone. `execute_code`'s docstring carries the whole sandbox contract. Claude Code moves an MCP call to the background after about 120s.
- **fastmcp 4 / MCP SDK 2**: protocol types are snake_case, and the camelCase bridge goes in fastmcp 5 (hence `<5.0.0`). `Client` defaults to sessionless, where `initialize_result` is `None`: read `client.instructions` / `client.server_info`. `ToolResult` imports from `fastmcp.tools`. Tag visibility is `server.disable(tags=...)`. Server-initiated sampling is gone. Pass `version=` or the server reports fastmcp's version.
- **An in-process `fastmcp.Client` test** needs `@pytest.mark.filterwarnings("ignore:Found propagated trace context:RuntimeWarning")`: an earlier CLI test in the worker installs logfire's propagator. Shard-dependent. Reproduce with `uv run pytest tests/interfaces/test_cli.py <the test> -n0`.

### Models and providers

- **`get_model` branches** for ollama, vllm, openai, openrouter, anthropic, google, groq, bedrock and mistral, applying `temperature`, `max_tokens`, `thinking` and `extra_body`. Any other provider is returned as `"provider:name"` and pydantic-ai applies none of them, so it logs a warning.
- **`get_model` sets `openai_chat_supports_multiple_system_messages: False`** on OpenAI-compatible endpoints, since strict vLLM chat templates reject more than one system message.
- **Thinking on self-hosted endpoints.** `VLLMProvider.model_profile()` infers the family by name prefix, and pydantic-ai strips unified `thinking` where the profile lacks `supports_thinking`. So `ollama`, `vllm` and `openai` with a `base_url` send `openai_reasoning_effort` directly (`reasoning_effort_settings`): `False` → `none`, `True` → `medium`, a level as written. The server's chat template decides what it means. Verify a kwarg by wire capture and generation, never a rendered prompt: vLLM's `/tokenize` skips the `reasoning_effort` translation. `provider: vllm` is pydantic-ai's chat provider on a `ModelConfig` and haiku.rag's multimodal client under `embeddings.model` / `reranking.model`.
- **pydantic-ai's request guard sits above VCR**, rejecting a replayed embedder call before VCR sees it. The autouse `allow_expected_model_requests` enables model requests for `vcr` and `integration` tests. Symptom if removed: many `RuntimeError: Model requests are not allowed` plus a hang at the end of the run.
- **`LogfireQueryClient`** is `logfire.query_client` in logfire 5.

### Testing and tooling

- **Windows.** Text mode follows the host locale (cp1252), so read and write repo files with `encoding="utf-8"`. Build an expected rendered path with `pathlib`, not a literal `/`. Write a file under test through `tmp_path`: Windows refuses to reopen an open `NamedTemporaryFile`. Creating a symlink needs Administrator or Developer Mode; without it the `requires_symlinks` tests (`tests/platform.py`) skip and the symlink branch of `sources/fs.py` goes uncovered. Guard a POSIX-only call with `sys.platform`, which ty narrows on, not `hasattr` or `getattr`. docling-parse's PDF decoder intermittently corrupts the heap on Windows and kills the xdist worker (`0xc0000374`), so a fast test that converts a PDF with `docling-local`, the `docling_local_models` fixture included, carries `converts_pdf_locally` (`tests/platform.py`).
- **`haiku.rag`'s logger does not propagate**, so `caplog` misses its records once any test in the worker has called `get_logger()`. Use `capture_logs(logger, level)` from `tests/conftest.py`.
- **HF Hub outages** stall unmarked-network tests at setup. `HF_HUB_OFFLINE=1 pytest ...` runs from the disk cache.
- **`uv run ty check <files>`** checks only those files. The pre-commit ty hook is broader: run `uv run pre-commit run --files <paths>`.
- **ty treats `dict` as invariant**: type read-only parameters that accept subtype dicts as `Mapping[...]`.
- **Mutating a pydantic model's `model_config`** needs `Model.model_rebuild(force=True)` to take effect.
- **Bulk rewrites:** Python one-liners, not `sed -i`. BSD sed lacks `\|` alternation and silently no-ops.
- **`git grep`** floods on cassettes: pass `-- ':!tests/cassettes'`.
- **`Citation` and `resolve_citations`** live in `haiku.rag.store.models.citation`.
- **Rich-renderer tests** render with `Console(record=True, width=200)` and assert on `export_text()`.
- **`textual_image`**: `widget.Image` for Textual apps, `renderable.Image` for Rich consoles. Both take PIL images.
- **Textual `auto` grid rows ignore margins**: use bottom padding. Assert layout with `parent.region.contains_region(child.region)`.
- **Textual `pilot.pause()`** does not reliably flush a handler awaiting I/O on a slow runner: await the handler directly.
- **A line that runs once per process** (module-level lazy init) is covered only on some shards. Assert it directly with the module global reset via `monkeypatch`.
- **Commenting out every child of a non-Optional config block** parses to `null`, which Pydantic rejects. Drop the key or keep one child.
- **`doctor`** renders details through Rich markup, so avoid literal `[...]`. Near-duplicate detection keeps one summed centroid per document from the single vector scan. `scan_vectors` holds the vector matrix only while it reduces it, and returns the centroids, so a caller never keeps the matrix.
- **`astral-sh/setup-uv`** has floating major tags only through v7. Pin exact tags (workflows use `@v10.0.1`).
- **`haiku.rag.sources`** imports without the `[s3]`/`[ingester]` extras because `obstore` is imported inside `S3Source` methods. Keep it there.
- **One-shot directory ingest** uses `sources.fs.walk_files`, which never follows directory symlinks and keeps a symlinked file only when its target resolves inside the root. No size cap on that path.
- **`is_local_uri` and `uri_to_path`** (`uri.py`) own "is this local" and "what path is this", Windows drive letters and file-URI hosts included. Don't re-derive either.
- **`tests/test_uri.py`** runs in its own CI matrix without the project installed (`uv run --no-project ... --noconftest -o addopts=`). Keep it stdlib-only.
- **`app.py` and `cli.py`** are covered through Typer's `CliRunner`. `cli.py` keeps one pragma, on the `__main__` guard.

## Planning and Commits

When planning work, break the plan into **self-consistent, commitable chunks**. Each chunk should:
- Be a logical unit that can stand on its own
- Leave the codebase in a working state
- Be small enough for meaningful review

## Common Patterns

**Test with VCR recording:**
```python
@pytest.mark.vcr()
async def test_my_feature(temp_db_path):
    async with HaikuRAG(temp_db_path, create=True) as client:
        doc = await client.create_document("Test content", uri="test://doc")
        results = await client.search("test")
        assert len(results) > 0
```

**Creating a test database fixture:**
```python
@pytest.fixture
async def client(temp_db_path):
    async with HaikuRAG(temp_db_path, create=True) as rag:
        yield rag
```

**Using the native RAG capability:**
```python
from pydantic_ai import Agent
from haiku.rag.capabilities.rag import create_capability

agent = Agent(
    "openai:gpt-4o",
    capabilities=[create_capability(db_path=db_path, config=config)],
)

result = await agent.run("What documents do we have?")
```

**Custom embedder:**
```python
from pydantic_ai.embeddings import Embedder
from haiku.rag.embeddings import EmbedderWrapper

# Wrap any pydantic-ai Embedder
embedder = EmbedderWrapper(Embedder("openai:text-embedding-3-small"), vector_dim=1536)
```

## Evaluations

Separate workspace package (`evaluations/`) for benchmarking. Its README is the operator guide. Runs use the full install, docling included, so sync from the repository root. Run its tests from inside `evaluations/` (`cd evaluations && uv run pytest ...`): from the repo root, its `tests/` collides with the core `tests/`.

```bash
uv sync --all-packages
uv run evaluations run <dataset>
uv run evaluations pair <treated>.jsonl <baseline>.jsonl
uv run evaluations download <dataset|all>    # Pre-built eval DBs from HuggingFace
uv run evaluations upload <dataset|all>      # Upload eval DBs
```

Datasets: `enterprise_rag`, `frames`, `hotpotqa`, `orb_text`, `orb_multimodal`, `orb_multimodal_nemotron`, `t2_finqa`, `t2_tatdqa`, `mtrag_clapnq`, `mtrag_clapnq_rewrite`, `mtrag_clapnq_live`, `mtrag_clapnq_live_uncompacted` (the four mtrag keys share one DB).

**Capability runs** go end to end through a native Pydantic AI agent (`evaluations/capability_runner.py`). The capability model is `config.qa.model` and the judge `config.evaluations.judge`, with no command-line override for either.

**Answer facts (`AnswerFactsJudge`, `evaluators/answer_facts.py`)**: every single-question QA run appends it beside `RefusalJudge`. A case whose metadata carries `answer_facts` (only `enterprise_rag`) gets `answer_completeness`, the fraction of facts the answer supports, from one judge call per fact with EnterpriseRAG-Bench's own validator prompt (verbatim, a "yes" on the reply's first line). That judge sees only the answer and the fact, and the `answer_equivalent` verdict never sees the facts: EnterpriseRAG-Bench judges the two independently. The summary prints mean completeness and mean `erb_score` (completeness if `answer_equivalent`, else 0, the paper's headline) over cases with both. QA Accuracy stays the `answer_equivalent` pass rate, and result rows record `answer_completeness`.

**Citation retrieval metric**: `CitationMAPEvaluator` scores the URIs the capability registers via `cite` against gold `expected_uris`. Score key: `cited_map`.

**The judge is pinned and frozen.** Every reference config under `evaluations/configs/` whose dataset is judged carries the same block (`temperature: 0.6`, `max_tokens: 16384`, `extra_body` with `top_p` 0.95 / `top_k` 20 / `min_p` 0 / `chat_template_kwargs.reasoning_effort: low`), guarded by `evaluations/tests/test_reference_configs.py`. Do not change it without an eval: greedy decoding lost verdicts to repetition spirals on Qwen, and a larger `max_tokens` did not help. `DEFAULT_JUDGE_MODEL` carries only what ollama honours (`temperature`, `max_tokens`, `top_p`). `t2_finqa` / `t2_tatdqa` set `qa_evaluator`, which replaces the evaluator list, so no judge is constructed for them.

**Gated judge (`evaluations.system_one`)**: optional, off by default, and our reference configs never set it. When set, `AnswerEquivalenceJudge` (`evaluators/answer_equivalence.py`, a `SystemOneJudge` from `evaluators/system_one.py`) asks the decision endpoint first and decides alone at p ≥ 0.8 or p < 0.2. The band between, endpoint errors and `ConversationInput` cases go to the `TranscriptLLMJudge` as its fallback. Unset `provider` means `/v1/systemone` (`SystemOneEndpoint`, the v1 request). A `provider` (`vllm`, `ollama`, `openai`) means `ChatDecisionEndpoint` (`evaluators/chat_decision.py`): the model through `get_model` with `thinking: false`, `temperature: 0`, `max_tokens: 1`, asked `ANSWER_EQUIVALENCE_QUESTION` in Tev1's `{state, question, options}` format with the criteria as the yes/no descriptions, both option orders averaged, p from the A/B logprobs. `thinking: false` becomes `reasoning_effort: "none"`, which vLLM 0.29 and Ollama both honour; without it Tev1 thinks and the letters leave the top logprobs. Ollama's own `/v1/systemone` rejects list criteria, so it is reached through `provider: ollama`. `check_system_one` sends the judge's own question and fails the run before any case when the endpoint does not answer it. The key comes only from `TYPESAFE_API_KEY`; a local server with a `base_url` needs none. The run prints the `judge_decided_by` counts (`system_one`, `fallback`, `fallback_on_error`, `fallback_conversation`), and metadata records `system_one_served_model` from the check. Live runs ignore the block and say so. Jev is not deterministic between runs: replaying 390 labelled cases twice on 2026-09-24 moved 8–11 across a band edge, so a gated run does not reproduce verdict for verdict. A gated and an ungated run of one system are not a null pair.

**Multi-turn (MTRAG)**: `mtrag_clapnq` runs gold-prefix QA (`ConversationInput` cases replay the reference prefix as message history) plus lastturn retrieval. `_rewrite` retrieves with the human rewrites. `_live` and `_live_uncompacted` set `spec.live` (one case per conversation, `--limit` counts conversations) and differ only in `spec.compaction`, which registers `EvidenceCompactionCapability`: the only eval coverage compaction has. Live runs carry `all_messages()` and one capability-state dict across turns, and record per-turn arrays (`turn_cited_uris`, `turn_n_search_calls`, `turn_n_sandbox_search_calls`, `turn_n_rejected_searches`, `turn_n_failed_tools`, `turn_n_executions`, `turn_n_requests`, `turn_citation_status`) counted from `new_messages()`. Gold-prefix and live pass rates are not comparable; compare compacted with uncompacted, paired by turn. `_live_summary`'s macro rate excludes conversations with zero judged turns.

**Run identity**: experiment metadata carries `git_sha` and `git_dirty` (untracked files ignored), `config_hash` (SHA-256 of the resolved `AppConfig`; `populate_db` disables `storage.auto_vacuum` on a copy), and the corpus fingerprint `db_path`, `db_documents`, `db_chunks`, `db_embedder_provider`, `db_embedder_model`, `db_embedder_dim`, `db_version`, `db_written_at` (newest table version time). The fingerprint is all None when `lancedb.databases` places the set, and only `db_path` is set for a missing path. `evaluations run` prints the revision and hash at start.

**Per-case result files**: a single-question QA run writes `<data dir>/evaluations/results/<name>.<trace id>.jsonl` (`evaluations/results.py`, `--results DIR`), one row per case: the `CaseOutcome` fields (`case_name`, `key`, `passed`, `cited`, `cited_map`, `aborted`), `answer_completeness`, trace id, answer, judge reason, `judge_decided_by` / `judge_probability` / `system_one_model` (set by the gated judge, else None), attributes, task duration. A pydantic-evals `CaseLifecycle` appends rows to `<name>.<run id>.partial.jsonl` as cases finish, and `write_results` removes it. The run id (`new_run_id`) keeps concurrent runs of one name apart and names the file when there is no trace (`notrace-<run id>`); a `--no-telemetry` run's all-zero trace id counts as none. The final file is opened exclusively. `key` is the case-metadata field named by `DatasetSpec.pair_key`, recorded as `pair_key`. Live runs write none. `evaluations pair TREATED BASELINE` (`evaluations/pairing.py`) joins two files on `key` and prints per-arm accuracy, floor, cite rate, mean `cited_map`, aborts and unjudged, exact McNemar on verdicts, the sign test on `cited_map`, and the smallest significant `|b - c|`. It warns when only one arm was judged through `system_one`, or when the arms were judged through different `system_one` models.

**Telemetry**: `evaluations run` configures Logfire itself and refuses to start when it resolved no token (`logfire.DEFAULT_LOGFIRE_INSTANCE.config.token`, covering `LOGFIRE_TOKEN` and a credentials file; Logfire does not read the token from its file configuration), unless `--no-telemetry`.

**Per-case diagnostics**: runs record `cited_chunk_ids`, `searched_uris`, `n_searches`, `n_search_calls`, `n_rejected_searches`, `n_failed_tools`, `n_executions`, `n_sandbox_search_calls` and `n_requests` as attributes, counted from the message history (`for_run` gives the run a `replace()` copy whose counters the host never sees):
- `n_rejected_searches` is search-budget exhaustion. `n_failed_tools` is any failed call of the capability's tools. A failed `execute_code` is an exhausted execution budget or an error in model-written Python
- `n_searches` counts distinct search *keys*, with every in-code `search()` filed under one `_sandbox` key. `n_sandbox_search_calls` counts those calls (`CodeExecutionEntry.search_calls`), each a `sandbox.search` span
- `n_requests` is the run's request count
- `citation_status` derives `grounded` / `ungrounded` / `missing` via `ledger.citation_status` (an explicit `cite([])` is `ungrounded`)

`evaluations run --filter/-f CLAUSE` restricts every benchmark search and is recorded as `document_filter`. Never compare a filtered run with an unfiltered one.

**Never steer on raw cite rate**: it is confounded by task success. Measure cite rate among *correct* answers, split by whether the case used `execute_code`.

**Eval prompts**: datasets carry no custom system prompts. `DatasetSpec` has no `system_prompt` field. Capability targets use the packaged instructions plus `config.prompts.domain_preamble`.

**Eval-side rules**: don't assert phrases in packaged instructions, test behaviour. `build_experiment_metadata` is additive, except that the `qa_*` model mirrors were removed: `capability_*` records the model that ran, always present on a QA run and absent on a retrieval run. `qa_max_searches`, `qa_max_executions`, `sandbox_code_timeout` and `sandbox_max_output_chars` stay.

**Reasoning knobs on vLLM**: a vLLM server started with `--reasoning-parser` consumes `chat_template_kwargs.enable_thinking` itself. Muse-Glimmer QA blocks set `chat_template_kwargs.reasoning_strength: high` (the mtrag reference config does). A template defaulting it to low silently cuts search calls, with no error. Verify a kwarg by rendering (`/tokenize` with `return_token_strs`) or by measured behaviour, never by HTTP acceptance.

**`HfApi.upload_large_folder`** has no `path_in_repo`: `evaluations upload` stages the database under a tempdir with `os.link` hardlinks. Keep that rather than `upload_folder`, which is not resumable.

## haiku.rag.app (Web application)

The `app/` directory is a browser application with four pages: chat, ingestion, inspector and doctor. User documentation is `docs/apps.md`.

```bash
cd app
docker compose -f docker-compose.dev.yml up -d --build
# Frontend: http://localhost:3000   Backend: http://localhost:8001   Ingester: http://localhost:8765
(cd backend && uv run pytest --cov)           # backend suite, 100% gate
(cd frontend && pnpm test && pnpm run typecheck && pnpm run check)
```

```
app/
├── backend/                # haiku_rag_app package, a uv workspace member
│   ├── haiku_rag_app/
│   │   ├── main.py         # create_app(config, sessions_path), Databases, /health, /api/databases
│   │   ├── chat.py         # AG-UI stream over stored sessions, the documents listing
│   │   ├── sessions.py     # SQLite session store, holds, the turn commit
│   │   ├── inspect.py      # inspector routes and /api/search
│   │   ├── images.py       # checks on images the page sends, pictures, visualize
│   │   ├── jobs.py         # Jobs: one worker thread with its own event loop
│   │   ├── doctor.py       # doctor and similarity routes
│   │   └── similarity.py   # document centroids compared, near-duplicate families, Retained
│   └── tests/
├── config/                 # haiku.rag.yaml.example, databases/ (fragments), mounted at /app/config
├── frontend/               # Next.js, Tailwind, shadcn/ui, @ag-ui/client
│   ├── app/                # pages: / (chat), /ingestion, (inspector)/ for /inspect/[[...path]] and /search, /doctor/[[...db]]; api/backend, api/ingester proxies
│   ├── components/         # chat/, ingestion/, inspect/, doctor/, ui/ (shadcn)
│   └── lib/                # tested logic: addresses, chat, sessions, ingestion, inspect, doctor, proxy
└── docker-compose*.yml     # backend, ingester, docling-serve, frontend: published images, or (.dev) built from the checkout
```

- **Published images.** `docker-publish-app.yml` pushes `haiku.rag-app-backend` and `haiku.rag-app-frontend` after the slim package publishes, tagged with the `haiku_rag_slim` version and `latest`. `docker-compose.yml` names them and `haiku.rag-slim` at `${HAIKU_RAG_VERSION:-<release>}`, a default `bump_version.py` rewrites. The backend bundles haiku.rag, so its release must match the ingester's. The backend runs as uid 1001 and mounts `DATA_DIR` at `/data` read-only.

- **The backend reads, the ingester writes.** `Databases` opens one read-only client covering the configured set on first use and retries a database that failed. The ingester is the only LanceDB writer, and the backend writes only its session file.
- **`POST /reload` swaps the database set.** `Configuration` (main.py) holds the current `Databases`, and `PinDatabases` puts it on `request.state.databases` for the whole request, a streamed chat run included. Handlers read it there, never `app.state`, and the chat agent lives on it. A replaced set closes when its last request ends, and the lifespan awaits those closes. The reload refuses any change outside `lancedb.databases` (`changed_sections`, shared with the ingester) except the `ingester` section. Doctor jobs and kept centroids are keyed by `DatabaseRef`, so a repointed name shows nothing of its old location.
- **One uvicorn worker.** `Sessions` holds a session for a run, a rename or a deletion in process (`admit`, synchronous, before any await), and the Dockerfile and dev compose pin `--workers 1`.
- **The backend owns the run input.** `stream_chat` takes only the new user message from the client and builds the run from the stored messages and state, clearing tools, context and forwarded props.
- **The commit is the durable boundary.** On run completion `chat_events` commits the turn through `Sessions.start_commit`, shielded. The session stays held until the write lands, even when the response is collected (`tie`). A run cancelled earlier stores nothing. The frontend reconciles a lost stream by run-id membership in `runs`, and keeps the pending run in `sessionStorage` across a reload.
- **Question identities are message counts**, so a run's messages and the state it advanced are kept or dropped together.
- **Doctor and similarity jobs** run on one daemon thread with its own event loop, one at a time across databases, each over the `Store` it opens. Progress reaches the request loop through `call_soon_threadsafe`. A similarity job's unit centroids are kept within one 256 MB budget (`Retained`), and answer the nearest-documents lookup.
- **The frontend proxies.** `/api/backend/*` and `/api/ingester/*` stream bodies and forward the browser's abort. The backend proxy refuses `reload`, and the ingester proxy adds `INGESTER_TOKEN` and forwards only the operator actions as writes (refresh a source, retry a job or a dead letter, cancel a job, edit or delete a document). The inspector offers document edits only on an untagged database listed by the ingester's `/sources`, and shows the ingester's answer to a save while the document stays open. Elsewhere the backend's read shows an edit or a delete after up to `read_consistency_interval_seconds`. The dev server allows the `127.0.0.1` origin.
- **`BASE_PATH` is build-time.** `next.config.ts` validates it (`basePathFrom`, `lib/paths.ts`) and sets Next's `basePath`, which prefixes router navigation and `<Link>`. Every other root-absolute URL (`fetch`, `<img>`, a plain `<a>`, the chat's `HttpAgent`) goes through `withBasePath`. The published frontend image serves at the root, and the Dockerfile takes `BASE_PATH` as a build argument. The dev compose passes it as the dev server's environment.
- **The address is the inspector's state.** `app/(inspector)/layout.tsx` renders `InspectPage` for `/inspect/…` and `/search`, which reads its view from the path and the query (`lib/addresses.ts`), so it stays mounted while the address changes. `DocumentView`, `ChunkView` and `SearchPanel` take that state as props. The similarity job's controls and families live in `components/doctor/SimilarityJob.tsx`, and the memory estimate in `components/doctor/Estimate.tsx`, shared by the doctor and the inspector's similar documents. An image search's image and answer live in a module variable in `SearchPanel`. `components/SearchOverlay.tsx` hosts the same panel in a dialog mounted in the root layout: Ctrl+K or Cmd+K, or `openSearch()`, opens it.
- **Shared page pieces.** `components/Page.tsx` (`Page`, `PageHeader`, `SectionHeading`) frames the doctor and ingestion pages, and `components/Status.tsx` (`Loading`, `Empty`) is every loading and empty state. The theme follows `prefers-color-scheme` in `globals.css`.
- **Frontend tests.** `lib/*.test.ts` run in node. `components/**/*.test.tsx` run in jsdom (`@vitest-environment jsdom` per file) with React Testing Library against mocked `fetch`. jsdom has no `ResizeObserver` or `PointerEvent`, which `vitest.setup.ts` supplies, and no 2D canvas context.
- Run `biome check` in `app/frontend/` after frontend changes (`pnpm run format` to fix).