# Retrieval architecture Extend parses documents into source text, pages, and layout blocks. Text formats are decoded locally. The index records heading structure, sections that continue across pages, page ranges, links, and exact block references. Markdown code fences do not create headings. On multi-column pages with clear placement conflicts, block coordinates repair paragraph ownership under the preceding heading in its column or spanning article. Existing indexes receive the same repair when read; source Markdown, block IDs, and coordinates are preserved. Prose becomes overlapping 2,400-character source passages. HTML tables instead split at complete row boundaries, repeat column headers, captions, and bounded adjacent context, and keep rows connected by rowspan together. A single row or connected group may exceed the target size rather than lose cells. A versioned read-time upgrade rebuilds old passages from existing extracted content without another parsing-provider call. Indexing does not call a chat model or generate summaries. It stores structural descriptions from heading outlines and page ranges, alongside the original source text. Chat providers generate conversational answers and, separately, occasional folder names and descriptions for [automatic filing](organization.md). The two JEV references solve different problems. Hierarchical classification traverses a supplied tree using Choice distributions; it does not generate that tree. GPT Researcher's context filter scores raw passages from pages already found by web search, then supplies accepted passages to a writer. That filtering stage does not require document summaries or a persistent hierarchical index. This library uses its accessible document tree for candidate selection and the same usefulness-scoring principle for evidence filtering. Heading quality influences initial document selection; section routing also receives query-aware source excerpts. At query time the backend reads lightweight metadata and stored structural outlines, then constructs the caller's accessible hierarchy. Full indexes and section trees load only when traversal explores a document, after fresh authorization checks. Reaching a section supplies its original passages to the context filter directly; a second routing decision cannot discard passages before they are scored. Attached documents constrain the initial database read and hierarchy. Category descriptions include bounded child outlines that are authorized again for each request; these descriptions are never stored as a shared summary. Their size decreases for wide sibling menus. All reads use SpiceDB permissions. Whole-library searches can also enter a small direct-candidate branch. Corpus-frequency-weighted matches in authorized names and headings propose up to eight sources; extracted acronyms and captions supply weaker additional hints. Those hints never replace the folder's heading previews or bypass JEV routing and evidence scoring. The original hierarchy remains available for recovery. A document's full index is shared across its routes within the request, with authorization checked on each use. Retrieval traces restore the source's authorized folder ancestry when it is reached through this shortcut, without inventing routing probabilities for those ancestors. Search profiles are bounded derived metadata, generated from extraction during indexing. Stored generated columns keep routing reads independent of the large parsed payload. Existing ready documents can receive profiles with `scripts/reindex-search.ts --org=`; it uses cached extraction, requires organization administrator access, and updates only an unchanged ready revision. It does not reparse documents or alter their original text, blocks, structure, or index timestamp. Each explored document also exposes a statistics branch to the same router and evidence filter. It computes PDF page count, extracted word count, distinct parsed figure/table counts, numbered caption counts, abbreviation frequencies, query-term occurrence counts, and a bounded section inventory from the full extraction. Complete token matches and literal occurrences within compound tokens or URLs are reported separately. Unicode, line-wrap hyphenation, and dash/plus spacing are normalized consistently. The evidence records its counting method and inventory truncation; parsed objects can differ from logical figures or photographs in the visual document. Numbered source captions group split panels under their original figure label, but do not count unrecognized or unlabeled figures. A heading inventory locates evidence without establishing complete author lists or factual statements. This branch never uses reference answers or forces metadata into the returned context. It undergoes the same live authorization and usefulness checks as source passages. Automatic filing changes the folder portion of this hierarchy. After parsing, JEV classifies an incoming document against existing folders using headings and bounded original-text samples. It may descend through existing subfolders or request a new branch whose names and descriptions come from a separately selected model. JEV validates that branch before it is created. This does not replace the document's parsed section and subsection structure, and it does not generate document summaries. Upload-driven organization reviews reuse this hierarchy to shortlist related older documents. This lookup preserves category branches even when constrained to eligible document IDs, scores at most 24 passages, and returns at most eight passages. It keeps conservative routing: branches rejected by none are excluded rather than recovered. Only those shortlisted documents can receive full placement classification; JEV compares their current and proposed paths before moving. Normal chat retrieval and attached-document behavior keep their existing budgets and scoping. JEV Choice evaluates sibling menus. Each frontier batches up to four independent routing questions, with a 96,000-character menu budget per request. Menus have at most 64 children plus a none option, letting moderately sized folders expose document names and full structural descriptions directly. Larger menus gain intermediate routing groups that share the available menu budget; category routes remain directly visible when they fit alongside grouped documents. The backend never compares probabilities from unrelated flat batches. Authorization work coalesces into bulk requests under a shared limit of eight concurrent requests. Checks are grouped by exact stored ZedToken, deduplicated, and bounded to 10,000 items per request. Completed decisions are retained only within the current request and committed graph version and token. Every reuse reads the current snapshot, invalidating cached decisions when permissions change. Permissions are validated before each provider request and after responses; final results and traces reuse that same snapshot-aware cache. A stable 128-document retrieval checks each resource once in one bulk request. Descriptions are partial outlines: a none probability at least as large as a child defers that route rather than permanently excluding its contents. Automatic filing keeps its separate 16-child menu limit. The beam retains four routes by `exp(sum(log(edge_probability)) / decision_count)`. Single-child transitions do not count as decisions. A probability floor of 0.000001 permits bounded exploration of routes assigned zero probability. Traversal yields candidate sections before continuing deeper, allowing evidence quality to guide further exploration. After each eight section visits in one document, unvisited documents gain priority over its remaining sections. Pending passages similarly share the scoring budget across documents, and unused passages stay available for later batches. Deferred routes remain available when evidence is weak. Exploration is capped at 96 node expansions and 96 passage scores per query. JEV Score then evaluates candidate source passages independently on a four-level usefulness rubric, following GPT Researcher's separation of candidate collection and context filtering. Scores below 1.5 are excluded. Up to sixteen passage questions share one scoring request. Each question contains only its own full evidence; the query is shared in state. At most two scoring requests run concurrently across the process, with live authorization checks inside the limiter. Missing, unexpected, or invalid batch answers reject the entire batch. Normal retrieval stops when a passage scores at least 2.75 for the entire question. When three or more partial passages accumulate, a bounded evidence-set check determines whether they jointly cover every requested entity, relationship, time period, and qualifier. A partial set keeps exploration open; unchanged sets are not checked again. These checks share the scoring limiter and reauthorize their input. Conservative organization lookups retain their three-passage stopping rule. Results are ordered by usefulness, then route score, and limited to twelve passages and 24,000 characters. A routing score ranks a path; it is not a calibrated probability that a passage answers the question. Empty results remain empty, and invalid or unavailable providers fail explicitly. Retrieval paths include per-edge probabilities, normalized route scores, and scored passage visits. Citations identify the original document, section, passage, page range, and source blocks. Answer tools expose a source path and precise block references alongside each passage reference. The answer policy asks for the smallest block that supports each claim. Inline citation pills resolve those references to stored block IDs and page positions, open the original source, and highlight its recorded bounding box when geometry is available. Full-document links preserve that same target. Repeated passages on different pages retain separate citation identities. The chat model answers from these passages and may make another bounded search when evidence is insufficient. Search tool instructions ask for self-contained queries that include relevant topics and document references from prior conversation turns. This implementation supports TypeSafe and Cloudflare Clef as decision providers and does not require PageIndex, its cloud service, or a vector store. Metadata matches propose a routing branch, rather than returning keyword matches as final evidence; missing or invalid provider responses still fail explicitly. Exact query-term counts are metadata evidence. PageIndex's separation of document structure and source reads informs the design; its runtime is not integrated. With Cloudflare Clef, candidate passages containing figure or image blocks from PDFs or uploaded images are scored with the original visual evidence. Stored block IDs select the corresponding pixels; valid coordinates and rotation metadata produce a crop, otherwise the original page provides context with the target block identified. Uncaptioned visual blocks receive temporary retrieval passages without changing the stored index. Images are mapped to each scoring question and batched within the four-image limit; passages with more images are evaluated in subsets and retain their highest usefulness score. Ordinary text passages, hierarchy navigation, evidence-set coverage checks, and TypeSafe requests remain text-only. Rendering is limited across the process, cached only during the retrieval operation, and rechecks document access and source freshness before images reach the provider. The answer model still uses its existing page-viewing tool to inspect visual facts. Section routing descriptions include bounded normalized source excerpts by default after a document is explored and authorized. Query-term weighting, length normalization, opening context, and a redundancy penalty select up to three windows from the section and its children within the existing 1,200-character cap. Sections without usable source text and the computed statistics branch retain their structural descriptions. Sampling does not alter stored extraction or answer evidence. The [October 2 pilot](section-sampling-2026-10-02.md) records the matched evaluation, manual label review, and payload-size tradeoff. References: [TypeSafe hierarchical classification](https://docs.typesafe.ai/cookbooks/hierarchical_classification), [JEV context filtering](https://docs.gptr.dev/docs/gpt-researcher/gptr/context-filter), and [PageIndex document trees](https://docs.pageindex.ai/sdk/documents).