# Long-document indexing and recovery

[Handbook](README.md) · [Related guide](EVENTS_AND_PROGRESS.md)

Each knowledge base records its latest indexing run in `.indexing-run.json`.
The Files/detail view exposes **Indexing recovery details**, including individual
source outcomes, current phase and phase-local counts, elapsed runtime, time
since output/activity, and time since measurable progress. A phase reaching its
last batch is not proof that the whole index is ready. Unknown work remains
unknown after interruption; it is never counted as a successful document.

`GET /api/knowledge-bases/{id}/indexing-readiness` performs a lightweight parser,
format and embedding-configuration check without downloading models or sending
a provider request. Readiness does not guarantee later downloads, available RAM
or GPU memory, nor a provider's runtime response. Runtime failures retain their
bounded, credential-redacted reasons in the durable run.

Retry uses the existing content-addressed parser cache: source bytes and parser
signature must match, and usable text or blocks must still exist. MinerU cloud
PDF slices reuse only validated checkpoints. Changing embeddings can reuse
parsing, but builds a separate index version for the new embedding signature.
Interrupted stages without a usable checkpoint restart; the journal itself is
not a second cache and does not fabricate partial embeddings.

Cancellation is task-specific and takes effect at a safe boundary. Managed
local MinerU processes also observe cancellation and fail after 600 seconds
without output or 7200 seconds total. Other synchronous engines cannot be
interrupted arbitrarily; cancellation remains pending until they return. A
stalled LlamaIndex executor is barred from overlapping same-process retries
until it actually exits, and cannot publish its candidate after losing ownership.

LlamaIndex rebuilds and incremental additions use fresh version directories.
A candidate stays unpublished until its persisted index reloads, its vectors
validate and a retrieval probe using a stored vector returns a node. Publication
metadata is written atomically. Source changes during indexing reject publication.
A failed or partial candidate does not overwrite a previous queryable version.
Completed and failed sources are reported separately, including partial success.
The durable journal is scoped to the active user/workspace knowledge base;
linked/read-only knowledge bases cannot be cancelled through mutation routes.

LlamaIndex hybrid retrieval keeps dotted identifiers such as `2.6.8` intact
and retains single-character terms, subject to the existing English stopword
filter, during BM25 indexing and queries. Persisted
BM25 sidecars record their token pattern. Older sidecars without that pattern
are rebuilt in memory from the stored documents, without new embedding calls.
A bounded cache reuses these rebuilds for the loaded index and requested result
count; publishing an updated index loads a new identity and rebuilds as needed.
Queries leave the old sidecar unchanged; the next successful indexing run
persists the updated BM25 cache.

## Implementation and verification

- [deeptutor/knowledge/indexing_run.py](../deeptutor/knowledge/indexing_run.py)
- [deeptutor/services/parsing/cache.py](../deeptutor/services/parsing/cache.py)
- [tests/knowledge/test_indexing_run.py](../tests/knowledge/test_indexing_run.py)