# Live-query materialization architecture This document defines the architecture for correlated live-query materialization in `@tanstack/db`. It follows [RFC #1658](https://github.com/TanStack/db/issues/1658). The central rule is simple: > Keep relation contents, routes, nested materialization, and propagation in > one D2 graph. Use custom state only at asynchronous source and public > Collection boundaries. The correlated-materialization oracle suites listed below are behavioral contracts for this design. Functional projections accept inline include values, not compiled Collection-valued inputs. Suites for adjacent planner and query-db ownership boundaries may also contain exact classifiers for defects outside this graph. ## Scope This architecture covers: - compiled identities for sources, relations, and materialization edges; - weighted contributions and public-key reduction; - correlated routes and ordered bucket contents; - nested inline and Collection-valued materialization; - lazy and progressive source demand; - coherent publication to public Collections; - the boundaries with query-db ownership and physical query planning. The applied-settlement receipt described below is its only new public boundary contract. Optimistic transactions are another source of weighted input changes; they do not have a separate routing model. ## One relational graph Correlated materialization is part of the compiled D2 graph, not a second incremental engine around its output. ```text raw weighted query rows | v public-key reduction | v CanonicalRow(base row, order, outgoing parameters) | +------------------------------+ | | v | Route(bucket, cell) | | | +--> distinct --> ActiveBucket-+--> async demand adapter | | | | v | | child rows --> BucketValue | | +------------------------------+ | v CellValue(cell, value) | v CanonicalRow + outgoing CellValues | v MaterializedRow | v one normal root Collection transaction ``` Canonical base rows flow down to derive correlation routes and source demand. Fully materialized child rows flow up into their parents. Because the include graph is acyclic, these streams form one acyclic D2 graph even though demand and results move in opposite conceptual directions. The graph owns the data plane. A small adapter owns asynchronous demand. The normal Collection transaction boundary owns public publication. For each active sync run, the Collection builder keeps one set of ordered source loaders. It checks them after graph steps and once when a turn starts without graph input. The transaction scheduler coalesces graph jobs; a sync generation fences jobs left over from an older run. The loaders' request latches, rather than per-job loader callbacks, stop repeated acquisitions. The turn processes synchronous loader writes through the graph before it publishes, including when the turn started without graph input. The scheduler waits for a dependency only while that dependency has a queued job in the same context. Source demand and ordered loading use their own settlement barriers outside the scheduler. ## Concrete implementation map The relation and identity names in this document describe the graph's logical model. They are not a second set of runtime objects, nor does every name need a matching TypeScript type. The implementation maps this model onto existing D2 operators and a few boundary adapters: | Architectural role | Concrete implementation | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------ | | Compile relation IDs and demand plans | `packages/db/src/query/compiler/index.ts`, `packages/db/src/query/compiler/joins.ts` | | Reduce public keys | `packages/db/src/query/live/public-key-reduction.ts` | | Build include routes | `packages/db/src/query/live/materialized-pipeline.ts` | | Run the graph and publish root rows | `packages/db/src/query/live/collection-config-builder.ts` | | Schedule Collection and Effect graph turns | `packages/db/src/query/live/graph-scheduler.ts` | | Publish Collection-valued buckets | `packages/db/src/query/live/bucket-facade-adapter.ts` | | Start and release asynchronous demand | `packages/db/src/query/live/subset-demand-controller.ts`, `packages/db/src/collection/subscription.ts` | | Ordered provider loading and continuation | `packages/db/src/query/live/ordered-source-loader.ts` | Operator values carry their compilers, so an app bundles only the operators it imports. `include()`, `toArray()`, and `materialize()` carry the materializer and the Collection-facade adapter (`includes-runtime.ts`); aggregates and `groupKeys()` carry the grouping compiler. A query that uses an operator always holds the value that carries it, so compilation stays synchronous. Joins are core: the join compiler and public-key reduction (`public-key-reduction.ts`) ship with every live query, without the include materializer. Queries without includes keep the original compiled pipeline and do not pay for facade state. The one exception is a joined query with a custom public-key function: its possible duplicate contributors still pass through the keyed reduction that enforces public-key congruence and multiplicity. The include materializer uses the same reduction for its root rows. ### Loading handoffs These owners cooperate; they are not phases of one exclusive state machine. The detailed loading and publication laws below still apply. | Owner | Accepts / retires | Does not establish | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | | Subscription acquisition | Retires the old acquisition lease before replay acquisition; installs tentative ownership before adapter callbacks; each lease gets one cleanup attempt | Replay completion or permission to publish | | OrderedSourceLoader | Tracks request settlement, safe continuation and repair debt; reset discards the cursor, disposal ignores late settlement | Provider exhaustion or acceptance of an imperative window | | Subscription replay | Counts setup and logical demand participants; checks completion after reentrant release callbacks; success releases the source replacement hold | Success of a previously failed window operation | | Query builder | Tracks ordered publication participants in one sync run and accepts a window only for its operation generation | Physical adapter ownership or cancellation | | D2 and public Collection boundary | D2 accumulates private result changes; the builder flushes root and child changes when the existing gates allow it | Source completeness merely because graph work drained | Sync-run and participant checks precede changes to the builder's ordered failure state, not just scheduling. An obsolete rejection cannot close a replacement sync run's publication gate. Loader-local stale-result guards are separate. `hasPendingTruncateReplacement` means publication is still withheld, including after replay failure. `pendingTruncateReplacement` exposes only an unsettled completion promise. Neither is a general readiness flag. A direct subscriber buffers and diffs its own replacement rows; a query subscription delegates publication to the builder while the graph keeps its private contributions. ## Identity Aliases are lexical query-language names rather than source runtime identities. The query builder requires source aliases to be unique within each lexical scope. A nested query may shadow an ancestor alias: a reference captured from an earlier callback retains that source's lexical binding, while a callback in the nested scope reads its local source. Sibling scopes may also reuse aliases. The builder gives each source reference a binding ID when it is created and preserves that ID when it places or rewrites the query. Compilation assigns separate opaque IDs to the accepted plan: A source declaration may be placed in sibling query scopes, but one builder's source declaration cannot serve as both an ancestor and a descendant in an include. Those scopes need distinct binding IDs. Start the descendant with a fresh `new Query().from(...)`, even when it reads the same Collection. A bound reference captured from an unrelated query is outside the lexical scope, even when its alias matches a local source, and must be rejected before its expression can accept rows. This applies to join operands, predicates, grouping, ordering, and projections. Functional callbacks receive row values at runtime and do not carry statically inspectable binding references. A `QueryRef` or union branch exports its result, not its internal source declarations, to the containing query or an include. A joined `QueryRef` may read actual ancestor route context, but it does not receive the enclosing query's sibling source rows. A captured sibling reference inside that joined source must be rejected; the enclosing join condition combines the two sources. ```ts type SourceId = Brand type RelationNodeId = Brand type MaterializationEdgeId = Brand ``` An explicit structured projection can alpha-normalize aliases because its field names define the public shape. Its query identity resolves captured references by binding role, including when the captured alias is shadowed. A functional callback may inspect its input keys; alias equivalence applies to it only when the callback is alias-equivariant. Without a projection, joined and grouped queries return a namespaced row whose keys are the lexical aliases. Those observable keys are part of query identity. A functional callback still receives the user's lexical keys. Alias text may otherwise remain as debug metadata without becoming source identity. A plan rewrite must preserve each Collection reference's `SourceId`. The optimizer reuses the reference when it copies, wraps, or collapses a source, and compilation reads source inputs by `SourceId` only. A source whose identity was lost raises `CollectionInputNotFoundError`; it never reads a same-named source from another scope. The branches of one `unionAll()` share an alias namespace, and the compiler rejects a name that two branches repeat. A union row holds the branches' projected fields, so a branch alias is not visible to the union's own joins or includes. Join conditions are an equality or a nonempty, possibly nested conjunction of equalities. `JoinClause.on` keeps the complete expression tree. Every equality contributes to query identity, optimizer copies, and parent-reference discovery. Each operand pair binds the joined source to an available source expression; the pairs may reverse operand direction independently. OR and non-equality predicates are rejected at join admission. A compound match requires every component equality to be TRUE. Any nullish operand prevents a match. Each component uses the same graph-scoped equality identity as a single-condition join; raw JSON representations do not define value equality. Lazy loading may use the first equality to acquire a superset of candidate rows, while the compiled join checks the full tuple. A tuple with any nullish component contributes no keyed lazy demand. A parent reference in any joined-side component requires route context before evaluation. A `CanonicalCorrelationKey` is the canonical tuple of every evaluated parent-dependent value that can affect the child plan. This includes values used by filters, joins, grouping, aggregates, ordering, projections, limits, and nullable predicates, not only the obvious foreign-key equality. A bucket key identifies one such correlated partition at one relation node: ```ts type BucketKey = readonly [ relationNodeId: RelationNodeId, correlationKey: CanonicalCorrelationKey, ] ``` Correlation equality must use the same value semantics as query predicates. Implementations use canonical values, interned handles, or nested maps; they do not reconstruct array or object keys and expect JavaScript `Map` identity to match. Equality tokens collapse `-0` with `0`, compare Date, Temporal, and binary values by the same normalized value as `eq`/`in`, and retain runtime reference identity for other objects, functions, and symbols. These tokens are valid only for equality-keyed routing, grouping, and demand. Output values and arbitrary function arguments keep their exact runtime identity and value. Tree indexes give symbols a stable runtime-local order because JavaScript relational comparison throws for them; comparator equality still holds only for the same symbol. That order is a physical index detail: symbol range predicates fall back to the evaluator instead of treating it as query semantics. Range predicates also fall back when the live indexed values do not share the bound's relational domain. An index's advertised comparison options also define its executable comparator; metadata cannot claim an order that the index does not use. Explicit `undefined` range and cursor bounds denote the indexed nullish comparator group, while an absent bound denotes the start or end of the index. An ordered index groups exact value buckets that compare at the same position and keeps a live representative for each group, so range traversal and ordered limits cannot drop rows whose distinct values are comparator-equal. Compiler tokens belong to one compiled graph. This keeps every operator in the graph on the same identity relation. Objects, functions, and local symbols are weakly keyed where the runtime supports weak symbol keys. Older runtimes retain local symbols strongly within the scope rather than collapse distinct symbols and corrupt equality. Registered symbols use their registry key because the runtime registry already retains them. A demand controller owns a separate scope and discards it when the controller is cleared. Process-wide query identity and opaque public group keys keep their own runtime scope because equivalent query plans and retained public keys must survive graph replacement. For grouping, the equality token is the D2 group key. The group retains a raw value from a currently positive contributor only as the projected representative. The representative is the contributor with the smallest exact value: another primitive before -0, and every primitive before an object. Objects are ordered by an explicit type tag (`Buffer`, `Date`, a Temporal type, then `Uint8Array`). Contributors of one tag are equal in content, so the contributor with the smallest row key supplies the instance; the choice does not depend on update history, and a deleted contributor's instance is never projected. Equal primitives share a representative key and consolidate. D2 consolidates contributions whose hashes match, and its hash treats -0 as 0 and equal Dates as one value. Each contribution therefore carries the exact identity of every value a contributor can supply: the representative key, and the exact input of every min or max, taken from the value that the min or max compares. A primitive input is keyed by its exact value and an object input by its contributor's row key, because a rebuilt argument is a new instance at each evaluation and its retraction must still cancel its insert. A sum, avg, or count adds coerced numbers, so merging its equal inputs cannot change its result. Contributions merge only when those identities are equal, so a merged contribution supplies only a value that a positive contributor holds. A correlated include's route representative carries the correlation key and parent context instances. Members that read the correlation key from their own row hold distinct instances when the key is an object, such as a Date or a binary array, so those members do not consolidate. A count, sum, or avg over consolidated contributions updates without re-reading the group; a min or max over distinct primitive values keeps one contribution per distinct exact input, and a min or max over objects keeps one contribution per contributor. D2 sees only safe exact-value identity for a representative, not the raw value itself. A separate public group key preserves primitive keys and serializes opaque equality identity; graph-local identity tokens never cross the Collection boundary. Compiler group fields use a query-local namespace disjoint from every selected alias. Direct correlated joins canonicalize both sides before the first D2 join; normalizing only the later group key is too late. An aggregate query has one row per group: one group without `groupBy`, or one per group key, within each route inside an include. Each select value must have one value for its group outside its aggregates: a literal, a group key (a `groupBy` expression, a ref or a computed one), or a parent field that the route's parent context supplies. Every such value is evaluated with its group keys substituted, as the parts of a wrapped aggregate are. Any other field of the query's own sources has none, wherever it appears, so the query throws `NonAggregateExpressionNotInGroupByError`. A spread throws too, because a parent context holds only the parent fields the query names, and so does a nested include in a grouped or aggregate select, which has one Collection per row. A source ref is identified by its lexical binding, so a parent ref keeps its meaning when the include's source shadows the parent alias. Group keys match by structure, with literals compared by value. A HAVING condition follows the same rule: it reads group keys and parent fields, and a source field outside an aggregate throws. A HAVING aggregate reads the same aggregate that the select names as a top-level field. Recorded limit: parents that are equal under query equality but differ exactly, such as `x: 0` and `x: -0`, share a route, so a parent field can show the other parent's exact value, which is equal under query equality. ### Route-context transport A parent reference is a lexical dependency, even when it appears below the immediate child query. For every parent reference that the builder can inspect, the compiler must: 1. discover it across nested includes, `QueryRef` sources, union branches, and joined sources; 2. include its evaluated value in the route identity; 3. attach that route context before the first operator that evaluates it; and 4. preserve it through each later recursive source, join, grouping, and materialization edge. The third rule fixes the evaluation order. A parent-dependent filter, join key, aggregate wrapper, order, or window must run once per parent route. It cannot run on a shared child relation first and receive a route after the fact. The route-context grammar crosses these dimensions: ```text lexical dependency scope x recursive source boundary (nested include, QueryRef, union) x recursive result shape (record, scalar, nullable scalar) x evaluation phase (filter, join, group, aggregate, order, window) x join side and correlation attachment point x materialization form x parent or child update ``` Adding one dimension to the query language requires checking its product with the others. A passing one-level filter case does not prove a nested aggregate, joined subquery, or union branch transports the same context. The executable oracle factors that product into valid compiler sub-grammars: - parent field projection by whole-row projection; - unmatched correlation values by null correlation values; - lexical scope, including nested outer and inner materialization forms; - grouping mode by aggregate-expression placement; - recursive source boundary by evaluation phase; and - join-key side by correlation attachment point; - union form and public-key identity; and - derived-result boundary by selection mode and scalar nullability; and - user namespace collision by parent alias and selected child field, crossed with direct, `QueryRef`, join, and group boundaries; and - public-surface shape across opaque atomic values, opaque wrappers, nested reference identity, user symbol keys, adversarial property keys, functional spreads, and implicit joins. At user-value boundaries, compiler expressions are identified by their constructed IR classes. A plain user object remains data even when it has fields such as `type: 'val'` and `value`; source rows, selected objects, functional results, and predicate literals must not be unwrapped or rejected because of that shape. Selected arrays of references evaluate each member into an array of row values. User fields named `__refProxy` remain data; only constructed reference proxies have proxy semantics. Plain record results carry route metadata under a private symbol while the compiler moves them through recursive sources. Primitives and opaque objects, such as `Date`, use an internal envelope at those edges. FROM and JOIN adapters extract the route before exposing a source value. Route context, include routing and source metadata attribution live beside the aliases on the namespace. Source values exposed to callbacks therefore contain no compiler-owned state. Callbacks and result finalization strip private keys at the known row root. Materialization compiles a resolver for its declared include paths. It resolves facade references and cleans nested materialized rows along those paths only. Ordinary source payloads remain opaque, including in correlated queries and grouped HAVING. Clean values retain their references. Copies preserve property descriptors, prototypes and user symbols; discovering private state does not invoke accessors. The internal facade adapter requires an explicit resolver. Compiled queries supply their declared-path resolver; direct component drivers can supply the descriptor-preserving public-container walker. Internal clean references can be equality operands, not just render identities. A cleanup copy must not change a later `eq(projected.key, parent.key)` match. The public-container copy matrix checks reference keys through ordered and unordered subqueries, materialization forms, and parent/child updates. The includes-work owner checks zero metadata descriptor scans inside source payload containers through fourteen direct and correlated receiving paths. This work law does not bound hashing, user code or elapsed latency. Reserved public row fields obey the boundary-only metadata rule (maintainer decision, 2026-11-05, superseding the October 28/30 callback-normalization rules). Actual Collection row roots expose genuine computed `$key`, `$origin`, `$hasPendingWrites` and `$collectionId`. Structured whole-source selections and spreads retain source attribution; functional selections retain it only when returning the source row directly. Joined occurrences use their computed pair key for `$key`, or the explicit `getKey` result. Source attribution retains `$origin`, `$hasPendingWrites` and `$collectionId`; a source key cannot collapse several joined occurrences. Inline values keep their authored `$key` and multiplicity. Other published projections receive their output Collection's metadata. An authored reserved field cannot override that attribution. Intermediate QueryRef projections, functional callback inputs, inline includes and Effect outputs keep their authored values. For example, an intermediate `{ id: 1, $collectionId: 'application' }` exposes `'application'` to downstream `fn.select`, `fn.where` and `fn.having`, and to inline consumers. Direct source rows already carry genuine metadata; copying them copies those fields, while an explicit override remains application data until Collection publication. Reading a projected field is not evidence of Collection attribution. The compiler transports one authored value per occurrence and a separate record of at most four source metadata facts. The record permits genuine publication attribution after an authored spread override, without keeping a second full callback view. Cache slots distinguish authored roots, actual Collection publication, and intermediate sources. An authored root owns its movable top-K independently of a consumer of the same QueryIR. Grouped intermediate results inherit a missing generated `$key`; an explicitly authored field still wins until publication. Each ordered top-K retains its own window; the source loader's registered plan covers every window sharing that source, including subsequent moves. Ordered plan aggregation also retains every joined source whose pending demand gates a retained plan. An outer ordered QueryRef without a join cannot erase its child's joined dependency. A second ordered join adds its dependency to the first. Ordered continuation and Collection/Effect publication wait for all of them. The ordered-work owner checks this composition through actual Collection and Effect receivers while either joined acquisition is held. The runtime owner checks select, where and having source inputs through root, wrapped child, direct child and real Effect receivers. Predicate-with-select controls distinguish callback syntax from placement. Collection receivers observe initial readiness, held mutation, settlement and remote next use. Effects additionally observe every delivered value and previousValue, exit/reentry and post-disposal silence. Its bounded evidence is in [the callback metadata record](../../../../../docs/contributing/oracle-reviews/callback-metadata.md). Reusing a compilation cache on one graph with the same input streams must retain the callback inputs and authored selected output that separate compilations produce. The exported `compileQuery` receiver checks both compilation orders through FROM, JOIN and union placement at initial graph drain. Its raw output comparison is differential and supplies no independent Effect-root attribution policy. Optimizer cloning can prevent an original raw-key cache hit in union placement; that path checks propagation and transparency instead. Separate static receiving-type limits remain in the [deferred oracle record](../../../../../docs/contributing/oracle-reviews/2026-10-29-pr-2171-deferred-oracles.md). D2 hashes enumerable symbol keys and uses exact local-symbol identity plus registry keys for registered symbols. D2 rejects structural cycles with a clear error, including cycles through arrays, Maps, Sets, and enumerable symbol keys. Shared acyclic subtrees remain supported and are hashed once per traversal. Structural hashing limits recursion depth and value visits; it rejects values that exceed these limits instead of expanding a shared graph or overflowing the JavaScript stack. This does not bound the cost of arbitrary user getters or key sorting. A failed hash does not publish partial structural cache entries, so retrying the same value cannot bypass a guard. A graph-run failure marks the current live query as errored and preserves the thrown error. It must not continue publishing from a partly advanced graph; recovery requires a fresh live-query sync run. Opaque reference-hashed leaves are resolved before structural recursion; their own properties, including self-references, are not traversed. Hash inputs must remain immutable once successfully cached, as with other retained D2 values. Collections register as opaque handles at construction using the existing hash cache. Their identity, not their mutable internal state, is visible to hashing operators in a downstream query. This does not add child-row dependencies to a functional projection that reads a Collection-valued field. The descriptor-preserving boundary walkers may still encounter cycles, but that does not make cyclic structural results valid input to a hashing operator. Symbol-only changes cannot disappear before publication, and unsupported cycles fail rather than silently merge. Neither boundary mutates values retained by D2. Compiler-created parent contexts use a separate internal envelope that keeps projected user aliases apart from the equality identity derived from their leaves. The whole parent-context envelope is structural D2 state. This avoids reserving user aliases or selected field names while keeping the context stable across D2 operators without collapsing two reference-sensitive leaf values that happen to have the same object shape. A functional projection consumes fully materialized inline input before downstream operators run. Compiled Collection-valued includes are not supported as `fn.select()` inputs, including nested descendants. The compiler rejects the plan before invoking the callback, even if the callback would ignore or pass through the Collection. This keeps callbacks inside the ordinary D2 pipeline without temporary Collection views or graph continuations. Use `toArray()` or `materialize()` in the upstream expression `select()` to make child values available to a functional callback. Child changes then update the inline value and rerun the projection. To keep live child Collections, use expression projections, or do parent-only functional work before adding the Collection-valued include. This restriction concerns compiled include inputs; it does not inspect arbitrary source-row fields or captured Collections. Reading an already published Collection from a callback does not add a child-row dependency. Include paths describe a functional projection's input, not its arbitrary output. A callback may drop or rename a field, or return a scalar. Its input paths must not be attached to that output by a downstream QueryRef consumer. The compiler consumes those descriptors through the existing D2 materializer; downstream keys, distinct, ordering, and QueryRef consumers see the callback's actual output. Queries without includes keep their original pipeline. The projection matrices keep Collection-input cases as rejection checks and exercise supported inline forms across route changes, child updates, recursive sources, unions, and chained callbacks. Expression controls retain ordinary Collection reads, indexes, subscriptions, rollback, pending loads, and cleanup/restart coverage. Work counters check repeated reads on public facades. A manual forced-GC probe checks retained public handles and captured methods after cleanup, with live facades as a positive retention control; it is not a whole-application heap or throughput measurement. A materialization cell identifies one include field on one parent-row occurrence: ```ts type MaterializationCellId = readonly [ containingBucket: BucketKey | 'root', parentPublicKey: PublicKey, edgeId: MaterializationEdgeId, ] ``` The containing bucket prevents equal child keys in separate correlated contexts from colliding. Only work that crosses an asynchronous boundary needs a generation token. A live-query graph generation invalidates work from an old graph. A demand generation invalidates an old load for the same bucket. Synchronous route rows inside D2 do not need their own lifecycle objects or generations. ## Weighted relations and public keys D2 multisets are the source of truth. A row with positive weight contributes; a row with negative weight retracts the same contribution. Internal contribution identity is independent of the user-visible Collection key. When several internal rows collapse to one public key, a keyed D2 reduction retains all contributors and derives at most one canonical row: ```ts type CanonicalRow = { publicKey: PublicKey value: Row order: OrderKey | undefined outgoingParameters: ReadonlyMap< MaterializationEdgeId, CanonicalCorrelationKey > } ``` ```text raw weighted rows -> reduce by [containing bucket, public key] -> CanonicalRow +-> derive route rows +-> compose with materialized include values ``` For every affected public key, the reduction compares its complete before and after state and emits no change, one replacement, or one removal. It does not infer the previous state from `collection.has()`. Routes derive only from canonical rows. Raw contributors never create routes that must later be reconciled. The same boundary applies recursively: roots reduce by root public key, while child rows reduce by their containing bucket and child public key. All positive contributors collapsed under one public key must be congruent on: - the visible value; - the total order key; - every outgoing correlation input. Query aggregation occurs upstream in the query graph. This reduction only preserves multiplicity while collapsing congruent contributors under the public Collection key. Incongruent contributors are a duplicate-key invariant error; flush order never chooses a winner. A zero aggregate removes the public row. A negative aggregate is an invariant violation. This is a specialized use of the existing D2 keyed reduction. It is not a separate contribution-ledger subsystem. ## Routes and buckets are relations For each materialization edge, the compiler produces these keyed relations: ```ts type RouteRow = readonly [bucketKey: BucketKey, cellId: MaterializationCellId] type ActiveBucket = readonly [bucketKey: BucketKey] type BucketRow = readonly [ bucketKey: BucketKey, child: readonly [publicKey: PublicKey, row: Row, order: OrderKey | undefined], ] type BucketValue = readonly [bucketKey: BucketKey, value: Value] type CellValue = readonly [cellId: MaterializationCellId, value: Value] ``` Physical layout: a value that D2 keeps in join or reduce state under a key it shares with other values leads with its own string, number or bigint key, as the `[publicKey, row, order]` child of a bucket row and the routed parent of an inline include do. The D2 index tells apart values with different leading keys without hashing them, so a value is hashed only against a value with the same leading key, such as its own retraction. A value of another shape is still correct, but the index hashes it whenever it shares a key. This is a cost rule, not part of the relation semantics. A route move is an ordinary weighted batch: ```text -1 [old bucket, cell] +1 [new bucket, cell] ``` Distinct route keys produce `ActiveBucket`. For inline modes, child rows are ordered and reduced once per active bucket into exactly one `BucketValue`. Routes then join with bucket values to fan the same immutable logical value out as a `CellValue`: ```text Route(bucket, cell) -> distinct -> ActiveBucket(bucket) | ActiveBucket + BucketRow -> reduce ------+-> BucketValue(bucket, value) | Route(bucket, cell) -------------------------------+ v CellValue(cell, value) ``` The bucket-value reduction belongs to the materialization edge because two edges may apply different materialization modes to the same child relation. Computing it before fan-out means ordering and materialization happen once per unique bucket rather than once per parent. `ActiveBucket` also seeds the empty value. Every active inline bucket therefore has exactly one value even when it has no child rows: - `array`: `[]`; - `singleton`: `undefined`; - `concat`: `""`. A null or otherwise unsatisfiable correlation may route to an active empty bucket without creating source demand. This preserves the materialization mode's empty value instead of relying on a placeholder or a missing join path. D2's retained join indexes provide the required lifecycle behavior: - adding a route joins it with the bucket's existing value; - removing a route retracts only that cell's value; - moving a route retracts the old rows and adds the new rows in one graph run; - several cells may consume one bucket without recomputing its value; - changing a bucket value reaches every current route; - a departed route receives no later value changes. Root rows and nested rows use the same relation shape and operators. There is no special root routing path. The implementation must not recreate these semantics with route registries, reverse indexes, drained buffers, or per-depth snapshots outside D2. Existing retained operator state is the first implementation choice. Add a reusable arrangement only if counters show that the compiler duplicates indexes or state; arrangements are a physical optimization, not part of correctness. The total-materialization law is: > Every active inline materialization cell has exactly one canonical value, > including when its bucket contains no rows. ## Nested materialization The compiler builds each include from the materialized output relation of its child: ```text child base rows + child include values -> child materialized rows -> rows in the parent's bucket relation -> parent include value ``` A descendant update therefore becomes an ordinary change to the child's materialized row and propagates through the same joins and reductions at every depth. There are no depth-specific flush passes, dirty-cell registries, or manual relation revisions. Inline modes are reductions over the rows in one active bucket: - `array`: total-order the rows and return their values; - `singleton`: choose the first row under the total order; - `concat`: total-order the rows and concatenate their scalar values. A total order is the query's order keys followed by a deterministic stable tie-breaker, normally the child public key. An order-only change is a bucket-value change for arrays, singletons, concatenation, and Collection layout. `include(childQuery)` is a Collection-valued include. A bare child query in a select throws `IncludeOperatorRequiredError`. The include exposes one stable public Collection facade per active bucket in that edge: Facade rows obey the Collection virtual-field boundary. Structured whole-row selections and source spreads retain the selected source's attribution. Joined rows use pair keys for `$key`, preserving separate occurrences of one source row. The other three fields retain source attribution. Other projections receive the facade Collection's metadata. A functional selection preserves source metadata only by returning that source row directly; a new object, including a spread, receives its output Collection's metadata (maintainer decision). Selected `$hasPendingWrites`, `$origin`, `$key`, and `$collectionId` fields cannot change this attribution. Inline include values keep their selected shape because they are not published as Collection rows. ```text ActiveBucket + BucketRow -> ActiveBucketRow -> BucketFacade(bucket, Collection) Route + BucketFacade -> CellValue(cell, Collection) ``` Parents sharing a bucket share its facade. Child changes update that Collection without re-emitting every parent, and moving a route changes the parent field to the destination bucket's facade. A facade is never retargeted to another bucket. The D2 join retains inactive bucket rows and emits their current snapshot when the bucket becomes active; the facade adapter does not buffer discarded deltas. It keeps only the deltas a flush consumed until that flush publishes: if the root commit fails, the adapter restores them, so the next successful flush publishes each pending child change exactly once. The adapter retains a facade only while at least one parent route uses its bucket. When the last route leaves, it drops its strong reference. The graph has already retracted the bucket's rows, but a facade is a Collection: a user transaction can show an optimistic row in it, and a sync commit can be held behind a persisting transaction. Retiring a facade that still shows such rows is legal, not an invariant violation. Those rows have no synced row, so they leave when their transactions settle. A row that the graph sent and never retracted is different. The adapter reads each facade's projected synced rows: the synced rows once every queued sync transaction applies, including a transaction a persisting one holds and a transaction still open. It decides inserts, updates and deletes from them rather than from the visible rows. A projected row left at retirement is a contradictory graph signal: the retirement throws, and the flush aborts and restores every facade it wrote. The live query owns its facades: several parent rows can share one, so a holder's `cleanup()` rejects (code 241). The adapter disposes facades itself, when the live query is cleaned up and when a failed flush removes the facades it created. Ordinary Collections keep their cleanup and restart. A restore undoes only the keys the flush wrote, including writes that a persisting transaction still holds, so a held write cannot land after the rollback. A facade that `resolve()` created before its bucket activated can stay ready and empty after a failed flush; the next flush writes its rows (a recorded limit). An external holder may keep the retired Collection alive, but a later active interval gets a new facade. Inline modes do not create child Collections. Composition is pure. It constructs a new result along changed paths and does not mutate a previously published row or use public routing metadata: ```ts compose( baseRow: BaseRow, includeValues: ReadonlyMap, ): MaterializedRow ``` When a parent result changes, unchanged inline include arrays may receive new object identities. Cross-publication `===` equality for those arrays is not a contract. Their values and prior snapshots must remain correct; a downstream query whose selected result is unchanged must not emit a spurious update. This does not guarantee that a UI component using shallow prop comparison skips a render, nor does it relax the stable public Collection facade contract above. ## Demand plane ### Demand grouping and ownership Demand is derived from data, but it performs asynchronous side effects outside D2: ```text ActiveBucket(bucket, demand parameters) -> group by [source, parameterized child plan] -> current demanded parameter set -> demand adapter -> source loadSubset / release -> source deltas return to D2 inputs ``` The adapter groups demand into shared source work, not one request per bucket: ```ts type DemandPlanId = Brand type DemandSet = readonly [ planId: DemandPlanId, parameters: CanonicalSet, ] ``` One request may serve many buckets according to the compiled demand plan. This does not imply transport sharing between independent subscriptions. The exact-request deduper reuses completed requests and shares in-flight work only when callers supply no abort signal. Independently cancelable requests use separate transports, trading duplicate concurrent fetches for simpler ownership. An adapter may share its own resources, but releasing one owner must not cancel work or remove rows still owned by another. `LoadSubsetOptions.refetch` starts a new acquisition attempt for the same exact demand even when an adapter has completed or cached it. It is operation control, not request data: exact-demand identity, Query cache identity, and unload identity ignore it, and adapters must not expose it as user query metadata. The new attempt still establishes its own acquisition lease. Its settlement and release follow the same rules as every other acquisition: the returned promise waits for authoritative applied rows and rejects if its final logical owner releases it first. A shared transport may be replaced to satisfy the fresh attempt, but peer logical owners remain attached to the replacement. Request data is immutable from submission onward, including the options, expression trees, comparison options, and constant payloads such as Dates, byte arrays, and membership arrays. Core and adapters retain that data without cloning or freezing it. Changed demand needs new request data, not edits to an old constant, even after its first load settles: deduplication and query state may retain its identity. Request data uses stable data properties, not stateful getters. The signal and subscription references do not change, but their lifecycle remains live. Cancellation and release are not data mutations. A truncate replay asks again for each retained demand with that demand's request data, so an adapter sees the same `where` expression it received before the truncate. The immutable-demand boundary matrix checks direct and deferred sync startup, adapter return and asynchronous settlement, cancellation, and release identity. A Collection subscription installs each logical subset owner before it calls the source adapter. Reentrant release during `loadSubset` therefore retires the logical owner at once, but physical release waits until the adapter returns and proves that it established an acquisition. A synchronous `loadSubset` throw is a rejection: the load failed, it keeps its owner, and the owner's release unloads it as it does any failed load. Logical demand retires even when `unloadSubset` fails. Each physical acquisition gets one release attempt, marked before calling adapter or error-listener code. Reentrant and repeated teardown cannot repeat it. Other acquisitions still receive cleanup, and a cleanup failure cannot replace an earlier request failure. An `unloadSubset` that throws is an adapter bug. Core reports its error uncaught, once, from a microtask, and continues as if the release had succeeded: the error is not a load error, so it sets no `lastError`, emits no `loadSubset:error`, fails no replay, window move, or Effect, and no `releaseSnapshot()`, `unsubscribe()`, or `dispose()` call throws or rejects with it. Core retains no retry debt: a broken adapter can leak external resources if it throws before freeing them. Adapters must make their own cleanup reliable. Replay replaces acquisition leases sequentially: detach and release the old lease, then acquire a fresh one only if the logical demand and replay are still current. A load throw leaves the logical demand detached; a later authoritative replay can reacquire it. Neither path restores an already released lease. A sole adapter resource may stop and restart in this gap; adapters must not tear down resources held by another owner. The public replacement barrier remains closed throughout the gap and through failed startup, so visible results do not flicker. Once a new load returns successfully, its lease is active before status callbacks run. Reentrant callbacks therefore see either detached demand, tentative startup, or one active lease, not an old and new lease being transferred together. Request predicates describe acquisition, not row ownership. Releasing a demand does not delete matching rows from either the public snapshot or an unfinished replacement. The source controls retention through actual row writes; a successful authoritative replacement reconciles the retained public snapshot. This rule also applies when another demand overlaps the released predicate or an independent source write happens to match it. Source deletions during replay stay private until successful publication; failure preserves the last complete snapshot. Query filters and routes, not demand retirement, decide which retained source rows belong in a query result. ### Cleanup, restart, and detached waiters Public cleanup first checks admission. `cleanup()` rejects with `CollectionHasSubscribersError` while a live query or Effect reads the Collection; `cleanup({ dependents: true })` first cleans up each such live query, with its own dependents, and disposes each such Effect. A pooled live query counts as one live-query dependent per equality partition. A direct subscription is not refused. `collection.subscribers` lists all three kinds. Internal teardown, such as a retiring Collection facade, bypasses admission. Collection cleanup has two boundaries. Cleanup start is an internal synchronous boundary. It closes restart admission, invalidates the current sync run, detaches its demand, crashes dependent live queries, and marks dependent Effects disposed. Both dependent transitions happen before adapter cleanup settles. This prevents either dependent from using work that belongs to the discarded sync run. The Effect becomes disposed and releases its source subscription synchronously. Its disposal promise may still wait for in-flight handlers; cleanup start does not prove those handlers settled. Cleanup start is not a Collection status or resource-settlement signal. The source Collection keeps its prior public status while adapter cleanup is pending. After adapter cleanup settles and local teardown finishes, the Collection publishes `cleaned-up` and settles its public cleanup promise. A cleanup-start observer therefore cannot infer that provider sessions, transports, or other adapter resources have been released. Restart is not allowed inside an active cleanup callback. `startSyncImmediate()` throws `CollectionStateError` and `preload()` rejects with it before acquiring new work. Nested cleanup does not open a new lifecycle turn. The Collection holds this guard until sync, state, subscriptions, and indexes finish retiring; it releases the guard even if teardown throws. Restart after cleanup completes, including from its final `cleaned-up` status event, remains supported. This avoids letting old teardown clear a replacement graph or its source ownership. Collection cleanup detaches surviving logical demand from the discarded sync run. It aborts that sync run's physical work and rejects its replay barrier, and rejects an unfinished initial preload with `AbortError`. Cleanup never invokes first-ready callbacks; those callbacks belong to the discarded run. Physical acquisitions belong to the sync run that created them; cleanup retires them instead of sending an old release to a replacement adapter. Unlike individual subset releases, a failed sync adapter cleanup callback remains retryable only while that retirement is current; it cannot replace a newer sync run's cleanup callback. Demand requested while the Collection is cleaned up remains detached rather than pretending that a physical acquisition succeeded. When the Collection starts a new sync run, the subscription enters `loadingSubset` before it queues reacquisition, then reacquires all detached demand through a fresh private publication barrier. Settlements from the old sync run cannot publish rows, report errors, or change readiness in the new sync run. This is the direct subscription's restart contract, not automatic recovery of a dependent live query. Public cleanup refuses while a live query reads the source; internal teardown that bypasses admission crashes its live queries. Restarting that source alone does not revive their graphs or publish replacement results; callers must restart or recreate the live query itself. This differs from a source truncate, which keeps the live query active behind its replay publication barrier. An initial sync error also leaves newly requested demand detached, even when the adapter has installed a loader. The error is final for its sync run: `markReady()` in the same sync run does nothing, and only a restart, which begins a new sync run, acquires that demand. Releasing it before the restart creates no physical acquisition or unload. Queued reacquisition must not retry a failed attempt merely because both loading and ready notifications scheduled it. Requests waiting for a loader report one pending promise synchronously through `onLoadSubsetResult`, including requests made during initial error or after cleanup. The callback is not delayed until acquisition: query callers capture its result before the snapshot request returns. This promise waits for the restart's publication barrier, not just adapter return. Failure rejects it with the replay error; release, external abort, unsubscribe, or another cleanup rejects it with `AbortError`. Later transport settlement cannot change that outcome. Cleanup may retain logical demand for the next sync run, but it does not retain the old caller's unfinished wait. Eager collections have no subset reacquisition barrier. After cleanup, their next public batch reconciles retained subscriber rows against the installed state, including deletions for keys that do not return. An empty ready batch also reconciles an empty replacement. On-demand sources cannot infer absence from their partial installed state; their replay barrier owns replacement. A live-query observer's subscription retains no rows across cleanup: the observer's `cleaned-up` snapshot shows none, so every row of the next sync run, including one equal to a row it showed before, reaches it as new. ### Source cancellation and applied settlement The initial-demand contract is: > Every active, satisfiable bucket must be served by a settled current demand > request before initial preload completes. A request may remain in flight after some served buckets become inactive. Those buckets no longer participate in readiness and cannot receive rows through routes that no longer exist. Sharing source work never merges the route rows themselves. The source contract stays abstract: a demand request eventually establishes one coherent baseline and identifies when that baseline is complete. Each request receives an `AbortSignal`. Cancellation is cooperative at this source boundary. An obsolete request cannot satisfy current demand. An abort ends the load for core: from that moment its later success or rejection decides nothing, holds no readiness, and no replay waits for it. A source must honor the signal and install no more rows for the aborted request; rows it still writes apply as ordinary sync writes. A `loadSubset` that throws synchronously is a rejection: a failed load. Buffering, snapshot tokens, shape offsets, Collection transactions, and local indexes are source-specific ways to satisfy it; they are not materializer state. A persisted wrapper is a source at this boundary. Over an opted-in on-demand source, it may establish a request's baseline from durable rows instead of the network. That exact demand must have persisted demand coverage in the current persisted cache generation. The request then settles once its persisted subset restore is visible and the configured wait has passed, or its upstream load has failed. The upstream load continues as work the wrapper owns. Its later rows are ordinary source writes. Releasing the demand, or cleanup, aborts it. Coverage proves completeness when the covered load settled. It proves neither freshness nor broader extent than the exact demand (law 7). A demand without coverage settles only when its upstream load does. A load that a reset made obsolete records no coverage (law 6), a `refetch` load never settles from coverage, and coverage counts only while the restore returns at least as many rows as the covered load left. This settlement applies only under the default single-process coordinator; under a multi-tab coordinator, demand routes through the coordinator and waits for its upstream load. Lazy-demand growth requests only uncovered keys. When contraction or churn would leave fragmented coverage, the controller requests one complete current key union while retaining every established acquisition that still intersects current demand. It releases those acquisitions only after the replacement's applied settlement. A failed or obsolete replacement cannot retire established coverage, and a retry requests the same union again. A partial shrink may keep one intersecting acquisition without replacement. This avoids repeatedly loading the growing prefix while allowing a changing bounded window to return to one acquisition. The controller starts new coverage before it releases an acquisition that no longer intersects current demand. A demand change therefore never leaves its subscription without logical demand partway through, which would end that subscription's truncate replay. Every sync `commit()` returns an applied receipt: `true` when that transaction's writes and events are already visible, or a promise when the transaction is parked in the causal queue. Commit accepts the transaction, and an accepted transaction always applies in commit order. The promise resolves only after the writes and events become visible. It rejects with `AbortError` only if its signal aborted before acceptance or collection cleanup abandons the transaction; a later abort cannot withdraw its rows. A source discards a stale page by checking the signal before `commit()`, and rejects a load whose caller aborted with `AbortError` once its accepted rows apply. The receipt also carries its acceptance moment for handler-facing writes and Collection readiness (`whenSyncAccepted`): readiness counts accepted rows, and a mutation handler waits for acceptance, because a transaction parked behind its own persisting optimistic transaction becomes visible only when that transaction settles. A successful `loadSubset` implementation must await or return every receipt for the transactions that establish its result, so a handler that awaits an on-demand load of its own Collection waits for itself. A source must not add priority merely to make a subset load settle. Truncate retains its queue-bypass contract; if it applies a parked subset transaction as part of that prefix, the subset receipt settles only after the writes are visible. Rejected acquisitions establish no result. Canceled or obsolete acquisitions stop before publishing more request-scoped rows, and the active replay barrier does not wait for them. ### Ordered requests, continuation, and recovery Core constructs cursors only from the leading order column. A direct `requestLimitedSnapshot()` call with a nonempty `minValues` must supply one leading value and at least one order term; multiple boundary values throw before local delivery or source acquisition. For a multi-column query, the provider still receives the complete order, while the continuation predicate selects rows after the leading value. A separate first-column equality request loads that complete tie before local ordering applies the trailing terms. This is a leading-column continuation plus tie expansion, not a composite cursor. Successful settlement proves only that the exact request finished and that its writes were applied. It does not prove source exhaustion or broader coverage. Ordered loading reaches a fixed point from public rows and exact request identity; it must not invent source extent from a requested limit. A local row seen before the first ordered source request proves neither a continuation boundary nor a remote offset. This matters when a zero-sized window admits live source changes before it opens: the first nonzero window must still request its prefix from the start. An optimistic delete or update can hide one provider row per changed key from the local ordered window. A finite source-prefix request must leave room for those rows, including during ordered repair. An optimistic insert or rank move may fill a visible window, but it does not establish a provider offset or cursor boundary. The loader therefore counts source-derived rows for continuation even when optimistic rows make the visible window full. The extra provider limit used for optimistic headroom does not enlarge the local count from which a settled boundary is inferred; a held mutation may become authoritative before that request settles. Starting that request proves nothing until it succeeds; if it rejects, an explicit retry must also start at offset zero without a cursor. The same holds after any later ordered request fails or is canceled: the adapter may already have written only part of its response, so those rows cannot establish a continuation boundary. The next explicit retry starts from the source as one authoritative filtered full-source request. Core cannot know which rows a failed request wrote, and a successful limited request proves neither how many authoritative rows it applied nor source exhaustion. Recovery therefore does not infer a safe finite prefix from local row count or boundary values. This rare error path trades bandwidth for a small, sound rule and keeps the last settled public snapshot visible until recovery succeeds. It also lets multi-column windows revalidate after a non-boundary row leaves. If the provider predicate cannot express the local order relation, such as locale string order, ordinary refinement likewise loads the full source instead of treating boundary equality as an ordered continuation. An asynchronous failure of that full-source acquisition does not start duplicate recovery work. It keeps the logical demand so a later truncate replay can retry one authoritative replacement, and clears the loader's completion marker so an explicit retry of the window can issue the request again. That explicit retry retires and releases the earlier failed acquisition before installing its replacement, so a later truncate replays one logical demand rather than both attempts. The failed read stops being the loader's source demand before its lease is released. A replay that settles during or after a throwing release therefore cannot count the released read as complete source coverage. A successful authoritative replay clears its source-recovery gate, but it does not clear an unrelated failed window operation. A later explicit window move revalidates that physical window before publishing it. A finite page or tie-boundary request can remain in flight when full-source repair starts. Its later success still settles its publication participant, but cannot clear a recorded failure, change the repair's state, or start more finite work. The full-source request owns that repair outcome. Explicit retry releases a failed full-source acquisition once before replacing it. If overlapping finite and full-source requests both fail, retry retires every failed acquisition, even if one release throws. A successful full-source replay repairs only that demand; obsolete failed finite demands still retire on retry. Successful authoritative full-source recovery also retires settled successful page and tie demands. Later replay therefore reacquires the full source without repeating those finite requests. Retirement waits for each original request to settle and for any active replay to finish; it does not cancel unfinished work merely because the full-source request finished first. A failed finite demand that authoritative work superseded also retires then, or at once if it fails after full-source coverage completed: the full-source read covers its partial writes. Other failed finite demands still follow the explicit-retry rule above, and a full-source replay does not retire them. Release callbacks retire ownership before adapter code runs, and reentrant truncate or disposal stops the current retirement pass. No copied rows or additional cursor history are retained. A visible delete or source-order change after an otherwise settled, expressible window uses a narrower authoritative path. So does a sync adapter's `invalidateLoadedSubsets()` report: after a refetch that the adapter started on its own has applied, served pages may have shifted so that a row now sorts between two of them and is in neither result. The adapter reports that instead of keeping each served prefix whole (the `LoadSubsetFn` rule). Core emits `loadedSubsets:invalidated` once per turn, and each ordered loader on the Collection runs this repair. A loader whose coverage is one full-source acquisition, or a `requiresFullSource` plan, ignores the report: the refetch has already re-read that acquisition, and it has no page boundary to repair. A loader whose coverage is still its own first ordered request, of at least `offset + limit` rows with no continuation past it, also ignores the report when its window is inside that page: every window row sorts at or before the `offset + limit`-th source row, and the next source row sorts strictly after it in the leading term. The refetched page holds every row from the start through its end, and any row it left out sorts after that end, so the window is proven. The loader checks the next source row through an installed row, which is evidence only when the page read it, or when the loader read the tie of the page's current end: the refetch re-read that tie, so every row that ties with the end is installed. A row that another request installed proves nothing. Neither does a tie that the loader read for an older end, because a refetch can move the end earlier, so a row the provider never sent can tie with the new end. When the end moved, the loader repairs. (Reading only the new end's tie would also complete the proof, since the refetched page holds every row before the end, and saves one request; the bounded repair is kept because that read cost shipped bytes for one saved request per such refetch.) A window that needs a row past the page, such as one whose page holds rows a joined filter rejects, still repairs. So does any loader while its source holds accepted sync rows back, as behind a persisting optimistic transaction: the visible rows then predate the refetch. Core reissues the ordered prefix from the start with the query predicate, order terms, and `limit = offset + limit`; the transport `offset` is omitted because this acquisition starts at the provider's source prefix. It then reacquires the first-column tie boundary and any required refill before publishing. Only the new repair acquisitions remain. Every acquisition in this chain uses `refetch: true`, so an adapter must revalidate the exact demand instead of reusing completed or in-flight work. A source-order invalidation generation fences the chain. If another qualifying mutation arrives before it finishes, core starts a replacement repair before releasing the publication barrier and before the obsolete chain can issue another tie or refill. That holds for a step that a continuation the chain staged, or a window move that joined work blocked, would resume after the mutation: resuming it starts the replacement instead. A qualifying change that a retirement release delivers after the chain succeeds belongs to the same fence, including a delete of a row in its prefix, such as the boundary row or a tie the prefix holds: the repair keeps that newer debt, the next ordered pass starts its replacement before the barrier releases, and a window move waiting for the chain stays pending until the replacement settles. A repeated prefix or page fixed point ends the repair when no more data is needed; it cannot leave a held repair without pending work. The finite prefix and tie acquisitions they replace retire after the whole replacement chain succeeds. That retirement can make the adapter remove rows only the replaced acquisitions held. The chain holds every row through its boundary, so such a removal is not a source-order change, and the repair keeps its boundary. A loader that discarded it would restart paging from the start of the order with a smaller first page that cannot retire the repair's prefix. Full-source reads can be very expensive, so a repair avoids one whenever a bounded read is available. An invalidation that overlaps unfinished page or tie work after the source has settled once does not force one: the bounded repair starts at once, and that work's later result is ignored for its boundary. Such a prefix also cannot retire the repair's acquisitions: it was issued before the invalidation, so it proves no current coverage. The repair supersedes any such request, so one that settles after the repair finished is released at once, like the acquisitions the repair retires. A request is superseded when authoritative work (a bounded repair or a full-source read) started after it was issued. Its failure is not the loader's own failure and blocks no refills: its partial writes only raise the repair debt to the full-source read that the overlap replaced. A running repair restarts as that read. Otherwise the loader starts it at once, unless its own recorded failure is waiting for an explicit retry. This holds even when the repair finished and a newer invalidation's reaction is still deferred. A request belongs to the ordering epoch it was issued in, even when its own synchronous delivery invalidates ordering before it returns, so its result and continuation do not count as current coverage. An authoritative read also stops waiting on the superseded request: a repair that answers at once continues without it. A superseded failure cannot drop the repair's staged continuation, and a repair-chain step does not release failed acquisitions; the explicit retry that resumed it does. A repair request belongs to the chain it was issued in. When a truncate replay finishes a chain whose prefix is still in flight and a newer invalidation starts the replacement, that prefix settles outside the replacement's chain, and the replacement retires it. A repair chain is superseded once a newer repair or full-source read starts. A request of the superseded chain proves no current coverage: it issues no tie or refill, and its failure is not the loader's own. A chain that finished still owns a request that settles late. Once a full-source read covers the source, any finite or repair request that settles later is retired at once. Every lease release pass, including the explicit and backoff retries' release of failed acquisitions, stops when a release resets the cursor or disposes the loader. The replay that follows owns the leases the pass did not reach. A release can deliver deletes that run the graph synchronously, so ordered refinement stays blocked until the pass unwinds. No second pass can then reach the same leases within one generation, so a lease missing from its map during a pass is an invariant violation, not a case to skip. The loader's authoritative work is a small state machine. Its work slot is idle, a repair chain, or a full-source read that is held, complete, or failed; its repair debt is none, bounded, or full. Every move names the states it may leave, and the loader throws `OrderedLoaderInvariantError` for any other move: | Move | From | To | | ---------------------------------------------- | ---------------------- | -------------------- | | Start a bounded repair or a full-source read | idle | repair, held | | A chain finishes, is replaced, or is abandoned | repair | idle | | A chain request fails as the loader's own | repair | idle | | A full-source read succeeds | held, complete | complete (debt none) | | A truncate replay settles | held, failed, complete | complete (debt none) | | A full-source read fails as the loader's own | held | failed | | A cursor reset | complete | held | | An explicit or backoff retry | failed (or idle) | idle | Three work and debt pairs are coherent only one way: complete coverage owes no debt, a repair runs only to pay a debt, and a failed full-source read owes a full-source read. So full coverage absorbs what would otherwise raise debt: a failure or an invalidation under complete coverage owes nothing, and a full-source transport that fails after a replay reacquired its demand leaves the coverage and its lease in place. A replay that settles is the full-source read's success: it pays the debt and retires the settled finite leases and the superseded failed leases, as a full-source success does. A chain request owns authority only while its chain runs: when a replay finished the chain first, the request's later failure is superseded by that replay. Each request carries one stamp from issue: the ordered-load generation, the authority generation, the ordering generation, and its role (finite, repair, or full-source). A request is dead after a cursor reset. A repair-chain request is stale once a newer authoritative start supersedes it; a finite request is stale once the ordering epoch moves; a full-source read stays current. A request that fails reports its own error. When the full-source read that a superseded failure starts throws while it starts, that read is the loader's own failure and is reported on its own; the superseded request still reports its error. Dispose resets the cursor, so a disposed loader's requests are dead and it issues and releases nothing more. A truncate replay owns every request the loader issued before the truncate: their results and failures change no loader state beyond the debt a failure raises, and the replay settles their demands. The loader has no request timeout. A request the provider never answers keeps its work slot, so an authoritative read that never settles holds its repair or recovery until the subscription ends; recovery from a provider that stops answering belongs to the adapter or the subscription lifecycle. A request issued before a cursor reset that later fails still raises the repair debt, but it cannot drop work staged after the reset. A failed or canceled request, an invalidation before any request has settled, or a boundary that cannot be expressed still requires the full-source recovery above. A bounded repair failure likewise upgrades its next attempt to full-source recovery because partial writes cannot establish finite coverage. “Bounded prefix” describes the adapter acquisition, not the subscription's immediate local replay. `requestSnapshot()` composes the subscription and request predicates for both legs. For an on-demand source its local snapshot is predicate-only: it may deliver every matching row already installed in the Collection without applying the request's order or limit, and the repair chain relies on that delivery. An eager source's installed rows are the whole source, so an indexed request with one order term reads only the first `limit` matching local rows; a row in the source's first `limit` is also in the local first `limit`. A second order term keeps the full local read, because an index on the first term cannot order its ties. A filter on a LEFT-joined source also keeps the full local read: it drops rows only after the source's `limit`, so a full window after a bounded read would not prove that it holds the first eligible rows. A row that entered the graph through an earlier live update could fill the window in place of an eligible row the bounded read never sent. When every request in a bounded prefix repair (an eager source, an index, one order term, no joined filter) returns literal `true`, the repair settles synchronously, so its tie and refill steps finish inside the same graph run and a window move that consumes the repair still publishes once. A full-source repair keeps asynchronous settlement; its scheduling is not part of the initial synchronous cut. An initial full-source load follows that cut as described under initial readiness. The ordered graph's top-K operator owns the local result window. Provider transfer remains bounded by `orderBy` and `limit`; local delivery cardinality is a separate observation. Bounded repair is available only when a provider prefix is sufficient for the local plan. Core requires full-source recovery for an indirect order expression, a non-root ordered source, any join other than a LEFT join, a residual or cross-alias predicate, a functional predicate, grouping, `having`, functional `having`, or `distinct`. These cases can discard or reorder an otherwise valid provider prefix even when a cursor value itself is expressible. A right, full or outer join also adds joined rows whose root match may sit past the prefix; within it they would look unmatched and could take the window. A single direct LEFT join to a Collection is an exception for a cross-alias filter when every order term comes from the root source. Its joined filter can remove root rows, but cannot make a later root precede an earlier root. Before loading another root page, core waits for the current lazy join demand and any joined subset load to settle, then reruns the graph. A joined load can settle without delivering a row, so the joined source becoming ready schedules that run itself. Joined-side changes also recheck ordered demand so a removed match can refill a short window. An empty joined replay still releases held Effect callbacks. A finite indexed window is proven only up to the loader's settled boundary. Live root rows past the bounded prefix still enter the graph, so a window row after that boundary does not count toward a full window: an earlier eligible row may be unsent. The window then asks for the next page from the boundary, as an underfilled window does, until every window row is at or before it. This holds however a row leaves the window: a joined-side delete or insert, a join-key update, or a root change, and for Collection and Effect consumers. The adapter receives ordinary finite root requests; this contract does not imply that it can evaluate the relation filter remotely. Finite continuation still requires an order index; an underfilled unindexed prefix grows. Each growth reads a prefix twice as long as the last one, so every read stays bounded and none reads the full source. The prefix grows while growing still adds rows to the source. When the last growth added none, the source holds no more rows and the window stays short. A custom string comparator is a local ordering contract. If any resolved source order term uses `stringSort: 'custom'`, bounded and unbounded queries use one authoritative filtered full-source request for the first acquisition and every replacement: the source predicate remains, while `orderBy`, `limit`, `offset`, and cursor are omitted. Core then sorts locally and uses ascending public-key order for comparator-equal rows, including when the primary term is descending. It never interprets the comparator function as a stable backend collation identity or as evidence that a provider can paginate that order. Custom cursor pagination therefore remains unsupported. Implicit string collation is a query-level default inherited from the first source Collection. It applies even when an order term references a later source whose own Collection default differs. Provider hint admission resolves every term against that same query default before any source receives `orderBy`. A `defaultStringCollation` override on the output live-query Collection changes that Collection's downstream default only; it does not change the source query's ordering semantics or provider hint admission. The callback applies only to string/string comparisons. Null placement, non-string ordering, and primary direction keep their existing semantics. Runtime query identity and local index reuse use the exact callback reference; two extensionally equivalent functions are distinct. Callers must keep the callback deterministic and immutable for its lifetime, because an index cannot detect a mutable closure's semantic change. Ordinary `gt`/`gte`/`lt`/`lte` predicates retain their lexical evaluator semantics and cannot use a custom string index for range traversal. Stable cross-runtime collation identifiers, provider capability negotiation, comparator exception policy, and backend collation fidelity are outside this contract. An ordered request cannot start another ordered request through its own synchronous writes. A graph turn that those writes run inside the request may ask for more data; the loader ignores that call, stages its continuation, and resumes it once the request returns, because that turn has already processed the writes and no later turn will ask again. A continuation staged with input the graph has not yet processed waits for the graph turn that processes it. If the adapter then throws, graph callbacks scheduled by those writes still belong to the failed window operation and cannot retry it. A public `setWindow()` call made from inside that synchronous operation throws `SetWindowReentrancyError`; it must not claim that a nested window settled after the loader suppressed its work. The guard reads the loader's existing synchronous request state for initial and later refinement requests, including requests after an asynchronous page. It also rejects window changes during graph publication, before mutating top-K. A synchronous result callback is provisional until the whole snapshot request returns: a later local read or publication throw fails and retires that acquisition instead of letting its queued success erase the failure. A later explicit window operation has a new window-operation generation and may retry from the safe source boundary. The ordered loader retains one settled loading boundary, independently of live rows sent to D2. It derives invalidation from the existing contribution rows rather than tracking a second largest-row cursor. New keys may reopen refinement, while duplicate delivery and order-equal updates do not. After a successful request, only a delete or a change that compares differently on the first provider order term for a row previously contributed to D2 starts the authoritative repair path. An unseen/new row clears continuation state so the graph may request more work, but does not invalidate settled prefix authority; a later-order-term-only update also does not take this repair path. After a successful finite acquisition, it reads at most the requested limit within that request's filtered, ordered range. That range's last available row can advance the boundary; an unrelated live outlier cannot advance it merely by entering D2. This relies on the adapter fulfilling the exact ordered request, not just resolving after an arbitrary partial write. An empty range does not invent a boundary or prove source exhaustion. If an explicit window operation consumes a staged boundary continuation and that continuation has no boundary row, it starts no acquisition. The operation must continue normal demand selection for the enlarged window; consuming the empty continuation does not settle that window operation. If joined demand blocks an explicit retry, the loader retains its window generation and repair intent until it can select the request. A new failure or cursor reset discards that blocked intent. For no-index prefix loading, an unrelated new key does not reacquire an already full window. An explicit window move, an underfilled window, or a settled prefix smaller than a window widened during that request still requires acquisition. A full local window alone does not prove that the provider fulfilled a concurrent window change. A successful larger prefix retires settled smaller prefix acquisitions from the same ordered source plan, after the replacement has applied. It does not retire cursor suffixes, ties, unfinished work, or another subscription's leases. Adapter eviction must still preserve rows owned by the replacement or peers. A lease release reports an adapter's throwing `unloadSubset` uncaught, as described for subscription cleanup, and does not throw. The ordered loader also reports any error a release callback still throws uncaught, from a microtask, and continues as if the release had succeeded. The error changes no load result, window move, publication, or loader state: the successful replacement finishes its boundary and continuation bookkeeping and publishes. Automatic full-source repair after an established window fails can retry twice, after 250 ms and 500 ms. Every retry releases failed acquisitions before starting the replacement. It uses the same publication barrier; stale rows stay public and the last error stays observable if the budget is exhausted. Initial loads and explicit window failures do not auto-retry. Cleanup, truncate, and explicit retry supersede queued repair work. A successful repair resets the budget. An explicit window move counts current rows at or before that boundary in the requested prefix. It acquires only the missing portion, with both cursor and offset derived from that confirmed range, not from all observed rows. These reads reuse the Collection's indexed snapshot code; they retain no page list or second row index. Transfer checks and local-read work are separate costs: counting a long prefix can still revisit its rows. Boundary-read failures use the same authoritative recovery path as failed acquisitions. Deletes and source-order changes invalidate finite coverage as described below. Cleanup and truncate discard the boundary; replay establishes an authoritative source replacement instead of reviving a stale cursor. ### Atomic window publication An initial ordered load or imperative window move includes every page, tie-boundary request, and forward refill needed to reach its fixed point. Its preload or window promise cannot settle before that chain, and a failure in any required step belongs to the same operation. Rows may enter the private D2 result while the chain runs, but the public Collection publishes the completed window once. If refinement fails, the operation rejects and leaves the last settled public snapshot visible. The private source and D2 state may already have advanced, so core does not try to reconstruct the old window over that new state. A later successful retry publishes the coherent replacement. A superseding window also waits for older source work that still gates publication; it does not report success until its own chosen window is visible. A joined demand already pending when the move begins is part of this operation if it blocks the ordered continuation. Retiring that demand releases its participant; a later failure from the obsolete request cannot fail the move. The resumed root page and refill still belong to the operation. Window controllers treat `getWindow()` as settled state, not the current lease request. An overlapping preload joins its lease's pending window promise rather than replacing it with the smaller committed page count. Lease release may also settle asynchronously; completion, not the release call, establishes its window. Partial window options inherit omitted fields from the active requested window, or from the last settled window when no move is active. Collection cleanup rejects a pending window operation with `AbortError`; it cannot report success after discarding the graph and requested window. That error belongs to the operation even if cleanup precedes registration of its waiter. Cleanup does not retroactively cancel an already completed operation. Window-operation generations stay monotonic across cleanup and restart, so a late rejection from an abandoned sync run cannot reset the replacement sync run's requested window. A window move started during an active source replay waits for that replay and applies only after its replacement is complete. A failed replay rejects the move without advancing the reported window. A replay that a newer truncate or the retirement of its last demand discards did not fail, even though its completion rejects with `AbortError`: the move waits for any replay that still holds publication and then applies. If a replay it waits on fails, the move rejects with that replay's own error, its earliest failure that still decides it (maintainer decision). When a replay it waits on is discarded and the query's or a source's sync run changed meanwhile, as a reload changes it, the move rejects with that replay's `AbortError`. The move compares the sync runs only when a replay it waits on rejects, so a reload of another source whose replay it does not wait on leaves the move to apply. Replay completion callbacks carry their sync-run identity and become no-ops after cleanup or restart. Cleanup rejects the replay barrier, and therefore every window move waiting on it, with `AbortError`; no waiter may outlive the discarded subscription. Subscription-owned Promise observers carry the Collection's sync-run generation. Cleanup invalidates that generation before adapter teardown, so an obsolete replay cannot publish its private rows, report a late error, or emit a late `ready` transition even when the transport ignores cancellation. Ordinary source mutations stay synchronous except while an initial ordered load, imperative window move, or asynchronous repair of invalid finite source coverage owns this publication barrier. A visible delete or a change to a visible row's source-order value can invalidate a provider prefix because a hidden row may now belong in the window. After a settled request, that repair reacquires the bounded ordered prefix and its tie/refill chain; unsafe overlap, inexpressible ordering, and request failure retain the full-source fallback. Both paths keep the last complete public snapshot until they settle. An update that compares equal under the source order does not broaden demand. Mutations that arrive during a barrier join the private state and publish with the completed replacement; a failed operation keeps them private until retry or restart. Queued ordered-repair startup joins this barrier before invoking the adapter, so a synchronous throw cannot publish a partial replacement merely because it returned no acquisition promise. The queued task belongs to the loader that scheduled it, not a replacement created after cleanup. An optimistic overlay departure is not an authoritative loss of provider coverage. When an optimistic update removes query membership, or an optimistic delete removes a row from an ordered window, live-query Collections and Effects publish that departure immediately, even when provider work that could refill the window is pending. The intermediate result may be shorter than the requested window. The loader starts any needed acquisition while the optimistic transaction persists. A provider row stays private while its accepted sync transaction is queued behind that mutation; application may publish the recomputed window before the receipt promise resolves. If the provider answers later, acknowledgement can retain the short window until application. Rollback removes this overlay and recomputes from the then-current applied base and any remaining overlays; it restores the prior window only when that base is unchanged and no other overlay remains. This exception does not define a rank move or let a partially applied provider page or authoritative repair bypass the complete-window barrier. The loader tracks each sequential request as a bounded participant, not every recursive suffix of a long refinement chain. Effects use a separate callback-publication gate because they do not publish through a Collection. During an authoritative ordered repair, source changes continue to advance the private D2 graph and accumulate a net delta, but `onBatch`/`onEnter`/`onUpdate`/`onExit` callbacks retain the last complete result. Prefix, tie, and refill promises join one continuous gate; the final successful participant schedules one flush of the accumulated delta. A synchronous adapter result still contributes the loader's wrapped repair participant, except in a synchronous bounded prefix repair: that repair finishes inside the graph run that publishes its window, so no callback can observe a partial window and no participant is needed. At flush, equal insert/delete counts are classified against the last callback-visible membership and value: absent-to-absent produces no event, while present-to-present produces an update only when the value changed. If truncate replay aborts an obsolete repair participant, its replacement completion inherits the same callback hold; the abort alone does not freeze or dispose the still-live Effect. Any unhandled participant failure never exposes the private intermediate state: source-error handling disposes the Effect and clears the retained delta and participants. With `skipInitial`, an asynchronous initial ordered chain also stays behind the gate until all initial participants settle, so its rows do not become later `enter` callbacks. A joined-filter window also waits for joined demand before ending initial callback suppression, even if that source Collection already reports ready. Other ordinary initial/refinement requests keep their existing callback timing. ### Replay participants and failure A truncate replay is one publication barrier. Every acquisition started while that replay is active, including ordered full-source recovery, belongs to the barrier. Success publishes only after all current acquisitions settle. A released demand stops participating even if its canceled transport promise never settles. A newer truncate aborts prior acquisitions, and an abort ends the load: publication and readiness stop waiting for it at once, whether the source settles it promptly or late, and its later outcome decides nothing. The source must not install rows after observing cancellation. Settled historical attempts are discarded. Replacing an acquisition does not release its logical owner, but an owner whose current acquisition was aborted owes the replay nothing. Releasing the owner removes its work from readiness. Work started inside the current replay attempt holds publication until it settles or aborts. Core installs each tentative acquisition and binds it to the current replay attempt before calling adapter code. A reentrant release or newer truncate can therefore see and retire the exact work it supersedes; work returned after that reentrancy cannot attach itself to a newer attempt. Once reentrancy supersedes an attempt, core starts none of that attempt's remaining demands. A demand that releases itself during adapter or status callbacks cannot join readiness or poison the replay with a later synchronous failure. Successful replacement publication happens before the subscription emits `ready`. Cleanup runs every ownership step even when replacement publication throws. Subscriber errors raised by an asynchronous replacement do not turn source success into replay failure: core finishes its internal state and surfaces the exact callback error in a host microtask. Status callbacks may synchronously change demand. Generic and specific status delivery capture the transition revision and stop before a later listener when reentry supersedes it, including an ABA transition back to the same status label. Subscription teardown is a one-shot logical transition: it stops the listener set already being walked, emits no later status, and removes subscriber ownership once. A later `unsubscribe()` is a no-op, including after an acquisition release failed. Failure keeps the last complete result visible and partly replayed source state private for both direct subscribers and query graphs. Ordinary source deltas or snapshot requests do not reopen that gate because they cannot prove the source complete; only a later successful truncate replay provides the authoritative replacement. Replay failure is scoped to the logical demand that failed. If that demand retires, its failure cannot poison a successful replacement for the remaining demand. A demand change is one step: the demand it starts and the demand it retires count together, so a change that asks for data again does not retire the last logical demand, and its new demand joins the open replay. If the last logical demand retires, the now-unreachable source replay rejects its completion with `AbortError` and stops gating the shared graph; unrelated parent or sibling changes may then publish. A direct subscriber's replay whose current attempt has not failed instead publishes the rows it holds at that retirement and resolves its completion, because nothing can complete it and keeping the old snapshot would leave values the source replaced or deleted. A replay whose current attempt failed keeps the last public snapshot, even after its failed owner was released; a later request reconciles a row it still shows. If release publication or adapter unload synchronously acquires new demand, core checks completion after that callback: the new demand joins the private replacement, while the retired transport can no longer gate it. A genuine replay failure is normalized once by the subscription. The `loadSubset:error` event, `lastSubsetError`, and any window move waiting on that replay expose the same `Error` object. A live query's subscriptions to one collection, such as a join and an include of the same source, share one publication barrier. None of their replays publishes, and none whose last demand retired closes, while another of them still has replay work, so demand that any of them starts meanwhile joins its own open replay. While a replay of any of them still holds publication, a lazy demand of any of them that fails after initial-query readiness fails its own subscription's replacement too. This includes a subscription that held no demand at truncate. Such a subscription joins the barrier when its first acquisition starts, before adapter code can reenter; its current work still holds publication after the sibling settles. Settling a sibling's replay or retiring its failed demand cannot publish the window without this demand's rows. It holds until every demand load that joins it loads, a later replay of the source succeeds, or the subscription's last demand retires. The query retries the failed demand at once and the retry joins the hold, so the hold ends only after the failed demand's rows load. ### Mutation boundaries and initial readiness A transaction `mutationFn` must not start or await collection or live-query preloads. User persistence owns the causal queue while that function runs, so a preload that waits for a queued sync commit can wait on the mutation that is waiting on the preload. Use an adapter's documented mutation acknowledgement helper instead; it must confirm the optimistic write without starting new collection demand. This project uses a single graph-run order rather than multi-dimensional timely-dataflow frontiers. Do not introduce a general timestamp or frontier framework unless a source contract proves that the sync-run generation and source up-to-date protocol cannot express its ordering. **Initial readiness:** preload is complete when every demand currently reachable from the initial query graph is covered by a settled request, and every eager source Collection is ready. An on-demand source Collection's own readiness does not gate it. Demand that is no longer reachable does not block completion. An empty outer relation has no child demand, but its root demand must still settle. Later readiness transitions follow the existing Collection contract until an executable test defines another public behavior. The simple rule would wait for every source Collection to be ready. A progressive Electric source shows why that rule is wrong. Its "first query loads immediately (on-demand)" while it "syncs full dataset in background" ([TanStack DB 0.5](https://tanstack.com/blog/tanstack-db-0.5-query-driven-sync)). Its Collection stays `loading` until that full sync completes, so the simple rule holds a query whose rows are already present. The distinction is the evidence that a query's rows are present. An eager source loads everything it holds without per-query requests, so its Collection readiness is the only evidence. An on-demand source serves each query through subset requests, so a settled request is the evidence. Its Collection readiness says something about the rest of its data, not about this query. The adapter owns the other half of that bargain: a successful load settles only after its rows are visible (law 7). A load that installs no rows itself, such as a progressive load during a must-refetch, must settle when the sync that supplies the rows does. Every consumer of initial readiness uses the same rule: `sourcePermitsInitialReadiness()` in `utils.ts` holds it. Framework adapters derive `isReady` and Suspense release from live-query readiness, so they release together. A read that waits for readiness, such as `queryOnce`, resolves at the same point. Every Effect passes its gate there too, with or without `skipInitial`: its initial subset loads and lazy demand hold the gate until they settle. `skipInitial` only decides whether the initial rows are delivered, so with it the gate also ends initial suppression. An obsolete or aborted initial load releases its hold, and a truncate replacement takes the hold over. Neither can keep the gate open or close it for good. A real load failure before the gate disposes the Effect; after the gate, a source problem is health tree detail, delivered through `onHealthChange`. Two limits remain. Before an on-demand source completes a full sync, a ready query can hold rows that the source does not yet update. The source contract, not this law, decides when it delivers those changes; Electric applies them at its atomic swap. A source that resumes from complete local rows settles its loads at once, so its queries are ready on those rows before the network responds. A persisted wrapper applies the same rule to a single demand that has persisted demand coverage: after its wait, that demand settles on its restored rows. Initial readiness, ordered-window publication, and later demand therefore proceed offline on covered demands without a separate rule. A demand built from other rows' values, such as a cursor page, is covered only when the restored rows produce the same demand key. **Deferred acquisition:** a live-query Collection reads local memory and starts no provider work on its own. Until it has a subscriber or a preload in its current sync run, its own subscriptions to source Collections defer acquisition: they start no idle source Collection's sync run and no acquisition attempt. Their demand stays active and reads the rows the source Collections already hold. A framework render or `startSync: true` may start the live-query Collection's sync run; that is not a subscriber. An ordered window over an on-demand source Collection stays unpublished until its acquisitions settle, by atomic window publication; an unordered query publishes the rows its source Collections hold as a partial result. Without a subscriber or preload, the live-query Collection is ready only when it needs no provider work: every source Collection is eager and already started, or is a live-query Collection whose acquisitions settled. The first subscriber or preload resumes deferred acquisition. A deferring subscriber may start an inner live-query Collection, which reads local memory, but not a base source Collection, so the rule is transitive. Cleanup ends the sync run, and a restarted live-query Collection defers again. A preload answered by a `DbClient` stream still counts. A pooled live query applies the same rule to its partition's source subscription: a view built during a render defers it until a view has a subscriber or a preload. An initial ordered load also has a synchronous observation cut. The initial load is the request chain needed to complete the first ordered window. It may start with an ordinary ordered request or, for a plan that requires the full source, a filtered full-source request. A full-source fallback after an inexpressible boundary is still part of that initial chain while the window remains unpublished. The cut holds when the first subscriber or preload starts the live-query Collection's sync run, or when its source Collections are eager. A sync run that started earlier resumes deferred on-demand acquisition through the subscription's restart path, which settles asynchronously. When every acquisition needed for its completed initial window returns literal `true` after its establishing applied receipts are visible, core drains the remaining synchronous ordered continuations and graph work before the initiating call stack returns. The live-query Collection rows and initial-query readiness are observable at that cut. A Promise result keeps that acquisition asynchronous. This cut does not apply to explicit window moves, full-source work started after the initial window publishes, repair other than a bounded prefix repair, truncate replay, or framework render timing, and it proves neither source exhaustion nor broader source coverage. The first-publication boundary limits this guarantee; it does not require later full-source work to wait for a microtask. A later literal `true` full-source fallback without ordering-repair debt may finish in the current graph turn. If any source subscriber adds input to the graph during a synchronous ordered continuation, core returns to graph work before deciding whether that ordered source needs another acquisition. Input progress is graph-wide: a quiet ordered source cannot drain its continuation around pending sibling input. Pending demand does not hide the parent row. An active empty bucket gives it the current canonical bucket value, and available partial source rows produce the current partial materialization when the source supports progressive delivery. Later source rows enter D2 as ordinary deltas and recompute the parent. “Fully composed” means that every include field has its canonical value for the graph's current input state; it does not mean that asynchronous demand has settled. ## Coherent publication D2 runs until the whole materialization graph has no pending synchronous work for its currently available inputs. Only fully materialized canonical root deltas cross into the public Collection. For each scheduled graph turn: 1. enqueue all currently committed input deltas into their D2 inputs; 2. run D2 until it has no pending synchronous work; 3. consolidate the already canonical final-output deltas; 4. install child-facade state through normal Collection transactions while deferring their subscriber delivery; 5. apply direct root insert, update, and delete writes through one normal Collection transaction; 6. release the deferred child-facade events after every synchronous read can see the complete root and facade state; 7. allow dependent live-query graphs to run through the existing transaction-scoped scheduler. The Collection boundary performs no identity reconciliation, routing, materialization, or multiplicity interpretation. The canonical root relation has already done that work. The public Collection is an output, never scratch state. Placeholder rows, in-place include repair, and forced secondary events are forbidden. Classify root deltas against authoritative membership, including earlier queued sync writes, not the optimistic public view. An optimistic delete must not turn a balanced graph update into an authoritative delete. This does not bypass the normal sync queue or publish part of a graph-output transaction early. When a publication hold spans several graph steps, compare the final row and position with the last subscriber-visible snapshot. If the value returns to its prior value at a new position, publish a layout notification even when private intermediate rows had different values. Build queued membership lazily on the first balanced delta in an output flush, preserving committed last-write and truncate semantics. Insert-only flushes do not scan the queue, and balanced rows share that flush's lookup. At the Collection boundary, optimistic mutations own whole validated row snapshots, including fields they did not change and insert schema defaults. Do not merge newer synced fields into those snapshots: that could publish a combination neither the mutation nor the server created. This applies to both ordinary sync and truncate. The mutation payload stays unchanged as well. Active snapshots are selected in transaction order. A transaction's optimistic state drops when its mutation function settles. Sync transactions accepted while it persisted are held and publish in the same update as that drop. A completed row is held only while an accepted, queued sync transaction touches its key; nothing else retains it. A later snapshot may contain values seen from an earlier sibling; rolling back that sibling does not rewrite the later snapshot. Sync publication compares actual previous and next visible rows, not just mutation identities. Truncate replay derives events and reads from the same snapshot overlay, without merging in its new authoritative fields. Installed state, synchronous reads, change-event payloads, and downstream queries must all observe the same fully materialized commit. The facade adapter may defer event delivery across its Collection transactions, but it must not defer state or index installation. Routing and identity remain inside D2. A failed flush is atomic. If a facade write or the root commit fails, every facade the flush wrote returns to its rows, order, and key mapping from before the flush, the facades it created are disposed, and no facade publishes an event or a layout revision. The child deltas that flush consumed stay pending with the builder's pending root rows, so the next successful flush publishes each of them exactly once. No graph output reaches the adapter between a flush and its rollback, because the flush runs inside the graph run; a rollback that finds new pending deltas is an invariant violation. It restores the facades and discards the deferred events first, so the facades keep delivering events, and then throws. A rollback after the adapter is cleaned up does nothing. ## External boundaries ### Query-db ownership Row ownership in `@tanstack/query-db-collection` is separate. Eager retention, active query acquisition, and persisted retention are distinct owner tokens. The live-query graph publishes coherent rows but does not own query-db cache or listener lifetime. For a static on-demand Query key, an active, applied full-source Query may own a later narrower acquisition. Query DB pins that acquisition to the full Query until its final release, including after the original full-source demand ends. An incomplete or failed full-source request does not establish this coverage; an explicit subset refetch retains its exact Query key. Function-built keys retain their caller-defined cache identity. The initial full fetch keeps its acquisition's signal in request metadata. When narrower acquisitions share its Query, later fetches give `queryFn` the Query fetch signal in `meta.loadSubsetOptions.signal`: releasing the original full acquisition cannot abort work still owned by those narrower acquisitions. ### Pooled live queries A framework adapter may serve a live query from an equality partition instead of this graph. That applies only when the query reads one eager source Collection, its `where` has at least one `eq(field, literal)` conjunct, every other conjunct reads only that row's own fields, any `orderBy` reads only that row's own fields without a custom string comparator, and it has no other clause, `limit`, `offset`, `DbClient`, or Suspense key. Such a pooled live query reads the partition group for its `eq` literal tuple and filters it by its remaining conjuncts with the compiler's evaluator. Queries with different orders use different partitions, whose groups sort rows with the compiler's comparator and break ties by key (`packages/db/src/query/pooled-live-query.ts`). It builds its live-query Collection only when the application reads it. A pooled live query must publish what its live-query Collection would: the same rows, in its order or else key order, with the same values and status. Its partition is one live-query dependent of the source: cleanup with dependents ends it like a live-query Collection, `cleaned-up` with no rows, and cleaning up a view releases an idle partition at once. Internal teardown that starts source cleanup under it crashes it the same way and keeps its last rows. The pooled live query oracle compares the two over generated histories. ### Physical planning and work Correct relation state does not prove efficient work. When an applicable index exists, irrelevant correlated rows must not cause scans of unrelated rows or activate unrelated downstream routes. Relation rows, indexed keys, active demands, materialization cells, and public facades are the relevant space units. Queries without includes retain their original pipeline unless a joined custom-key query needs contributor reduction. Inline materialization must not create recursive Collection machinery. A windowed query has an `orderBy`; its window is the ordered view, by offset and limit, that `setWindow` moves. A negative limit throws synchronously at `.limit()` or `setWindow()` admission, before changing the authored query, window or demand (maintainer decision, 2026-11-02). Zero, including -0, stays valid. Negative offsets retain their absolute-rank selection semantics. A `findOne` is bounded by a top-K too: it keeps its compiled limit (1, or 0 for `.limit(0).findOne()`), and `setWindow` throws `SetWindowFindOneLimitError` for any other limit. An ordered `findOne` has a movable window. An unordered `findOne` is a fixed one-row result that accepts only its compiled limit at offset 0; a nonzero offset throws `SetWindowRequiresOrderByError`, as `.offset()` without `orderBy` does. A query that is neither windowed nor an ordered `findOne` rejects offset moves, and one that is neither windowed nor a `findOne` rejects `setWindow`. Only the query's own top-K answers `setWindow`; a top-K inside a `from` or joined subquery or an include child never does. A windowed query or a `findOne` adds route relations for the includes in its own `select` only from the rows it shows, so a row it drops activates no bucket and creates no demand. A static filter creates static child demand before the query runs only when no enclosing query is windowed or a `findOne`, and either the query is neither windowed nor a `findOne`, or it is a `findOne` that shows one row for one named correlation value, with no `offset`, and offset 0 at start if a window was requested before start. Within one graph run, an index-backed top-K can briefly hold a row outside the rows it shows once its ordered acquisition completes, and that row's children may load (a recorded limit, #2151; the public snapshot stays correct). Includes inherited from a `from` subquery and inputs to `fn.select` still route from every row before the top-K (a recorded limit). ## Normative laws 1. **Alpha-renaming:** changing an accepted alias to another unused name in a structured plan cannot change an explicitly projected result when references are renamed with their binding. The same equivalence applies to a functional callback only when it is alias-equivariant. A captured reference retains its source even when a nested query shadows its alias. An implicit namespaced result and a functional callback keep the user's aliases as public field names. Aliases must be unique within one lexical scope. The branches of one `unionAll()` share one alias namespace. A builder's source declaration may be reused in sibling includes, but an include cannot reuse it across an ancestor/descendant scope boundary; the descendant starts from a fresh `new Query().from()`. A join operand captured outside the current and ancestor scopes is rejected. 2. **Contribution conservation:** a public row exists exactly when its reduced supporting weight and collision policy produce one. 3. **Batch partition:** equivalent valid split and atomic deliveries converge. 4. **Route relation:** current route rows joined with current bucket values equal current materialization-cell values. 5. **Total materialization:** every active inline cell has exactly one value, including its mode's empty value when its bucket has no rows. 6. **Stale demand:** an obsolete graph cannot settle current readiness, and an obsolete acquisition cannot satisfy current demand. A conforming source cannot publish its request-scoped rows after cancellation. 7. **Applied settlement:** a successful subset load settles only after its establishing sync transactions are visible; a source must not add queue priority merely to force the load to settle. Settlement proves no broader source extent than the exact request. A persisted subset restore of a demand with persisted demand coverage is an establishing transaction for that demand. 8. **Nested propagation:** every materialized relation consumes the fully materialized output relation of its children. 9. **Publication:** reads, events, and downstream queries observe the same complete graph result. A truncate replacement stays private until all work started by its active replay demands settles; failure keeps the prior public result and later partial source changes private until an authoritative replay succeeds. A failed replay with no remaining logical demand cannot gate other graph work. A lazy load that fails after initial-query readiness settles its demand: graph output that waited for that load publishes at once, and the failure shows as a request error (law 16). A demand change counts as one step, so a change that asks for data again leaves the replay its new demand. 10. **Initial demand:** preload completes when every initially reachable demand is covered and every eager source is ready; obsolete demand does not block it, and an on-demand source's own Collection readiness does not gate it. Lazy-demand growth loads only uncovered keys. A fragmented churn replacement retires prior coverage only after its union applies; failure preserves it. 11. **Ownership:** a query-db row exists exactly while an explicit owner remains. 12. **Work:** irrelevant rows do not cause unrelated scans or activate unrelated routes when an applicable index exists. 13. **Space:** state scales with retained D2 relation/index rows, active demands, materialization cells, visible rows, the current private replay state, and required Collection facades—not with settled historical replay attempts or raw delta history. 14. **Cleanup:** public cleanup rejects while a live query or Effect reads the Collection, and with `{ dependents: true }` it ends those dependents first; a direct subscription survives cleanup. Cleanup start synchronously closes restart admission, crashes any dependent live query left by internal teardown, and marks dependent Effects disposed. The source Collection keeps its prior status until adapter cleanup settles; only then does it publish `cleaned-up` and settle its cleanup promise. 15. **Deferred acquisition:** a live-query Collection or pooled live query starts no source Collection's sync run and no acquisition attempt until it has a subscriber that asks for data, or a preload in its current sync run; before that it reads only rows its sources already hold. A subscriber that survives cleanup still asks for data. Another consumer starting a source, or a source truncate, resumes nothing. A read that waits for readiness counts as a preload. A deferring live query keeps its sources from garbage collection but is not counted in their `subscriberCount` or `subscribers:change`, which adapters read as a request for live data. 16. **Status and the health tree:** status says whether the data can be shown; the health tree says whether it is still kept current. A sync run's failed first load (`fail()` before Collection readiness) is status `error` and is final for that sync run. A crash (IndexedDB closure, a terminal local persistence failure, a throwing sync function, or an internal invariant failure) is status `crashed` from any active status. Nothing recovers it in place; only cleanup, which discards all state, leaves it. A live query takes its sources' worst status until its initial-query readiness (`crashed` over `error` over `loading`). After it, only `crashed` propagates: a query cannot keep serving rows from a Collection that cannot serve reads. Every other problem after initial-query readiness, including an on-demand source whose own first load fails then, leaves the query's status and rows alone and shows on the node that owns it. A source's `reportError()` is its node's `failureReason`, which `reportHealthy()` clears and `reportHealthy(error)` trims by one failure; its `fail()` after Collection readiness is its node's `error`, which stays for the sync run, and every later report from that sync adapter throws. A lazy load or a root Collection's load that fails after initial-query readiness is an `error` on the query's own node. A lazy demand's error clears when that demand settles or is released; a root Collection's error clears when a later load of that Collection settles: one started after the failure, for a root Collection, since a load already in flight cannot cover the data whose request failed. A source reload starts the replay of every subscription's on-demand demand (a live query's, an Effect's, or a direct subscriber's) when it retires the sync run, as a truncate does, and the rows stay: the new run's writes apply on them. Only the new run's snapshot (`begin({ snapshot: true })`) or its own `truncate()` replaces them, and that replacement replays the demand again, also after the reload's replay settled. A reload keeps every row its new run does not rewrite: core deletes no row on its own, so removing a row the backend deleted is the sync adapter's job, through a snapshot, a `delete`, or row ownership, and an `insert` of a key the Collection holds throws `DuplicateKeySyncError`, during a reload as at any time ([#2269](https://github.com/TanStack/db/issues/2269)). A live query's own reload keeps its rows and its settled window until its new run's graph result is complete (law 9), then commits that result as its snapshot; the node then drops the retired run's request errors, and a window move still waiting for a replay ends with the retired run. During any reload the node shows the new run's own problems as they happen (maintainer decision): a failure the new run reports starts its frame at once, and its `reportHealthy()` clears the retired run's failure reason. One verdict or load that decides several request errors can announce `health:change` once per error, with no error shown between them; the state after it is exact. An abort ends the load: once core aborts a load (a window move, a demand retirement, a replay that replaces it, or a reload that retires its run), its later success or rejection decides no request error, holds no readiness, and no replay waits for it. A `loadSubset` that throws synchronously is a rejection: a failed load that holds its demand, which a later load or replay retries. A replay's verdict decides every request error of the demand it replayed: loads that settle while it is judged decide none themselves, its success clears them all, and its failure replaces them with its own failures, every failed load of its current attempt that the query still holds for a key no later held load loaded, in failure order. The replay loads the query's demand again as one unit: the query's first load waits for the whole replay, and the replay fails only for a demand the query still holds when it is judged. A failed replay stays a request error until a replay succeeds or the query holds no demand; once its attempt failed, a load that joins it and fails shows its own error at once. A lazy demand's error clears as soon as a later load of its failed key succeeds, without waiting for its other loads. When several request errors are open, the query's node shows the newest, as TanStack Query's `error`, and its `errors` lists every request failure still open, oldest first; two requests that fail with one error are one failure, which keeps the place of its first failure. While the source shows a failed first load, the query's demand changes start the acquisitions law 10 asks for without a request: they join an open replay, and the source's restart or next replay requests them. Query failures and contract violations, such as a graph failure or a Collection cleaned up under the query, crash the query. An Effect follows the same rules, with its initial-query readiness at its settled initial loads: a crashed source, a source whose first load fails before that readiness, or a failed load it needs ends it through `onSourceError`, because its incremental result can no longer be complete. An Effect has no request errors; after that readiness, a source's sync problem is health tree detail delivered through `onHealthChange`, while the Effect keeps running. 17. **Health tree:** a Collection's `health` is one node; a live query's has one node for itself and one per source alias in query order, recursing through nested live queries. A node shows at most one problem: `failureReason` (the sync adapter still tries) or `error` (the first load failed, the sync adapter stopped, the Collection crashed, or, on a query's own node, a request for more data failed). A problem shows its newest failure and lists every failure of its frame in `errors`: a failure reason's frame is its sync run's reports since the slot was last clear, once per error value, less each failure that `reportHealthy(error)` removed (the slot clears when none is left), an `error` from `fail()`, a failed first load, or a crash is one failure, and a request error lists the request failures still open. An include's source has `kind: 'include'`; its per-row child Collections are not in the tree, so adding a child changes nothing. Every change to the tree is announced by `health:change`, so readers can keep a tree until the next event. A pooled live query's tree is the tree a live-query Collection over its source would have, so it also reaches deeper levels. The shape is fixed by the query. Each node mirrors its Collection when read, and the query emits `health:change` whenever a node below it changes. When a Collection's status changes, its `status:change` fires first and its `health:change` follows at once; the tree is current when either fires. A node keeps its object identity until something in its subtree changes, and a change replaces only the nodes on its path. Adding or removing a failure in a problem's `errors` is a change, announced once, also for a Collection the query reads under several aliases. ## Glossary This subsystem glossary extends the shared [project glossary](../../../../../docs/contributing/glossary.md). Shared terms keep the meanings defined there. - **Relation:** an internal weighted multiset maintained by D2, not a public TanStack Collection. - **Weighted delta:** a positive or negative change to a relation row. - **Data plane:** the D2 graph that joins, reduces, orders, and materializes relations. - **Demand plane:** the async adapter that starts and releases source loads. - **Bucket key:** the canonical identity of one correlated child partition. - **Bucket relation:** child rows partitioned by bucket key. - **Active bucket:** a bucket referenced by at least one current route; it seeds empty materialization values and contributes to source demand. - **Bucket value:** the one inline value reduced from an active bucket's rows. - **Route relation:** weighted links from bucket keys to materialization cells. - **Materialization cell:** one include field on one parent-row occurrence. - **Arrangement:** retained relation state indexed for efficient keyed access and reuse. - **Reduction:** deriving one visible value from weighted rows sharing a key. - **Hydration:** establishing an initial snapshot before forwarding later changes. - **Generation:** a token that rejects obsolete asynchronous work. - **Collection facade:** a stable public Collection view shared by the parents routed to one active bucket. - **Coherent commit:** one publication in which state, events, and consumers see the same fully materialized result. ## Executable contracts | Contract | Test suite | | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | | State equivalence, route lifecycle, transition history, and batch partition | `packages/db/tests/query/includes-oracle.property.test.ts` | | Joined multiplicity, alias identity, and null-key normalization | `packages/db/tests/query/includes-query-shape-oracle.test.ts` | | Alias reuse, captured bindings, public keys, and `SourceId` preservation | (`packages/db/tests/query/includes-oracle.property.test.ts`, `packages/db/tests/query/includes-alias-shadowing-oracle.test.ts`) | | Same-scope and union alias rejection | `packages/db/tests/query/validate-aliases.test.ts` | | Demand, cancellation, progressive timing, and source readiness | `packages/db/tests/query/includes-temporal-oracle.test.ts` | | Optimistic confirmation, rollback, and later reactivity | `packages/db/tests/query/includes-optimistic-oracle.property.test.ts` | | Coherent layered publication | `packages/db/tests/query/includes-publication-oracle.test.ts` | | Collection facades, event coherence, and route activation | `packages/db/tests/query/includes-collection-oracle.property.test.ts` | | Correlated physical work | `packages/db/tests/query/includes-work-counter-oracle.test.ts` | | Constructed and retained facades in a nested Collection tree | `packages/db/tests/query/includes-space-oracle.test.ts` | | Route-context discovery and transport across recursive and join boundaries | `packages/db/tests/query/includes-context-transport-oracle.test.ts` | | Functional projection input boundaries, timing, and output preservation | `packages/db/tests/query/includes-functional-projection-oracle.test.ts` | | Functional input rejection and inline alternatives | `packages/db/tests/query/includes-functional-input-boundary.test.ts` | | Public-container descriptors and reference-key matches across internal query stages | `packages/db/tests/query/public-container-copy.test.ts` | | Cross-formulation equivalence and reference-sensitive route identity | `packages/db/tests/query/includes-cross-formulation-oracle.property.test.ts` | | Query-db ownership | `packages/query-db-collection/tests/ownership-lifecycle.oracle.test.ts` | | Failed replay retention, peer isolation, and explicit consumer-only recovery | `packages/db/tests/query/replay-failure-boundary.test.ts` | | Replay lease balance, reference-counted peers, and failed-start recovery | `packages/db/tests/replay-adapter-ownership.test.ts` | | Reachable nested shape | `packages/query-db-collection/tests/includes-work-counter-oracle.test.ts` | | Cleanup-start invalidation, settlement, and restart admission | `packages/db/tests/collection-cleanup-restart-oracle.test.ts` | | Pooled live queries match their live-query Collection, including cleanup of a mounted view | `packages/db/tests/query/pooled-live-query-oracle.property.test.ts` | | Deferred acquisition before a subscriber or preload, readiness, and the synchronous cut | `packages/db/tests/live-query-deferred-acquisition-oracle.test.ts` | Each oracle identifies the first divergent checkpoint and compares either the whole result or one exact structural difference. Correlated-materialization scenarios use direct assertions. A boundary suite may retain an exact expected-failure guard for a planner or ownership defect that this graph does not own. Run the DB oracle set with `pnpm test:oracles` from `packages/db`. Broad properties draw a random campaign seed, while structural matrices use fixed-seed campaigns so each run covers the same named cells. Increase both corpora with `TANSTACK_DB_ORACLE_RUNS_MULTIPLIER=10 pnpm test:oracles`. Preserve the reported campaign seed and shrink path while reducing a failure. Reseed a broad campaign with `TANSTACK_DB_ORACLE_SEED= pnpm test:oracles`, then add the smallest case as a deterministic regression trace. The nested-space test inspects facade construction and retained entries through test-only instrumentation. It adds no production metrics API. Run the related diagnostic workload with `pnpm bench:nested-includes` from `packages/db`; wall-clock timings are not a CI threshold. The broad relationship history changes correlation keys rather than freezing them. Set `TANSTACK_DB_ORACLE_STATISTICS=1` to print its generated depth, relationship-change, optimistic, and delete distribution. Collection-valued, array, and materialized includes are checked together for every Collection scenario instead of relying on a random mode sample. A separate metamorphic oracle compares nested includes with a flat join, fresh per-parent queries, and three-valued predicate partitioning. ## Implementation discipline - Express relation state with existing D2 inputs, joins, reductions, grouping, ordering, and consolidation before adding custom state. - Keep route and bucket rows in the same graph as parent and child query rows. - Add a reusable indexed D2 primitive only when existing operators cannot share or expose required retained state. - Keep asynchronous demand state outside D2 and make its generation boundary explicit. - Never use a public Collection, emitted event, or materialized row as internal routing or contribution state. - Add a reduced oracle trace before adding any special lifecycle branch. - Measure retained relation rows, active demands, and public facades. Preserve the no-includes fast path and verify any claimed space improvement with those counters.