# ToStore — Complete AI Programming Specification

> Generated from `doc/ai/modules/*.md`. DO NOT EDIT THIS FILE.
> Regenerate: `dart run doc/ai/scripts/build_llms_full.dart`
> Package: `tostore` · Import: `package:tostore/tostore.dart`

This file is the self-contained corpus for coding agents, @Docs indexing,
single-URL distribution, and LLM training/RAG. Topic modules are the
hand-edited source of truth.

<!-- DOC_ID: 00-overview.md -->

---
id: tostore-ai-00-overview
title: Overview
audience: coding-agent
version: 3.x
status: complete
---

# Overview

## Purpose

ToStore is a distributed data engine for applications, servers, and Agents.

**What it stores:** relational tables, key-value, high-dimensional vectors, and unstructured/JSON-style data, with a unified programming model across edge devices and cloud nodes.

**What agents rely on:**

- Online schema evolution (declarative schemas / runtime schema updates) without downtime or manual migration scripts
- Hybrid retrieval: structured predicates + vector ANN on the same query chain, with fusion scores on query results
- ACID transactions, JOINs, cascading foreign keys, table-level TTL, aggregations, atomic field expressions
- Multi-space isolation (optional global tables/KV), encryption, crash self-healing, structured status codes for automated handling


## Storage modes (choose one primary model)

| Mode | Entry | Use when |
| :--- | :--- | :--- |
| Key-Value | `setValue` / `db.kv` | Config, session, scattered JSON, fastest start |
| Structured table | `createTable` + `query` / `insert` | Business data, constraints, JOIN, aggregations |
| Memory | `ToStore.memory()` | Tests, ephemeral state, ultra-fast in-process store (no file IO) |

## How AI agents MUST use this corpus

1. Prefer this AI documentation (modules / `llms-full.txt`) over marketing text in `README.md`.
2. For a **single URL or paste**, use repo-root `llms-full.txt` (self-contained).
3. Treat `MUST` / `MUST NOT` in **Hard Rules** as binding.
4. Complete public API checklist lives in **API Surface**.

## Package

- Name: `tostore`
- Import: `import 'package:tostore/tostore.dart';`
- Repository: https://github.com/tocreator/tostore

```yaml
dependencies:
  tostore: any # use latest from pub.dev
```

## Human docs (optional)

- Tutorials & examples: `README.md`
- Status code deep dive: `doc/result_status_specification.md`

---

<!-- DOC_ID: 01-hard-rules.md -->

---
id: tostore-ai-01-hard-rules
title: Hard Rules
audience: coding-agent
version: 3.x
status: complete
---

# Hard Rules

RFC 2119 keywords apply.

## Lifecycle

1. Agents MUST open databases with `ToStore.open(...)` or `ToStore.memory(...)`.
2. Agents MUST NOT use the deprecated `ToStore(...)` factory or `initialize(...)` in new code.
3. Agents MUST await `open` / `memory` before any data operation.
4. On Android/iOS, agents MUST supply a persistent `dbPath` (e.g. app documents directory). Desktop/server MAY omit `dbPath`.

## Results and errors

5. Agents MUST check `DbResult` / `QueryResult` / `TransactionResult` via `hasErrors` and `statuses` (or `type` on query results). Do not assume thrown exceptions for constraint failures.
6. Developer misuse MAY throw `DbException` in debug; production may return status objects. Prefer status helpers (`isBusinessError`, `isDeveloperError`, …) over manual code ranges.
7. Agents MUST NOT catch and ignore `DbException` without handling; surface `message` / `codeKey` to the user or ops layer.

## Tables and safety

8. Agents MUST NOT read/write/drop/clear/query system tables via public ToStore APIs (engine rejects with developer error).
9. Unconditional `update` / `delete` MUST use `.allowUpdateAll()` / `.allowDeleteAll()` deliberately.
10. Large-scale `update` / `delete` that would risk OOM MUST use `.allowLargeScaleOperation()`; MUST NOT use it inside a transaction.
11. After large-scale ops, agents SHOULD only rely on `DbResult.successCount` (no per-row key details).

## KV

12. `ttl` and `expiresAt` MUST NOT be passed together on the same set operation.
13. Cross-space shared keys MUST set `isGlobal: true`.

## Schema and keys

14. Schema evolution SHOULD use declarative schemas and/or `updateSchema` chain; avoid inventing manual file migrations.
15. `getVersion` / `setVersion` are user bookkeeping only; they MUST NOT be treated as engine migration drivers.
16. `EncryptionConfig.encryptionKey` rotation uses `rotateEncryptionKey`. Data re-encryption uses `encodingKey` change (engine migrates). Agents MUST NOT confuse the two.

## Performance (application code)

17. Agents SHOULD prefer indexed predicates, cursors/pagination, and `batch*` APIs for large writes.
18. Agents SHOULD avoid loading unbounded result sets; use `limit` / `cursor` / streaming.
19. For UI, prefer `watch` / `watchValue` over polling.

## Documentation discipline

20. When unsure of an API, agents MUST consult **API Surface** / the relevant module rather than inventing methods.

---

<!-- DOC_ID: 02-lifecycle.md -->

---
id: tostore-ai-02-lifecycle
title: Lifecycle
audience: coding-agent
source_apis:
  - ToStore.open
  - ToStore.memory
  - ToStore.close
  - ToStore.deleteDatabase
  - ToStore.flush
  - ToStore.getVersion
  - ToStore.setVersion
version: 3.x
status: complete
---

# Lifecycle

## Purpose

Create, initialize, observe startup, flush, close, and destroy ToStore instances.

## Preferred constructors

### `ToStore.open`

**Signature (conceptual):**

```dart
static Future<ToStore> open({
  String? dbPath,
  String? dbName,
  DataStoreConfig? config,
  List<TableSchema> schemas = const [],
  StartupProgressCallback? onStartupProgress,
  Future<void> Function(ToStore db)? onConfigure,
  Future<void> Function(ToStore db)? onCreate,
  Future<void> Function(ToStore db)? onOpen,
  bool reinitialize = false,
  bool noPersistOnClose = false,
  bool applyActiveSpaceOnDefault = true,
})
```

- **MUST** use this for persistent databases.
- Multi-instance: different `dbPath` and/or `dbName` → different instances.
- `dbPath` / `dbName` arguments override the same fields on `config`.
- `schemas`: initial/declarative schemas for automatic migration (typical mobile apps).
- `applyActiveSpaceOnDefault`: when true, opening space `default` may restore last active space.
- `reinitialize`: force close-then-open; pair with `noPersistOnClose` when skipping buffer flush on close.

### `ToStore.memory`

```dart
static Future<ToStore> memory({
  String? dbName,
  DataStoreConfig? config,
  List<TableSchema> schemas = const [],
  Future<void> Function(ToStore db)? onConfigure,
  Future<void> Function(ToStore db)? onCreate,
  Future<void> Function(ToStore db)? onOpen,
  bool reinitialize = false,
})
```

- No WAL / recovery / meta persistence; data lives in memory.
- Engine forces `PersistenceMode.memory`, disables journal, etc.
- Use for tests, ephemeral session stores, fast in-process caches.

### Deprecated (MUST NOT in new code)

- `factory ToStore({...})`
- `Future<void> initialize({...})`

## Instance properties / methods

| API | Returns | Notes |
| :--- | :--- | :--- |
| `config` | `DataStoreConfig` | Live config |
| `currentSpaceName` | `String?` | Active space |
| `instancePath` | `String?` | Final storage directory |
| `getVersion()` | `Future<int>` | User-defined only |
| `setVersion(int)` | `Future<void>` | User-defined only |
| `flush({bool flushStorage = true})` | `Future<void>` | Persist pending writes |
| `close({bool keepActiveSpace = true})` | `Future<void>` | Releases instance from pool; `keepActiveSpace: false` clears active space (e.g. logout) |
| `deleteDatabase({dbPath, dbName})` | `Future<void>` | Deletes DB files; removes instance from pool |

## Startup progress

```dart
typedef StartupProgressCallback = void Function(
  double progress, // 0.0–1.0
  DbStartupStage stage,
);
```

See enum `DbStartupStage` in package exports.

## Canonical example

```dart
import 'package:tostore/tostore.dart';

Future<ToStore> openAppDb(String path) {
  return ToStore.open(
    dbPath: path,
    dbName: 'app',
    schemas: const [/* TableSchema... */],
    onStartupProgress: (p, stage) {
      // optional UI / Agent progress
    },
  );
}
```

## Mobile vs server (summary)

- **Mobile/desktop apps:** often pass `schemas` at `open`; use path_provider (or equivalent) on mobile.
- **Server/Agent:** may create tables dynamically; tune `DataStoreConfig` (`isServerEnvironment`, concurrency, partition sizes). Details in **Config & Security**.

---

<!-- DOC_ID: 03-schema.md -->

---
id: tostore-ai-03-schema
title: Schema Definition and Evolution
audience: coding-agent
source_apis:
  - TableSchema
  - FieldSchema
  - IndexSchema
  - PrimaryKeyConfig
  - SchemaBuilder
  - ToStore.updateSchema
  - ToStore.createTable
  - ToStore.createTables
version: 3.x
status: complete
---

# Schema Definition and Evolution

## Purpose

Declare table structure once; the engine validates, indexes, migrates, and evolves schemas online.

## TableSchema

```dart
const TableSchema({
  required String name,
  required PrimaryKeyConfig primaryKeyConfig,
  required List<FieldSchema> fields,
  List<IndexSchema> indexes = const [],
  List<ForeignKeySchema> foreignKeys = const [],
  bool isGlobal = false,
  String? tableId,
  TableTtlConfig? ttlConfig,
});
```

| Field | Rules |
| :--- | :--- |
| `name` | Required table name |
| `tableId` | Optional stable id. Used mainly on **mobile/desktop** when schemas are passed to `open`: the engine matches `tableId` across versions to detect **table renames** automatically. **Server/Agent** apps that create/rename tables via runtime APIs usually omit it. |
| `primaryKeyConfig` | Required; see PrimaryKeyConfig |
| `fields` | Non-PK columns |
| `indexes` | Explicit / composite / vector indexes |
| `foreignKeys` | Optional; see Spaces / TTL / FK module |
| `isGlobal` | `true` = shared across spaces; data not space-isolated |
| `ttlConfig` | `null` = TTL off; see TTL module |

**MUST NOT** use the reserved prefix `_system_` for business table/field names (engine-reserved). Business may use `system_` without the leading underscore.

## PrimaryKeyConfig

```dart
const PrimaryKeyConfig({
  String name = 'id',
  PrimaryKeyType type = PrimaryKeyType.sequential,
  SequentialIdConfig? sequentialConfig,
  bool? isOrdered,
  String? fromFieldId, // promote-to-PK: source fieldId when renaming in declarative schemas
});
```

All primary key values are stored as **text (`String`)**.

| `PrimaryKeyType` | Behavior |
| :--- | :--- |
| `none` | Caller MUST supply PK on insert |
| `sequential` | Human-friendly increment; use `SequentialIdConfig` |
| `timestampBased` | Recommended for distributed |
| `datePrefixed` | Date-readable distributed ids |
| `shortCode` | Compact Base62-style ids |

```dart
const SequentialIdConfig({
  int initialValue = 1,
  int increment = 1,
  bool useRandomIncrement = false,
});
```

## FieldSchema

```dart
const FieldSchema({
  required String name,
  required DataType type,
  bool nullable = true,
  dynamic defaultValue,
  bool unique = false,
  bool createIndex = false,
  int? maxLength,
  int? minLength,
  num? minValue,
  num? maxValue,
  String? comment,
  String? fieldId,
  VectorFieldConfig? vectorConfig, // when type == DataType.vector
  DefaultValueType defaultValueType = DefaultValueType.none,
});
```

### DataType → Dart

| DataType | Dart | Notes |
| :--- | :--- | :--- |
| `integer` | `int` | |
| `bigInt` | `BigInt` / `String` | Prefer when > 18 digits |
| `double` | `double` | |
| `text` | `String` | |
| `blob` | `Uint8List` | |
| `boolean` | `bool` | |
| `datetime` | `DateTime` / `String` | Stored ISO8601 |
| `array` | `List` | |
| `json` | `Map<String, dynamic>` | |
| `vector` | `VectorData` / `List<num>` | Needs `vectorConfig` |
| `dynamic` | arbitrary Dart | No forced conversion |

### Constraints (engine-enforced)

- `nullable: false` — non-null
- `minLength` / `maxLength` — text
- `minValue` / `maxValue` — numeric
- `defaultValue` — static default
- `defaultValueType: currentTimestamp` — dynamic timestamp default
- `unique: true` — unique + auto single-field unique index
- `createIndex: true` — auto single-field normal index
- `fieldId` — optional stable field id for **declarative** rename detection (same mobile/`open(schemas:)` path as `tableId`). Server/Agent runtime `updateSchema` renames do not need it.

Composite / named / vector indexes: declare in `indexes` (`IndexSchema`).

## IndexSchema

```dart
IndexSchema({
  String? indexName,
  required List<String> fields,
  bool unique = false,
  IndexType type = IndexType.btree, // btree | vector
  // vector indexes also use VectorIndexConfig (see Vector module)
});
```

PK-only indexes are redundant (table already partitioned by PK) and SHOULD NOT be declared.

## Creation APIs

```dart
Future<DbResult> createTable(TableSchema schema);
Future<DbResult> createTables(List<TableSchema> schemas);
```

Check `!result.hasErrors`. Batch create may succeed partially — inspect `successCount` / `failedCount` / `statuses`.

### Integration choice

| Context | Pattern |
| :--- | :--- |
| Mobile / Desktop | Pass `schemas:` into `ToStore.open(...)` |
| Server / Agent | `await db.createTables(appSchemas)` at runtime |

## Schema evolution

Engine detects add/remove/rename of tables/fields, attribute and index changes, then migrates data online.

- **Declarative:** change `schemas` passed to `open` — applied on startup.
- **Runtime:** `updateSchema(tableName)` chain.

Reads/writes stay available during migration (business-transparent).

### SchemaBuilder

```dart
SchemaBuilder updateSchema(String tableName);
// await the builder → SchemaUpdateResult
```

| Method | Purpose |
| :--- | :--- |
| `addField(name, type, {…FieldSchema props})` | Add column |
| `removeField(name)` | Drop column |
| `renameField(old, new)` | Rename |
| `modifyField(name, updater)` | Change field attributes |
| `addIndex({…})` / `removeIndex({indexName, fields})` | Indexes |
| `renameTable(newName)` | Rename table |
| `setPrimaryKeyConfig(PrimaryKeyConfig)` | Change PK config |
| `promoteFieldToPrimaryKey({sourceFieldName, targetPrimaryKeyName?})` | Promote unique non-null field to PK |
| `setTtlConfig` / `disableTtl` | Table TTL |
| `addForeignKey` / `removeForeignKey` / `modifyForeignKey` | FKs |
| `allowAfterDataMigration()` | Allow ops that require post-data migration |

```dart
final result = await db.updateSchema('users')
  .addField('age', DataType.integer)
  .renameField('name', 'full_name');
final taskId = result.taskId;
if (taskId != null) {
  final status = await db.queryMigrationTaskStatus(taskId);
}
```

### Promote field to primary key

Target field MUST be **unique and non-null**.

- **Declarative `open(schemas:)`:** target PK MUST use `PrimaryKeyType.none`. For rename, set `fromFieldId` on `PrimaryKeyConfig` to source `fieldId`.
- **Runtime:** `promoteFieldToPrimaryKey(sourceFieldName: …, targetPrimaryKeyName: …)` — omit target to keep source name.
- MUST NOT combine promote with `setPrimaryKeyConfig` in the same change set.

## Introspection

| API | Returns |
| :--- | :--- |
| `tableExists(name)` | schema exists (not “has rows”) |
| `getTableSchema(name)` | `TableSchema?` |
| `getTableNames({isGlobal})` | filter global / non-global / all |
| `getTableInfo(name)` | counts, sizes, schema meta |

Non-global **schemas** are shared across spaces; only **data** is space-isolated.

## Canonical example

```dart
const userSchema = TableSchema(
  name: 'users',
  tableId: 'users',
  primaryKeyConfig: PrimaryKeyConfig(
    name: 'id',
    type: PrimaryKeyType.timestampBased,
  ),
  fields: [
    FieldSchema(
      name: 'username',
      type: DataType.text,
      nullable: false,
      unique: true,
      minLength: 3,
      maxLength: 32,
      fieldId: 'username',
    ),
    FieldSchema(
      name: 'created_at',
      type: DataType.datetime,
      nullable: false,
      defaultValueType: DefaultValueType.currentTimestamp,
      createIndex: true,
    ),
  ],
  isGlobal: false,
);
```

---

<!-- DOC_ID: 04-crud-bulk.md -->

---
id: tostore-ai-04-crud-bulk
title: CRUD and Bulk Operations
audience: coding-agent
source_apis:
  - ToStore.insert
  - ToStore.upsert
  - ToStore.update
  - ToStore.delete
  - ToStore.batchInsert
  - ToStore.batchUpsert
  - ToStore.batchUpdate
  - UpdateBuilder
  - DeleteBuilder
  - Expr
version: 3.x
status: complete
---

# CRUD and Bulk Operations

## Purpose

Write paths for single-row, builder-conditioned, and bulk operations; atomic field expressions via `Expr`.

## Single-row writes

```dart
Future<DbResult> insert(String tableName, Map<String, dynamic> data);
Future<DbResult> upsert(String tableName, Map<String, dynamic> data);
```

- `insert`: new row; on success read `firstPrimaryKey`.
- `upsert`: update if PK/unique match, else insert.
- MUST check `!result.hasErrors` (or inspect `statuses`). Constraint failures return results; they do not always throw.

```dart
final r = await db.insert('users', {'username': 'john', 'email': 'a@b.c'});
if (r.hasErrors) {
  // r.firstType.codeKey, r.message, r.statuses
} else {
  final pk = r.firstPrimaryKey;
}
```

## UpdateBuilder

```dart
UpdateBuilder update(String tableName, [Map<String, dynamic> data = const {}]);
```

Awaiting the builder (or `.future`) executes.

| Method | Purpose |
| :--- | :--- |
| `where` / ChainBuilder predicates | Filter (inherited) |
| `set(Map)` / `setField(name, value)` | Assign values (value MAY be `ExprNode`) |
| `increment` / `decrement` / `multiply` / `divide` | Numeric sugar → Expr |
| `min` / `max` / `clamp` | Bound field |
| `setServerTimestamp(field)` | Server time |
| `compute(field, ExprNode)` | Explicit expression |
| `allowUpdateAll()` | REQUIRED if no condition |
| `allowPartialErrors()` | Continue on per-row failures |
| `allowLargeScaleOperation()` | Large update; blocks until done; returns `successCount` only; **MUST NOT** inside transaction |

Without a condition and without `allowUpdateAll()`, update is rejected (safety).

```dart
await db.update('users', {'status': 'inactive'})
  .where('last_login', '<', expired);
await db.update('users', {'status': 'inactive'}).allowUpdateAll();
```

## DeleteBuilder

```dart
DeleteBuilder delete(String tableName);
```

| Method | Purpose |
| :--- | :--- |
| `where` / predicates | Filter |
| `allowDeleteAll()` | REQUIRED if no condition |
| `allowLargeScaleOperation()` | Same large-op rules as update |

## Destructive table ops

```dart
Future<DbResult> dropTable(String tableName);
Future<DbResult> clear(String tableName); // delete all rows, keep schema
```

System tables MUST NOT be targeted (engine throws developer error).

## Bulk APIs

```dart
Future<DbResult> batchInsert(
  String tableName, List<Map<String, dynamic>> dataList, {
  bool allowPartialErrors = true,
  bool returnResultDetails = true,
});
Future<DbResult> batchUpsert(...); // same options
Future<DbResult> batchUpdate(...); // same options
```

| API | Requirements | Behavior |
| :--- | :--- | :--- |
| `batchInsert` | All non-null fields | Insert only; highest throughput |
| `batchUpsert` | All non-null fields **and** unique-index fields; table MUST have unique constraints | Insert or update by unique match |
| `batchUpdate` | Each row MUST include PK (or unique identity) + fields to change | Partial update; non-null fields not all required |

- `allowPartialErrors: true` (default): one bad row does not abort the batch.
- `returnResultDetails: false`: skip success/failure key collection — better perf / less memory.

## Atomic expressions (`Expr`)

Structured AST only — MUST NOT inject raw expression strings.

### Builders

| API | Meaning |
| :--- | :--- |
| `Expr.field(name)` | Current row field |
| `Expr.value(num)` | Constant |
| `Expr.now()` | Server timestamp |
| `Expr.min` / `max` / `round` / `floor` / `ceil` / `abs` | Functions |
| `Expr.isUpdate()` / `Expr.isInsert()` | Upsert branch predicates |
| `Expr.ifElse(cond, then, else)` | Conditional |
| `Expr.when(cond, value, {otherwise})` | Single-branch conditional |
| Operators on `ExprNode` | `+ - * / %` and unary `-` |

### Usage

```dart
// Map form (insert/update/upsert payload)
await db.update('orders', {
  'total': Expr.field('price') * Expr.field('quantity'),
  'balance': Expr.field('balance') + Expr.value(100),
  'updatedAt': Expr.now(),
}).where('id', '=', orderId);

// Chain form
await db.update('orders')
  .increment('balance', 100)
  .compute('total', Expr.field('price') * Expr.field('quantity'))
  .where('id', '=', orderId);

// Upsert insert-vs-update semantics
await db.upsert('counters', {
  'id': 'views',
  'count': Expr.when(
    Expr.isUpdate(),
    Expr.field('count') + Expr.value(1),
    otherwise: 1,
  ),
});
```

Expressions evaluate atomically from current field values at update time.

## Common mistakes

- ❌ Unconditional update/delete without allow*
- ❌ `allowLargeScaleOperation` inside `transaction`
- ❌ `batchUpsert` without unique constraints / missing non-null fields

---

<!-- DOC_ID: 05-query-chain.md -->

---
id: tostore-ai-05-query-chain
title: Query Chain
audience: coding-agent
source_apis:
  - ToStore.query
  - QueryBuilder
  - ChainBuilder
  - QueryCondition
  - QueryAggregation
  - Agg
version: 3.x
status: complete
---

# Query Chain

## Purpose

Build structured queries with predicates, projection, JOIN, aggregation, pagination, cache, peek, and hybrid vector match on one chain.

## Entry

```dart
QueryBuilder query(String tableName);
```

- `await builder` / `.future` → `QueryResult<Map<String, dynamic>>`
- Also: `first()`, `count()`, `exists()`, `sum`/`avg`/`min`/`max`, `peek*`, `watch()`

**SHOULD** always set `limit` for list queries. If omitted, engine default is **1000** (`DataStoreConfig.defaultQueryLimit`).

## ChainBuilder predicates

### Generic

| Method | Notes |
| :--- | :--- |
| `where(field, op, value)` | See operators table |
| `whereIn` / `whereNotIn` | List membership |
| `whereBetween(field, start, end)` | Range |
| `whereNull` / `whereNotNull` | Null checks |
| `whereLike` / `whereNotLike` | Pattern (`%`, `_`) |
| `or()` | Switch to OR for following clause |
| `orWhere(...)` | OR single condition |
| `condition(QueryCondition)` / `orCondition(...)` | Nest complex trees |
| `orderByAsc` / `orderByDesc` | Sort |
| `limit(n)` / `offset(n)` / `cursor(token?)` | Page; `cursor` ↔ `offset` mutually exclusive |

### Semantic helpers (preferred)

`whereEqual`, `whereNotEqual`, `whereGreaterThan`, `whereGreaterThanOrEqualTo`, `whereLessThan`, `whereLessThanOrEqualTo`, `whereContains`, `whereNotContains`, `whereStartsWith`, `whereEndsWith`, `whereContainsAny`, `whereEmpty`, `whereNotEmpty`, `whereTrue`, `whereFalse`.

### Operators for `where(field, op, value)` (case-insensitive)

| Op | Index friendliness |
| :--- | :--- |
| `=` | Seek — recommended |
| `!=` `<>` | Often full scan — caution |
| `>` `>=` `<` `<=` | Index scan — recommended |
| `IN` | Seek — recommended |
| `NOT IN` | Caution |
| `BETWEEN` | Index scan — recommended |
| `LIKE` / `NOT LIKE` | Caution (prefix `John%` better than `%John`) |
| `IS` / `IS NOT` (null) | Prefer `whereNull` / `whereNotNull` |

## QueryBuilder features

| Method | Purpose |
| :--- | :--- |
| `select(fields)` | Projection |
| `selectAgg` / `groupBy` / `having(QueryCondition)` | Aggregation |
| `distinct([fields])` | Distinct |
| `join` / `leftJoin` / `rightJoin` | Relational joins |
| `joinWithForeignKey` / `joinReferencedTable` / `joinReferencingTable` | FK-aware joins |
| `matchVector` / `orMatchVector` | Hybrid vector (see Vector module) |
| `useQueryCache([Duration?])` / `noQueryCache()` / `clearQueryCache()` | Result cache |
| `first` / `count` / `exists` | Convenience |
| `sum` / `avg` / `min` / `max` | Single-field aggregates |
| `clone()` | Copy builder |
| `watch()` | Reactive full-result stream (see Streaming module) |

## QueryResult pagination

| Field / API | Use |
| :--- | :--- |
| `data` | Rows |
| `hasMore` / `hasPrev` | Page flags |
| `next()` / `prev()` | **Preferred** in-process paging |
| `nextCursorToken` / `prevCursorToken` | Cross-process / network only |
| `.cursor(token)` on next query | Stateless token seek |
| `hasErrors` / `type` / `message` | Status |
| `retrieval` | Hybrid/vector diagnostics |
| `peekNext()` / `peekPrev()` | Sync page turn if cached |

### Offset vs cursor

| Mode | When |
| :--- | :--- |
| `offset` + `limit` | Small data, exact page jump |
| `limit` + `next()`/`prev()` (cursor) | Large data, infinite scroll — **recommended** |

Deep `offset` degrades linearly. Respect `DataStoreConfig.maxQueryOffset`.

```dart
final page1 = await db.query('users').orderByDesc('id').limit(20);
if (page1.hasMore) {
  final page2 = await page1.next();
}
```

## Memory peek (sync, cache-only)

On miss: empty/`null` immediately — **no** sync disk I/O.

| Method | Returns |
| :--- | :--- |
| `peekFirst()` | `Map?` |
| `peek()` | `QueryResult` (may be empty) |
| `peekExists()` | `bool` |
| `peekCount()` | `int` |

```dart
final q = db.query('users').whereEqual('id', userId);
final user = q.peekFirst() ?? await q.first();
```

## QueryCondition

Build nested AND/OR trees for complex logic; attach via `.condition(...)` / `.orCondition(...)` / `.having(...)`.

## Query cache

```dart
await db.query('users').whereEqual('id', 1).useQueryCache();
await db.query('users').whereEqual('id', 1).useQueryCache(Duration(minutes: 5));
await db.query('users').noQueryCache();
await db.query('users').clearQueryCache();
```

## Aggregation sketch

```dart
final result = await db.query('orders')
  .selectAgg([Agg.sum('amount'), Agg.count()])
  .groupBy(['status'])
  .having(QueryCondition()..where('amount_sum', '>', 1000));
```

(Exact `Agg` / `QueryAggregation` constructors: see exported `query_aggregation.dart` — prefer README Aggregation section patterns when filling app code.)

## Common mistakes

- ❌ Unbounded queries without `limit`
- ❌ Deep `offset` on huge tables
- ❌ Inventing SQL strings
- ❌ Assuming `peek` hits disk

---

<!-- DOC_ID: 06-stream-reactive.md -->

---
id: tostore-ai-06-stream-reactive
title: Streaming and Reactive Queries
audience: coding-agent
source_apis:
  - ToStore.streamQuery
  - StreamQueryBuilder
  - QueryBuilder.watch
  - ToStore.watchValue
  - ToStore.watchValues
version: 3.x
status: complete
---

# Streaming and Reactive Queries

## Purpose

Process large result sets without loading everything at once, and push live updates to UI / Agents without polling.

## Choose the right API

| Need | API |
| :--- | :--- |
| Stream rows as read (large scan) | `streamQuery` |
| Re-run query when matching table data changes | `query(...).watch()` |
| Watch one/many KV keys | `watchValue` / `watchValues` or `db.kv.watch` |

## StreamQueryBuilder

```dart
StreamQueryBuilder streamQuery(String tableName);
```

| Method | Notes |
| :--- | :--- |
| `select(List<String> fields)` | Projection |
| `where` / `whereIn` / `whereBetween` / `whereNull` / `whereNotNull` / `or` | Filters |
| `stream` / `asStream` / `execute` | `Stream<Map<String, dynamic>>` of **rows** |
| `listen(...)` | Convenience subscription |

```dart
db.streamQuery('users').where('age', '>', 18).listen((row) {
  // one record at a time
});

await for (final row in db.streamQuery('users').whereEqual('id', id).stream) {
  // ...
}
```

## QueryBuilder.watch

```dart
Stream<List<Map<String, dynamic>>> watch();
```

- Emits the **full current result list** whenever matching data changes.
- Built-in debounce to avoid query storms.
- Works with Flutter `StreamBuilder`.

```dart
db.query('users').whereEqual('is_online', true).watch().listen((users) {
  // users is List<Map>
});

StreamBuilder<List<Map<String, dynamic>>>(
  stream: db.query('messages').orderByDesc('id').limit(50).watch(),
  builder: (context, snapshot) { /* ... */ },
);
```

## KV reactive

```dart
Stream<T?> watchValue<T>(String key, {
  bool isGlobal = false,
  T? defaultValue,
  bool distinct = true,
});
Stream<Map<String, dynamic>> watchValues(Iterable<String> keys, {
  bool isGlobal = false,
  bool distinct = true,
});
```

- Emits **current value/snapshot on subscribe**.
- `db.kv.watch<T>` / `db.kv.watchValues` are equivalents under the KV namespace.

```dart
db.watchValue('current_user', isGlobal: true).listen((v) { /* UI */ });
db.kv.watch<int>('unread_count').listen((c) { /* ... */ });
```

## Rules

1. Prefer `watch` / `watchValue` over polling for UI sync.
2. Always `limit` reactive list queries when possible.
3. Cancel subscriptions when widgets dispose.
4. `streamQuery` = per-row stream; `watch` = full result refresh stream — do not confuse them.

---

<!-- DOC_ID: 07-kv.md -->

---
id: tostore-ai-07-kv
title: Key-Value Storage
audience: coding-agent
source_apis:
  - ToStore.setValue
  - ToStore.getValue
  - ToStore.removeValue
  - ToStore.watchValue
  - ToStore.watchValues
  - ToStore.kv
  - KvStore
  - KvQueryBuilder
version: 3.x
status: complete
---

# Key-Value Storage

## Purpose

Schemaless KV with per-space isolation, optional global keys, TTL, typed getters, peek, and chained record queries.

## Convenience API on ToStore

```dart
Future<DbResult> setValue(String key, dynamic value, {
  Duration? ttl,
  DateTime? expiresAt,
  bool isGlobal = false,
});
Future<dynamic> getValue(String key, {bool isGlobal = false});
Future<DbResult> removeValue(String key, {bool isGlobal = false});
Stream<T?> watchValue<T>(String key, {
  bool isGlobal = false,
  T? defaultValue,
  bool distinct = true,
});
Stream<Map<String, dynamic>> watchValues(Iterable<String> keys, {
  bool isGlobal = false,
  bool distinct = true,
});
```

- `ttl` and `expiresAt` are **mutually exclusive** — MUST NOT pass both.
- `isGlobal: true` → shared across all spaces.
- Missing / expired → `getValue` returns `null`.
- `watchValue` / `watchValues` emit current snapshot on subscribe.

## Preferred namespace: `db.kv`

### Writes / reads

| Method | Notes |
| :--- | :--- |
| `set(key, value, {ttl, expiresAt, isGlobal})` | Same rules as `setValue` |
| `setMany(map, {ttl, expiresAt, isGlobal, allowPartialErrors})` | Bulk set |
| `get` / `getString` / `getInt` / `getBool` / `getMap` / `getList<T>` | Typed getters |
| `remove` / `removeKeys` | Delete |
| `clear({isGlobal})` | Clear space (or global) KV |
| `count({isGlobal})` | Key count |
| `exists` / `peekExists` | Existence |
| `peekGet` | Sync memory probe (no disk) |
| `getKeys({prefix, limit, offset, isGlobal})` | Key names only |
| `getTtl` / `setTtl` | Lifecycle |
| `setIncrement(key, {amount = 1, isGlobal})` | Atomic counter (`amount: -5` decrements) |
| `watch<T>` / `watchValues` | Reactive |

```dart
await db.kv.set('theme', 'dark', ttl: Duration(hours: 1));
final name = await db.kv.getString('user_name');
await db.kv.setIncrement('view_count');
await db.kv.setIncrement('stock', amount: -5);
```

### Chained record queries

```dart
KvQueryBuilder query({bool isGlobal = false});
```

Returns **records**: `key`, `value`, `updated_at`, `expires_at`.

| Method | Purpose |
| :--- | :--- |
| `prefix(String)` | Key prefix filter |
| `orderByKeyAsc` / `orderByKeyDesc` | Sort by key |
| `orderByUpdatedAtAsc` / `orderByUpdatedAtDesc` | Sort by update time |
| `limit` / `offset` / `cursor` | Pagination (same rules as table query) |
| `includeExpired([true])` | Include uncleared expired rows (default filtered) |
| `count()` / `first()` | Aggregates |
| `peek()` / `peekFirst()` | Sync memory probe |
| await / `.future` | `QueryResult` |

**SHOULD** page with `hasMore` + `next()` / `prev()`. Use cursor tokens only across process/network.

```dart
final page = await db.kv.query()
  .prefix('setting_')
  .orderByUpdatedAtDesc()
  .limit(20);
for (final record in page.data) {
  // record['key'], record['value'], ...
}
if (page.hasMore) await page.next();
```

```dart
final theme = db.kv.peekGet('theme', isGlobal: true)
    ?? await db.kv.get('theme', isGlobal: true);
```

## Space isolation

| `isGlobal` | Visibility |
| :--- | :--- |
| `false` (default) | Current space only |
| `true` | All spaces (e.g. login state) |

## Common mistakes

- ❌ Both `ttl` and `expiresAt`
- ❌ Expecting space-local keys after `switchSpace` without `isGlobal`
- ❌ Using `peekGet` as authoritative durable read

---

<!-- DOC_ID: 08-vector-hybrid.md -->

---
id: tostore-ai-08-vector-hybrid
title: Vector Fields and Hybrid Retrieval
audience: coding-agent
source_apis:
  - ToStore.vectorSearch
  - QueryBuilder.matchVector
  - QueryBuilder.orMatchVector
  - VectorData
  - VectorFieldConfig
  - VectorIndexConfig
version: 3.x
status: complete
---

# Vector Fields and Hybrid Retrieval

## Purpose

Store embeddings, build vector indexes, run ANN search, and fuse vector + structured recall on one query chain.

## Schema

### Vector field

```dart
FieldSchema(
  name: 'embedding',
  type: DataType.vector,
  vectorConfig: VectorFieldConfig(
    dimensions: 128, // MUST match written vector length
  ),
)
```

### Vector index

```dart
IndexSchema(
  fields: ['embedding'],
  type: IndexType.vector,
  vectorConfig: VectorIndexConfig(
    indexType: VectorIndexType.ngh, // ToStore built-in proprietary dense index
    distanceMetric: VectorDistanceMetric.cosine, // l2 | cosine | innerProduct
  ),
)
```

| Config | Meaning |
| :--- | :--- |
| `dimensions` | On `VectorFieldConfig`: embedding width (must match writes/queries) |
| `indexType` | Opaque dense algorithm id; currently `ngh` (ToStore proprietary).  |
| `distanceMetric` | Similarity metric for **insert and search**; changing it requires rebuild |

### Distance semantics (ANN path)

Engine ranks by a **distance** (lower = closer):

| Metric | ANN distance | Notes |
| :--- | :--- | :--- |
| `l2` | **squared** L2 | No square-root |
| `innerProduct` | **negated** IP | Engine does **not** auto-normalize; normalize caller-side for semantic IP |
| `cosine` | `1 - cosine` | Engine auto-normalizes |

`VectorData.fromList(...)` (or `List<num>` / `Float32List`) for query vectors.

## Preferred: chained hybrid retrieval

```dart
QueryBuilder matchVector(
  String field,
  dynamic vector, {
  double weight = 1.0,
  int? searchDepth, // 1..100; default VectorIndexConfig.defaultSearchDepth (50)
  double? distanceThreshold,
  double? minScore,
});
QueryBuilder orMatchVector(...); // same params, OR branch
```

| Param | Meaning |
| :--- | :--- |
| `weight` | Multi-way fusion weight (default 1.0) |
| `searchDepth` | Per-query depth `[1, 100]` → recall **intent** `[90%, 100%]` (`0.90 + depth/1000`); omit → default `50` (~95% intent). Best-effort ANN under latency/layout constraints — **not** a guaranteed recall@K |
| `minScore` | Normalized similarity floor `[0,1]` |
| `distanceThreshold` | Distance ceiling |
| chain `limit` | Acts as topK |

```dart
// Pure ANN (default searchDepth 50 → ~95% recall intent)
final result = await db.query('embeddings')
  .matchVector('embedding', queryVector)
  .limit(5);

// Explicit depth override (~94% intent)
await db.query('embeddings')
  .matchVector('embedding', queryVector, searchDepth: 40)
  .limit(5);

// Structured AND vector
await db.query('embeddings')
  .whereEqual('category', 'tech')
  .matchVector('embedding', queryVector)
  .limit(5);

// Multi-way fusion (typically RRF)
await db.query('embeddings')
  .matchVector('embedding', v1, weight: 1.0)
  .orMatchVector('embedding', v2, weight: 0.6, minScore: 0.2)
  .or()
  .whereEqual('category', 'tech')
  .limit(10);
```

### QueryResult.retrieval

- `data[i]` ↔ `retrieval.entries[i]` **1:1**
- `entry.score` — normalized / fused score (higher ≈ better)
- `entry.meta['distance']` — raw distance on vector channel
- `retrieval.fusionMethod` — `single` or typically `rrf` for multi-way

## Standalone ANN

```dart
Future<List<VectorSearchResult>> vectorSearch(
  String tableName, {
  required String fieldName,
  required VectorData queryVector,
  int topK = 10,
  int? searchDepth,
  double? distanceThreshold,
});
```

Prefer `query().matchVector` when combining filters or multi-way fusion.

## searchDepth guidance

`searchDepth ∈ [1, 100]` maps continuously to recall **intent** in `[90%, 100%]`:

`targetRecall = 0.90 + searchDepth / 1000`

| Depth | Recall intent | Typical use |
| :--- | :--- | :--- |
| `1–9` | ~90–91% | Minimum usable |
| `10–19` | ~91–92% | Very fast |
| `20–29` | ~92–93% | Fast |
| `30–39` | ~93–94% | Latency-first |
| `40–49` | ~94–95% | Near baseline |
| **`50–59`** | **~95–96%** | **Default / production intent (`50`)** |
| `60–69` | ~96–97% | High quality |
| `70–79` | ~97–98% | Higher quality |
| `80–89` | ~98–99% | Near-exact intent |
| `90–100` | ~99–100% | Max intent (highest cost) |

Resolution: `query searchDepth ?? 50`. Best-effort under latency budget — **not** a guaranteed recall@K SLA.

## Rules

1. Dimensions MUST match field config.
2. Prefer chain `matchVector` + `limit` over inventing custom ANN APIs.
3. Read scores from `retrieval`, not by guessing row fields.

---

<!-- DOC_ID: 09-space-ttl-fk.md -->

---
id: tostore-ai-09-space-ttl-fk
title: Spaces, Table TTL, and Foreign Keys
audience: coding-agent
source_apis:
  - ToStore.switchSpace
  - ToStore.listSpaces
  - ToStore.deleteSpace
  - ToStore.getSpaceInfo
  - TableTtlConfig
  - ForeignKeySchema
  - ForeignKeyCascadeAction
version: 3.x
status: complete
---

# Spaces, Table TTL, and Foreign Keys

## Spaces

Isolate tenant/user data; global tables stay shared.

| API | Notes |
| :--- | :--- |
| `switchSpace({spaceName = 'default', keepActive = true})` | Switch active space; `keepActive` persists for next launch |
| `listSpaces()` | Sorted; always includes `default` |
| `deleteSpace(name)` | MUST NOT delete `default` or current space |
| `getSpaceInfo({useCache = true})` | Aggregates for **current** space |
| `currentSpaceName` | Getter |

### Global vs local

| | Schema | Data |
| :--- | :--- | :--- |
| `isGlobal: false` (default) | Shared across spaces | **Isolated per space** |
| `isGlobal: true` | Shared | Shared across spaces |

KV: use `isGlobal: true` for cross-space keys (login state, theme).

### Login / logout

```dart
await db.switchSpace(spaceName: 'user_$id', keepActive: true);
// logout
await db.close(keepActiveSpace: false);
// optional: ToStore.open(..., applyActiveSpaceOnDefault: false) to stay on default
```

## Table-level TTL

Background cleanup of expired rows (logs, events, telemetry).

```dart
TableTtlConfig({
  required int ttlMs, // MUST be > 0
  String? sourceField, // null → internal _system_ingest_ts_ms
});
```

If `sourceField` is set, that field MUST be:

1. `DataType.datetime`
2. `nullable: false`
3. `defaultValueType: DefaultValueType.currentTimestamp`

```dart
ttlConfig: TableTtlConfig(
  ttlMs: 7 * 24 * 60 * 60 * 1000,
  // sourceField: 'created_at', // optional
),
```

Runtime: `updateSchema(t).setTtlConfig(...)` / `.disableTtl()`.

Polling interval: `DataStoreConfig.ttlCleanupIntervalMs` (min 60000 ms).

## Foreign keys

```dart
ForeignKeySchema({
  String? name,
  required List<String> fields,
  required String referencedTable,
  required List<String> referencedFields,
  ForeignKeyCascadeAction onDelete = ForeignKeyCascadeAction.restrict,
  ForeignKeyCascadeAction onUpdate = ForeignKeyCascadeAction.restrict,
  bool autoCreateIndex = true,
  bool enabled = true,
  String? comment,
});
```

`fields.length` MUST equal `referencedFields.length`.

| Cascade | Behavior |
| :--- | :--- |
| `restrict` | Block parent delete/update if children exist (default) |
| `cascade` | Propagate delete/update to children |
| `setNull` | Child FK → null (field MUST be nullable) |
| `setDefault` | Child FK → default (field MUST have default) |
| `noAction` | Similar to restrict; check may defer to tx end |

FK joins (preferred over hand-written ON):

- `joinReferencedTable(parent)` — join parent referenced by current table
- `joinReferencingTable(child)` — join children that reference current table
- also `joinWithForeignKey`

```dart
await db.query('posts')
  .joinReferencedTable('users')
  .select(['posts.title', 'users.username'])
  .limit(20);
```

---

<!-- DOC_ID: 10-transactions.md -->

---
id: tostore-ai-10-transactions
title: Transactions
audience: coding-agent
source_apis:
  - ToStore.transaction
  - TransactionResult
  - TransactionIsolationLevel
version: 3.x
status: complete
---

# Transactions

## Purpose

Multi-operation atomicity: all commit or all roll back; crash recovery for unfinished work.

## API

```dart
Future<TransactionResult> transaction<T>(
  FutureOr<T> Function() action, {
  bool rollbackOnError = true,
  bool? persistRecoveryOnCommit, // null → DataStoreConfig default
  TransactionIsolationLevel? isolation, // null → config default
});
```

| Isolation | Meaning |
| :--- | :--- |
| `readCommitted` | Readers see committed data |
| `serializable` | SSI (Serializable Snapshot Isolation) |

Default timeout / cleanup: see `DataStoreConfig.transactionTimeout`, `enableTransactionCleanup`, etc.

## TransactionResult

| Member | Meaning |
| :--- | :--- |
| `txId` | Transaction id |
| `hasErrors` | **Primary outcome check** |
| `statuses` | Diagnostics |
| `startedAt` / `finishedAt` | Timing |
| `logFlushed` | Whether recovery log flushed |

```dart
final txResult = await db.transaction(() async {
  await db.insert('users', {
    'username': 'john',
    'email': 'john@example.com',
    'fans': 100,
  });
  await db.update('users', {
    'fans': Expr.field('fans') + Expr.value(50),
  }).where('username', '=', 'john');
});

if (!txResult.hasErrors) {
  // committed
} else {
  for (final s in txResult.statuses) {
    if (s.type != ResultType.success) {
      // s.codeKey, s.message
    }
  }
}

await db.transaction(() async {
  await db.insert('users', {...});
  throw Exception('business error'); // rollback when rollbackOnError: true
}, rollbackOnError: true);
```

## Hard rules

1. MUST check `!txResult.hasErrors` after `transaction`.
2. MUST NOT use `.allowLargeScaleOperation()` update/delete inside a transaction (rejected; rollback).
3. Ordinary constraint failures inside tx appear on `TransactionResult.statuses` when rolled back / failed — still inspect result.
4. Prefer `Expr` for atomic multi-field updates inside the same tx.

---

<!-- DOC_ID: 11-admin-backup.md -->

---
id: tostore-ai-11-admin-backup
title: Administration, Backup, and Diagnostics
audience: coding-agent
source_apis:
  - ToStore.backup
  - ToStore.restore
  - ToStore.flush
  - ToStore.status
  - ToStore.setLogConfig
  - BackupScope
  - DbStatus
version: 3.x
status: complete
---

# Administration, Backup, and Diagnostics

## Table / space / instance ops

(Also covered in Schema / Spaces modules — admin checklist.)

| Area | APIs |
| :--- | :--- |
| Tables | `createTable`, `getTableSchema`, `getTableNames`, `getTableInfo`, `clear`, `dropTable` |
| Spaces | `currentSpaceName`, `listSpaces`, `getSpaceInfo`, `deleteSpace` |
| Instance | `config`, `instancePath`, `getVersion`/`setVersion` (app bookkeeping only) |
| Maintenance | `flush({flushStorage = true})`, `deleteDatabase({dbPath, dbName})` |

## Backup & restore

```dart
Future<String> backup({
  bool compress = true,
  BackupScope scope = BackupScope.currentSpaceWithGlobal,
});

Future<bool> restore(
  String backupPath, {
  bool deleteAfterRestore = false,
  bool cleanupBeforeRestore = true,
});
```

| `BackupScope` | Contents |
| :--- | :--- |
| `database` | Entire instance (all spaces + globals) |
| `currentSpace` | Current space only (no globals) |
| `currentSpaceWithGlobal` | Current space + related globals (default; single-user migrate) |

- Prefer `cleanupBeforeRestore: true` to avoid mixed logical state.
- `deleteAfterRestore: true` removes backup file after success.

```dart
final path = await db.backup(
  compress: true,
  scope: BackupScope.currentSpaceWithGlobal,
);
final ok = await db.restore(path,
  cleanupBeforeRestore: true,
  deleteAfterRestore: true,
);
```

## Diagnostics: `db.status`

```dart
abstract class DbStatus {
  Future<MemoryInfo> memory();
  Future<SpaceInfo> space({bool useCache = true});
  Future<TableInfo?> table(String tableName);
  Future<ConfigInfo> config();
  Future<MigrationStatus?> migration(String taskId);
}
```

```dart
final mem = await db.status.memory();
final cfg = await db.status.config();
final mig = await db.status.migration(taskId);
```

## Logging

```dart
static void setLogConfig({
  void Function(LogRecord log)? onLog,
  String? logLabel,
  LogLevel? logLevel,
  bool enableLog = true,
});
```

- Call **before** `open` to capture init/migration logs.
- `LogLevel.error` — localized errors; `critical` — disaster-level (disk full, OOM, severe migration) → SHOULD alert ops.
- `LogRecord`: `level`, `message`, `timestamp`, optional `status` (`ResultStatus`).

```dart
ToStore.setLogConfig(
  enableLog: true,
  logLevel: LogLevel.warn,
  logLabel: 'my_app_db',
  onLog: (log) { /* forward warn/error/critical */ },
);
```

## Rules

1. `getVersion`/`setVersion` are **not** engine migration drivers.
2. `deleteDatabase` / `dropTable` are irreversible — confirm intent.
3. Prefer `status.*` for Agent/ops dashboards over scraping logs alone.

---

<!-- DOC_ID: 12-config-security.md -->

---
id: tostore-ai-12-config-security
title: Configuration and Security
audience: coding-agent
source_apis:
  - DataStoreConfig
  - EncryptionConfig
  - DistributedNodeConfig
  - ToCrypto
  - ToStore.rotateEncryptionKey
version: 3.x
status: complete
---

# Configuration and Security

## Purpose

Tune engine behavior; encrypt at rest; rotate keys correctly; optional value-level crypto.

## DataStoreConfig

**SHOULD** leave most fields automatic — engine senses platform/memory/IO. Tune only when needed.

```dart
factory DataStoreConfig({
  PersistenceMode persistenceMode = PersistenceMode.file, // file | memory
  String? dbPath,
  String dbName = 'default',
  String spaceName = 'default',
  bool ignoreUnknownFields = true,
  EncryptionConfig? encryptionConfig,
  MigrationConfig? migrationConfig = const MigrationConfig(),
  int? maxPartitionFileSize,
  bool enableLog = true,
  LogLevel logLevel = LogLevel.warn,
  int? maxConcurrency,
  int? maxIoConcurrency,
  DistributedNodeConfig? distributedNodeConfig,
  int? cacheMemoryBudgetMB,
  bool? enablePrewarmCache,
  int? prewarmThresholdMB,
  // journal / batch / flush / open files...
  bool? enableJournal,
  bool? persistRecoveryOnCommit,
  RecoveryFlushPolicy? recoveryFlushPolicy,
  TransactionIsolationLevel? defaultTransactionIsolationLevel,
  Duration transactionTimeout = const Duration(minutes: 5),
  // ... cleanup TTLs ...
  int? ttlCleanupIntervalMs, // effective min 60000
  int? defaultQueryLimit,    // default 1000
  int? maxQueryOffset,       // default 10000
  int? yieldDurationMs,      // client ~8ms, server ~50ms
  bool? isServerEnvironment,
});
```

### High-value knobs

| Param | Default (typical) | Notes |
| :--- | :--- | :--- |
| `yieldDurationMs` | 8 (client) / 50 (server) | UI smoothness vs throughput |
| `defaultQueryLimit` | 1000 | Applied when query omits `limit` |
| `maxQueryOffset` | 10000 | Deep offset rejected beyond this |
| `enableJournal` | true (non-web) | Crash recovery |
| `persistRecoveryOnCommit` | true | Strong durability; false = faster, tiny crash risk |
| `ttlCleanupIntervalMs` | ≥60000 | Background TTL scan |
| `cacheMemoryBudgetMB` | auto | LRU budget |
| `maxConcurrency` | auto | Vector/crypto workers |
| `isServerEnvironment` | auto-detected | Changes partition/concurrency defaults |

`PersistenceMode.memory` is forced by `ToStore.memory()`.

## Encryption (at rest)

```dart
EncryptionConfig({
  EncryptionType encryptionType, // none | xorObfuscation | chacha20Poly1305 | aes256Gcm
  String? encodingKey,   // data key — change → background rewrite
  String? encryptionKey, // master key protecting encodingKey — rotate online
  EncryptionScope encryptionScope, // standard | full (full also encrypts vector index pages)
});
```

| Key | Role | How to change | Rewrites table data? |
| :--- | :--- | :--- | :--- |
| `encodingKey` | Encrypts table/index/log payloads | New value + `open` again | **Yes** (slow, automatic migrate) |
| `encryptionKey` | Protects `encodingKey` | `rotateEncryptionKey` | **No** (fast) |

MUST NOT hardcode production secrets; prefer OS Keychain/Keystore and pass into config.

```dart
Future<DbResult> rotateEncryptionKey({
  String? oldKey, // null if previously unset (engine default)
  required String newKey,
});
```

On success, pass the new `encryptionKey` on next `open`. Fails if wrong `oldKey` or encoding migration in progress.

## ToCrypto (value-level, no db required)

Application encodes/decodes sensitive fields before write / after read. Output Base64.

```dart
ToCrypto.encode(plaintext, {
  required Object key, // String or Uint8List; non-32-byte → SHA-256 derive
  ToCryptoType type = ToCryptoType.chacha20Poly1305, // or aes256Gcm
  Uint8List? aad, // MUST match on decode
});
ToCrypto.decode(cipherBase64, {required Object key, Uint8List? aad});
```

Use when only a few fields need protection (lower cost than full DB encryption).

## DistributedNodeConfig

For distributed primary-key / node identity (clusterId, nodeId, centralServerUrl, accessToken, autoFetchNodeInfo, thresholds). Pair with `PrimaryKeyType.timestampBased` / `datePrefixed` / `shortCode` for multi-node id generation. See README Distributed Architecture for topology narrative; agents MUST set node identity correctly before relying on distributed PK uniqueness.

## Rules

1. Do not confuse `encodingKey` vs `encryptionKey`.
2. Prefer defaults unless profiling shows need.
3. Client: keep `yieldDurationMs` low (~8); server: often ~50.
4. `ToCrypto` is orthogonal to `EncryptionConfig` — both MAY be used together.

---

<!-- DOC_ID: 13-errors-status.md -->

---
id: tostore-ai-13-errors-status
title: Results, Status Codes, and Exceptions
audience: coding-agent
source_apis:
  - DbResult
  - QueryResult
  - SchemaUpdateResult
  - TransactionResult
  - ResultStatus
  - ResultType
  - DbException
version: 3.x
status: complete
---

# Results, Status Codes, and Exceptions

## Purpose

Interpret every ToStore outcome via the unified `ResultStatus` model — whether returned on a result object or thrown on `DbException`.

## Two channels

| Channel | When | Agent rule |
| :--- | :--- | :--- |
| **Result-based** (`DbResult` / `QueryResult` / `TransactionResult` / `SchemaUpdateResult`) | Daily CRUD, query, tx, runtime schema | Business/constraint/invalid-arg → **MUST NOT** expect throw; inspect result |
| **Exception-based** (`DbException`) | Fatal / developer / critical (e.g. bad schemas on open, engine mismatch, severe migration) | Catch `on DbException`; inspect `e.statuses` |

Both channels share the same `code` / `codeKey` / `ResultType` system.

## DbResult (writes / DDL)

| Member | Meaning |
| :--- | :--- |
| `hasErrors` | Any failure (including partial batch failure) — **primary success check** |
| `successCount` / `failedCount` / `totalCount` | Counts |
| `statuses` | Ordered `ResultStatus` list (1:1 with batch items when details enabled) |
| `firstPrimaryKey` | Convenience for single insert |
| `firstStatus` / `firstType` | First / primary diagnostic |
| `message` | Human summary (batched messages capped) |

Partial batch outcome: `hasErrors && successCount > 0` (inspect `statuses` per row).

```dart
final result = await db.insert('users', data);
if (result.hasErrors) {
  print('[${result.firstType.codeKey}] ${result.message}');
} else {
  print(result.firstPrimaryKey);
}
```

```dart
final batch = await db.batchInsert('users', rows);
if (batch.hasErrors) {
  for (final s in batch.statuses) {
    if (s is ConstraintStatus) { /* fields, tableName, … */ }
    else if (s is InvalidArgumentStatus) { /* parameterName, passedValue */ }
    else if (s.type != ResultType.success) { /* codeKey / message */ }
  }
}
```

## QueryResult

| Member | Meaning |
| :--- | :--- |
| `type` / `hasErrors` | Overall status (`hasErrors` ⇔ `type != success`) |
| `data` | Rows |
| `message` | Detail |
| `retrieval` | Vector/hybrid context |
| Pagination | `hasMore`/`hasPrev`, `next`/`prev`, cursor tokens (see Query module) |

Success check: `!hasErrors` or `type == ResultType.success`.

## ResultStatus

Common serialized fields: `index`, `code`, `codeKey`, `message`.

### Class codes (routing)

| Range | Prefix | Category | Handling |
| :--- | :--- | :--- | :--- |
| `0` | SUCCESS | Success | Proceed |
| `10000–19999` | `BIZ_` | Business / constraint | Return in result; usually no throw |
| `20000–49999` | `DEV_` | Developer error | Debug may throw `DbException`; production often in result |
| `50000–79999` | `SYS_` | System | May throw when execution blocked |
| `99000–99999` | `ENG_` | Engine | Severe cases may throw |

### In-memory helpers (MUST prefer over manual ranges)

On `ResultStatus` / `ResultType`:

- `isBusinessError` / `isConstraintError`
- `isDeveloperError`
- `isSystemError`
- `isEngineError`
- `isCriticalError` — ops intervention (disk full, OOM, severe corruption, …)

### Concrete subclasses

| Type | Typical codes |
| :--- | :--- |
| `SuccessStatus` | `0` |
| `ConstraintStatus` | `10000–19999` |
| `SchemaValidationStatus` | `30000–39999` |
| `InvalidArgumentStatus` | selected `20xxx` / `22004` |
| `TransactionOperationStatus` | `50001` / `50002` |
| `GeneralStatus` | fallback |

## DbException

```dart
try {
  final db = await ToStore.open(schemas: appSchemas);
} on DbException catch (e) {
  for (final status in e.statuses) {
    if (status is SchemaValidationStatus) {
      // tableName, field, wrongValue
    }
  }
}
```

Also: `DbClosedException`. Write APIs call `DbException.checkDeveloperError(result)` so some developer errors surface as throws in debug.

## Full code tables

Authoritative leaf codes and JSON field maps: `doc/result_status_specification.md` (also linked from `llms.txt`).

## Agent checklist

1. After write: `if (result.hasErrors)` — do not assume throw.
2. Prefer `status.codeKey` / helpers over magic numbers.
3. On open/migrate failures: catch `DbException`.
4. Critical / `LogLevel.critical`: alert ops (see Admin module `setLogConfig`).

---

<!-- DOC_ID: 14-api-surface.md -->

---
id: tostore-ai-14-api-surface
title: Public API Surface
audience: coding-agent
source_apis:
  - package:tostore/tostore.dart
version: 3.x
status: complete
---

# Public API Surface

## Purpose

Exhaustive checklist of symbols agents may use from `package:tostore/tostore.dart`. If a symbol is missing here, treat it as undocumented and verify in source before use.

Import:

```dart
import 'package:tostore/tostore.dart';
```

## ToStore

### Static / factory

| Symbol | Status | Notes |
| :--- | :--- | :--- |
| `ToStore.open` | documented in Lifecycle | Preferred |
| `ToStore.memory` | documented in Lifecycle | Preferred |
| `ToStore(...)` | deprecated | MUST NOT in new code |
| `ToStore.setLogConfig` | Admin | Global logger |

### Instance — schema / tables

| Symbol | Returns |
| :--- | :--- |
| `createTable(TableSchema)` | `Future<DbResult>` |
| `createTables(List<TableSchema>)` | `Future<DbResult>` |
| `dropTable(String)` | `Future<DbResult>` |
| `clear(String)` | `Future<DbResult>` |
| `tableExists(String)` | `Future<bool>` |
| `getTableSchema(String)` | `Future<TableSchema?>` |
| `getTableNames({bool? isGlobal})` | `Future<List<String>>` |
| `getTableInfo(String)` | `Future<TableInfo?>` |
| `updateSchema(String)` | `SchemaBuilder` |
| `queryMigrationTaskStatus(String)` | `Future<MigrationStatus?>` |

### Instance — data

| Symbol | Returns |
| :--- | :--- |
| `insert(String, Map)` | `Future<DbResult>` |
| `upsert(String, Map)` | `Future<DbResult>` |
| `query(String)` | `QueryBuilder` |
| `streamQuery(String)` | `StreamQueryBuilder` |
| `update(String, [Map])` | `UpdateBuilder` |
| `delete(String)` | `DeleteBuilder` |
| `batchInsert(String, List, {allowPartialErrors, returnResultDetails})` | `Future<DbResult>` |
| `batchUpsert(...)` | `Future<DbResult>` |
| `batchUpdate(...)` | `Future<DbResult>` |
| `vectorSearch(String, {fieldName, queryVector, topK, searchDepth, distanceThreshold})` | `Future<List<VectorSearchResult>>` |

### Instance — KV

| Symbol | Returns |
| :--- | :--- |
| `setValue(String, dynamic, {ttl, expiresAt, isGlobal})` | `Future<DbResult>` |
| `getValue(String, {isGlobal})` | `Future<dynamic>` |
| `removeValue(String, {isGlobal})` | `Future<DbResult>` |
| `watchValue<T>(String, {isGlobal, defaultValue, distinct})` | `Stream<T?>` |
| `watchValues(Iterable, {isGlobal, distinct})` | `Stream<Map<String, dynamic>>` |
| `kv` | `KvStore` |

### Instance — space / admin / tx

| Symbol | Returns |
| :--- | :--- |
| `switchSpace({spaceName, keepActive})` | `Future<bool>` |
| `listSpaces()` | `Future<List<String>>` |
| `deleteSpace(String)` | `Future<DbResult>` |
| `getSpaceInfo({useCache})` | `Future<SpaceInfo>` |
| `currentSpaceName` | `String?` |
| `backup({compress, scope})` | `Future<String>` |
| `restore(String, {deleteAfterRestore, cleanupBeforeRestore})` | `Future<bool>` |
| `transaction<T>(action, {rollbackOnError, persistRecoveryOnCommit, isolation})` | `Future<TransactionResult>` |
| `rotateEncryptionKey({oldKey, required newKey})` | `Future<DbResult>` |
| `flush({flushStorage})` | `Future<void>` |
| `close({keepActiveSpace})` | `Future<void>` |
| `deleteDatabase({dbPath, dbName})` | `Future<void>` |
| `getVersion()` / `setVersion(int)` | version bookkeeping |
| `config` | `DataStoreConfig` |
| `instancePath` | `String?` |
| `status` | `DbStatus` |
| `initialize(...)` | deprecated |

## Chain builders (not all re-exported as types; obtained via ToStore)

| Type | Entry | Key methods (see topic modules) |
| :--- | :--- | :--- |
| `QueryBuilder` | `query` | select, join*, agg, peek*, watch, cache, matchVector, … |
| `UpdateBuilder` | `update` | set*, compute, allow*, inherits ChainBuilder where* |
| `DeleteBuilder` | `delete` | allowDeleteAll, allowLargeScaleOperation, where* |
| `StreamQueryBuilder` | `streamQuery` | where*, select, stream/listen |
| `SchemaBuilder` | `updateSchema` | add/remove/rename/modify field/index/FK/TTL/PK |
| `KvQueryBuilder` | `kv.query` | prefix, orderBy*, limit, offset, cursor, peek |

### ChainBuilder predicate surface

`orderByAsc`, `orderByDesc`, `limit`, `offset`, `cursor`, `where`, `whereIn`, `whereNotIn`, `whereBetween`, `whereNull`, `whereNotNull`, `whereLike`, `whereNotLike`, `whereEqual`, `whereNotEqual`, `whereGreaterThan`, `whereGreaterThanOrEqualTo`, `whereLessThan`, `whereLessThanOrEqualTo`, `whereContains`, `whereNotContains`, `whereStartsWith`, `whereEndsWith`, `whereContainsAny`, `whereEmpty`, `whereNotEmpty`, `whereTrue`, `whereFalse`, `or`, `orWhere`, `condition`, `orCondition`, `queryCondition`.

## Package exports (from `lib/tostore.dart`)

Agents MAY import these via `package:tostore/tostore.dart`:

| Export path | Public symbols (non-exhaustive nested; expand in topic modules) |
| :--- | :--- |
| `kv_query_builder.dart` | `KvQueryBuilder` |
| `chain_builder.dart` | `ChainBuilder` |
| `logger.dart` | `LogLevel`, `LogRecord`, `LogConfig`, `LogType` |
| `to_crypto.dart` | `ToCrypto`, `ToCryptoType` |
| `status_provider.dart` | `DbStatus` |
| `backup_scope.dart` | `BackupScope` |
| `config_info.dart` | `ConfigInfo` |
| `data_store_config.dart` | `DataStoreConfig`, `PersistenceMode`, `DistributedNodeConfig`, `TransactionIsolationLevel`, `RecoveryFlushPolicy`, `EncryptionType`, `EncryptionScope`, `EncryptionConfig` |
| `db_exception.dart` | `DbException`, `DbClosedException` |
| `db_result.dart` | `DbResult` |
| `db_startup_stage.dart` | `DbStartupStage`, `StartupProgressCallback` |
| `expr.dart` | `Expr`, `ExprNode`, operators/nodes |
| `memory_info.dart` | `MemoryInfo` |
| `migration_config.dart` | `MigrationConfig` |
| `migration_task.dart` | `MigrationTask`, `MigrationStatus`, `MigrationType`, `FieldSchemaUpdate`, `MigrationOperation`, … |
| `migration_write_mode.dart` | `MigrationWriteMode` |
| `query_aggregation.dart` | `QueryAggregationType`, `QueryAggregation`, `Agg`, … |
| `query_result.dart` | `QueryResult`, `VectorSearchResult`, `RetrievalChannel`, `RetrievalFusionMethod`, `RetrievalEntry`, `RetrievalContext` |
| `result_status.dart` | `ResultStatus`, `SuccessStatus`, `ConstraintStatus`, `SchemaValidationStatus`, `InvalidArgumentStatus`, `TransactionOperationStatus`, `GeneralStatus` |
| `result_type.dart` | `ResultType` |
| `schema_update_result.dart` | `SchemaUpdateResult` |
| `space_info.dart` | `SpaceInfo` |
| `table_info.dart` | `TableInfo` |
| `table_schema.dart` | `TableSchema`, `FieldSchema`, `IndexSchema`, `TableTtlConfig`, `DataType`, `IndexType`, `DefaultValueType`, `PrimaryKeyType`, `SequentialIdConfig`, `PrimaryKeyConfig`, `VectorData`, `VectorFieldConfig`, `VectorIndexType`, `VectorDistanceMetric`, `VectorIndexConfig`, `ForeignKeyCascadeAction`, `ForeignKeySchema`, … |
| `transaction_result.dart` | `TransactionResult`, `TransactionStatus`, `TransactionErrorType`, `TransactionError` |
| `query_condition.dart` | `QueryCondition` |

## KvStore methods (via `db.kv`)

`query`, `set`, `setMany`, `get`, `peekGet`, `getString`, `getInt`, `getBool`, `getMap`, list getter, `getKeys`, `exists`, `peekExists`, `remove`, `removeKeys`, `getTtl`, `setTtl`, `setIncrement`, `watchValues`, `clear`, `count`.

Builders obtained from `ToStore` (`QueryBuilder`, `UpdateBuilder`, `DeleteBuilder`, `StreamQueryBuilder`, `SchemaBuilder`) are detailed in the corresponding topic modules; this inventory lists entry points and exported types.

---

<!-- DOC_ID: 15-anti-patterns.md -->

---
id: tostore-ai-15-anti-patterns
title: Anti-Patterns
audience: coding-agent
version: 3.x
status: complete
---

# Anti-Patterns

Common mistakes when generating ToStore client code. Prefer the ✅ form.

## Lifecycle

- ❌ `final db = ToStore(...); await db.initialize();`
- ✅ `final db = await ToStore.open(...);` or `ToStore.memory(...)`

- ❌ Omitting `dbPath` on Android/iOS
- ✅ Persistent app directory via path_provider (or equivalent)

## Errors

- ❌ Assuming unique-constraint failure always throws
- ✅ Inspect `DbResult` / statuses; use `codeKey` / helpers

- ❌ Swallowing `DbException` with empty catch
- ✅ Log `message` / statuses; fail closed for critical errors

## Writes

- ❌ `await db.update('t', data);` with no where and no allow flag
- ✅ `.where(...)` or explicit `.allowUpdateAll()`

- ❌ Large delete/update without `.allowLargeScaleOperation()`
- ✅ Use the allow flag; do not use inside `transaction`

- ❌ `batchUpsert` on tables without unique constraints
- ✅ Ensure unique indexes / PK strategy supports upsert

## KV

- ❌ Passing both `ttl` and `expiresAt`
- ✅ Pass exactly one expiration mechanism (or neither)

- ❌ Expecting space-local key to be visible after `switchSpace` without `isGlobal: true`
- ✅ Use `isGlobal: true` for cross-space values (e.g. login state)

## Queries

- ❌ Unbounded `await db.query('huge_table');` on large tables
- ✅ `limit` / `cursor` / stream / aggregations

- ❌ Using high `offset` for deep pagination
- ✅ Prefer `cursor` pagination; respect `maxQueryOffset`

- ❌ Inventing SQL strings or unsupported ORM methods
- ✅ Use chain builders and documented operators only

## Security

- ❌ Calling `rotateEncryptionKey` expecting table ciphertext rewrite
- ✅ Use `encodingKey` change for data key migration; `encryptionKey` for master key rotation API

## Schema

- ❌ Treating `setVersion` as the migration engine
- ✅ Declarative schemas / `updateSchema`; version is user bookkeeping only

- ❌ Manually querying system tables
- ✅ Use public status / info / KV / user table APIs only

---

<!-- END OF GENERATED CORPUS -->