# Entity Context Directories

Entity context directories are a high-level API for generating a parallel file-system tree that mirrors your database — one directory per entity, one file per relationship type. They replace ad-hoc `defineMulti()` patterns for per-entity context generation.

## Overview

When an agent system manages many entities (agents, projects, users, tickets…), each entity often needs its own context directory:

```
context/
├── agents/
│   ├── AGENTS.md           ← index listing all agents
│   ├── alpha/
│   │   ├── AGENT.md        ← the agent row itself
│   │   ├── TASKS.md        ← tasks assigned to Alpha
│   │   ├── SKILLS.md       ← skills via junction table
│   │   └── CONTEXT.md      ← combined file (all of the above)
│   └── craft/
│       ├── AGENT.md
│       └── CONTEXT.md
└── .lattice/
    └── manifest.json       ← tracks what Lattice generated
```

Without entity context directories, you'd build this with `defineMulti()` — one definition per output file, many definitions per entity type. Entity context directories collapse the entire pattern into a single `defineEntityContext()` call.

## Basic Example

```ts
import { Lattice } from 'latticesql';

const db = new Lattice('./data/app.db');

db.define('agent', {
  columns: {
    id: 'TEXT PRIMARY KEY',
    slug: 'TEXT NOT NULL',
    name: 'TEXT NOT NULL',
    bio: 'TEXT',
  },
  render: 'default-list',
  outputFile: 'agents-flat.md',
});

db.define('task', {
  columns: {
    id: 'TEXT PRIMARY KEY',
    agent_id: 'TEXT',
    title: 'TEXT NOT NULL',
    status: 'TEXT DEFAULT "open"',
  },
  render: 'default-list',
  outputFile: 'tasks-flat.md',
});

db.defineEntityContext('agent', {
  slug: (row) => row.slug as string,

  index: {
    outputFile: 'AGENTS.md',
    render: (rows) => rows.map((r) => `- [${r.name}](${r.slug}/)`).join('\n'),
  },

  files: [
    {
      filename: 'AGENT.md',
      source: { type: 'self' },
      render: (rows) => {
        const agent = rows[0]!;
        return `# ${agent.name}\n\n${agent.bio ?? '_No bio._'}`;
      },
    },
    {
      filename: 'TASKS.md',
      source: { type: 'hasMany', table: 'task', foreignKey: 'agent_id' },
      render: (rows) => rows.map((r) => `- [ ] ${r.title}`).join('\n'),
      omitIfEmpty: true,
    },
  ],

  combined: {
    outputFile: 'CONTEXT.md',
  },

  directoryRoot: 'agents',
  protectedFiles: ['SESSION.md'],
});

await db.init();
await db.render('./context');
```

## Source Types

The `source` field on each `EntityFileSpec` determines how rows are resolved for that file.

### `self`

The entity row itself. Always exactly one row.

```ts
{
  filename: 'AGENT.md',
  source: { type: 'self' },
  render: (rows) => `# ${rows[0]!.name}`,
}
```

### `hasMany`

Rows on a related table where a foreign key points back to this entity.

```ts
{
  filename: 'TASKS.md',
  source: {
    type: 'hasMany',
    table: 'task',
    foreignKey: 'agent_id',      // FK column on the task table
    references: 'id',            // optional — PK on the agent table (default: agent's first PK)
  },
  render: (rows) => rows.map((r) => `- ${r.title}`).join('\n'),
}
```

SQL equivalent: `SELECT * FROM task WHERE agent_id = :entityPk`

### `manyToMany`

Rows from a remote table via a junction table.

```ts
{
  filename: 'SKILLS.md',
  source: {
    type: 'manyToMany',
    junctionTable: 'agent_skill',
    localKey: 'agent_id',        // FK to agent on the junction table
    remoteKey: 'skill_id',       // FK to skill on the junction table
    remoteTable: 'skill',        // table to fetch rows from
    references: 'id',            // optional — PK on the skill table
  },
  render: (rows) => rows.map((r) => `- ${r.name}`).join('\n'),
}
```

SQL equivalent: `SELECT skill.* FROM agent_skill JOIN skill ON skill.id = agent_skill.skill_id WHERE agent_skill.agent_id = :entityPk`

### `belongsTo`

A single parent row accessed via a foreign key on this entity.

```ts
{
  filename: 'ORG.md',
  source: {
    type: 'belongsTo',
    table: 'org',
    foreignKey: 'org_id',        // FK column on THIS entity's table
    references: 'id',            // optional — PK on the org table
  },
  render: (rows) => rows.length ? `Org: ${rows[0]!.name}` : '_No org._',
}
```

SQL equivalent: `SELECT * FROM org WHERE id = :entityRow.org_id`

### `custom`

A fully custom synchronous query using the raw `StorageAdapter`.

```ts
{
  filename: 'RECENT.md',
  source: {
    type: 'custom',
    query: (row, adapter) => {
      return adapter.query('event', {
        where: { agent_id: row.id },
        orderBy: 'created_at',
        orderDir: 'DESC',
        limit: 10,
      });
    },
  },
  render: (rows) => rows.map((r) => `- ${r.description}`).join('\n'),
}
```

## `omitIfEmpty`

When a source resolves zero rows, Lattice normally writes an empty file. Set `omitIfEmpty: true` to skip writing the file entirely.

```ts
{
  filename: 'TASKS.md',
  source: { type: 'hasMany', table: 'task', foreignKey: 'agent_id' },
  render: (rows) => rows.map((r) => `- ${r.title}`).join('\n'),
  omitIfEmpty: true,   // skip file if no tasks
}
```

If the entity previously had tasks and the `TASKS.md` file existed, enabling `omitIfEmpty` alone does not delete it. Pair with lifecycle cleanup to remove stale files — see [Lifecycle Management](#lifecycle-management).

## `budget`

Limit the rendered output of a file to a maximum number of characters. When the content exceeds the budget, it is truncated with a notice:

```
*[truncated — context budget exceeded]*
```

```ts
{
  filename: 'NOTES.md',
  source: { type: 'hasMany', table: 'note', foreignKey: 'agent_id' },
  render: (rows) => rows.map((r) => r.body).join('\n\n'),
  budget: 4000,
}
```

## Combined file

The `combined` option writes a single file per entity that concatenates all rendered files (joined with `\n\n---\n\n`). This is useful for LLM context injection where you want one file per entity.

```ts
db.defineEntityContext('agent', {
  slug: (row) => row.slug as string,
  files: [
    { filename: 'AGENT.md', source: { type: 'self' }, render: renderAgent },
    { filename: 'TASKS.md', source: { type: 'hasMany', ... }, render: renderTasks },
  ],
  combined: {
    outputFile: 'CONTEXT.md',
    exclude: ['TASKS.md'],    // optional — skip TASKS.md from combined output
  },
});
```

Files skipped via `omitIfEmpty` are automatically excluded from the combined output.

## Index file

The `index` option writes one file at the `directoryRoot` level (not inside per-entity subdirectories) with a listing of all entities.

```ts
db.defineEntityContext('agent', {
  slug: (row) => row.slug as string,
  index: {
    outputFile: 'AGENTS.md',
    render: (rows) => rows.map((r) => `- [${r.name}](${r.slug}/CONTEXT.md)`).join('\n'),
  },
  files: [...],
  directoryRoot: 'agents',
});
```

The `render` function receives **all entity rows** for the table — not per-entity rows.

## Custom directory path

By default Lattice writes each entity to `{directoryRoot}/{slug(row)}/`. Override with the `directory` function:

```ts
db.defineEntityContext('project', {
  slug: (row) => row.slug as string,
  directory: (row) => `projects/${row.org_slug}/${row.slug}`,
  files: [...],
});
```

## `protectedFiles`

Files listed in `protectedFiles` are never deleted by Lattice's cleanup, even if they appear to be orphaned. Use this for files that agents write into entity directories:

```ts
db.defineEntityContext('agent', {
  slug: (row) => row.slug as string,
  files: [...],
  protectedFiles: ['SESSION.md', 'NOTES.md'],
});
```

Protected files are recorded in the manifest so the protection survives across restarts.

## Reverse-Sync (v0.16+)

In agentic systems, AI agents frequently edit rendered context files directly. Without reverse-sync, those edits are destroyed on the next render cycle because Lattice overwrites files from DB state.

Reverse-sync solves this by running **before** the render phase inside `reconcile()`:

1. For each entity file with a `reverseSync` function, reads the current file from disk
2. Compares its SHA-256 hash against the last-rendered hash stored in the manifest
3. If the file was modified, calls the `reverseSync` function to parse changes back into DB updates
4. Applies those updates to the database
5. The subsequent render writes from the now-updated DB — preserving the agent's edits

### Defining a reverse-sync function

Add an optional `reverseSync` function to any `EntityFileSpec`:

```ts
db.defineEntityContext('agent', {
  slug: (row) => row.slug as string,
  files: {
    'AGENT.md': {
      source: { type: 'self' },
      render: ([r]) => `# ${r.name}\n**Role:** ${r.role}\n`,
      reverseSync: (content, entityRow) => {
        const updates: ReverseSyncUpdate[] = [];
        const nameMatch = content.match(/^# (.+)$/m);
        if (nameMatch && nameMatch[1] !== entityRow.name) {
          updates.push({
            table: 'agent',
            pk: { id: entityRow.id },
            set: { name: nameMatch[1] },
          });
        }
        return updates;
      },
    },
  },
});
```

Each `ReverseSyncUpdate` describes a single row-level mutation:

| Field   | Type                      | Description                             |
| ------- | ------------------------- | --------------------------------------- |
| `table` | `string`                  | Target table name                       |
| `pk`    | `Record<string, unknown>` | Primary key columns identifying the row |
| `set`   | `Record<string, unknown>` | Columns to update                       |

### Controlling reverse-sync behavior

Pass the `reverseSync` option to `reconcile()`:

```ts
// Default: reverse-sync enabled
await db.reconcile(outputDir);

// Dry-run: detect changes, count updates, but don't modify DB
const result = await db.reconcile(outputDir, { reverseSync: 'dry-run' });
console.log(result.reverseSync);
// { filesScanned: 5, filesChanged: 2, updatesApplied: 3, errors: [] }

// Disabled: skip reverse-sync entirely
await db.reconcile(outputDir, { reverseSync: false });
// result.reverseSync is null
```

### Edge cases

- **File deleted externally**: Skipped (no content to parse).
- **`reverseSync` throws**: Error captured in `result.reverseSync.errors`; other files still processed. DB transaction for that file is rolled back.
- **No manifest yet (first render)**: Reverse-sync has no baseline hashes — all files skipped.
- **v1 manifest (pre-0.16)**: Empty hashes — reverse-sync skips gracefully. After the first v2 render, hashes are populated and reverse-sync activates.
- **Files without `reverseSync`**: Not scanned. Agent edits to those files are still overwritten on render.

## Lifecycle Management

Over time entities are created, renamed, and deleted. Without cleanup, Lattice leaves behind directories and files for entities that no longer exist. The lifecycle system uses a manifest to track what was generated and remove orphans.

### The Manifest

After every render cycle that includes entity contexts, Lattice writes:

```
{outputDir}/.lattice/manifest.json
```

The manifest records which directories and files were generated for each entity. It is the single source of truth for lifecycle management.

### `reconcile()` — one-shot render + cleanup

Use `reconcile()` instead of `render()` when you want lifecycle management in a script or one-off invocation:

```ts
const result = await db.reconcile('./context', {
  removeOrphanedDirectories: true,
  removeOrphanedFiles: true,
});

console.log(`Removed ${result.cleanup.directoriesRemoved} stale directories`);
console.log(`Removed ${result.cleanup.filesRemoved} stale files`);
```

`reconcile()` always renders before cleaning up, so the new manifest is used to determine what is current.

### `watch()` with cleanup

In a long-running process, pass `cleanup` options to `watch()`:

```ts
const stop = await db.watch('./context', {
  interval: 5_000,
  cleanup: {
    removeOrphanedDirectories: true,
    removeOrphanedFiles: true,
  },
  onCleanup: (result) => {
    if (result.directoriesRemoved || result.filesRemoved) {
      console.log(`Cleaned ${result.directoriesRemoved} dirs, ${result.filesRemoved} files`);
    }
  },
});
```

### Dry run

Inspect what would be deleted without modifying anything:

```ts
const result = await db.reconcile('./context', {
  removeOrphanedDirectories: true,
  removeOrphanedFiles: true,
  dryRun: true,
  onOrphan: (path, kind) => console.log(`Would remove ${kind}: ${path}`),
});
```

`dryRun: true` is safe to run in CI or staging to audit cleanup without side effects.

### What gets cleaned up

**Orphaned directories** — subdirectories inside `directoryRoot` that match no current entity slug. On deletion, Lattice removes all non-protected files first; if the directory is then empty it is removed. If protected files remain, the directory is skipped (counted in `directoriesSkipped`).

**Orphaned files** — files inside a surviving entity directory that appear in the previous manifest but were not written in the current render cycle. This catches files that were removed because `omitIfEmpty` now applies or because the file spec was removed from the definition.

Files listed in `protectedFiles` (at the definition level or passed via `CleanupOptions`) are never touched.

## Reading the manifest directly

```ts
import { readManifest, manifestPath } from 'latticesql';

const manifest = readManifest('./context');
if (manifest) {
  console.log('Manifest path:', manifestPath('./context'));
  console.log('Generated at:', manifest.generated_at);

  for (const [table, entry] of Object.entries(manifest.entityContexts)) {
    const slugCount = Object.keys(entry.entities).length;
    console.log(`${table}: ${slugCount} entities in ${entry.directoryRoot}/`);
  }
}
```

## Protected Entity Contexts (v0.18+)

Protected entity contexts prevent data from leaking across context windows. When an entity context is marked `protected: true`, its data is **never auto-rendered** into other entities' files.

```ts
db.defineEntityContext('agents', {
  slug: (r) => r.slug,
  protected: true,   // ← agents can't see each other's context
  files: {
    'AGENT.md': { source: { type: 'self' }, render: ... },
  },
});

db.defineEntityContext('projects', {
  slug: (r) => r.slug,
  files: {
    'PROJECT.md': { source: { type: 'self' }, render: ... },
    // This would normally list all agents — but agents is protected,
    // so it returns empty results:
    'AGENTS.md': {
      source: { type: 'hasMany', table: 'agents', foreignKey: 'project_id' },
      render: (rows) => `Found ${rows.length} agents`,  // always 0
      omitIfEmpty: true,
    },
  },
});
```

**Rules:**

- Protected entity's own files render normally (agent A gets its own `AGENT.md`)
- Sources from other entity contexts referencing a protected table → empty `[]`
- Sources within the same protected entity referencing itself → self-only (current row)
- `self` sources are never affected
- `custom` sources bypass protection (caller has full control)

## At-Rest Encryption (v0.18+)

Entity contexts can enable transparent at-rest encryption for their table's columns.

```ts
const db = new Lattice('./secrets.db', {
  encryptionKey: process.env.MASTER_KEY,  // required
});

db.defineEntityContext('secrets', {
  slug: (r) => r.name,
  protected: true,
  encrypted: { columns: ['value'] },  // only encrypt the 'value' column
  files: { ... },
});
```

- **`encrypted: true`** — Encrypt all text columns except structural ones (`id`, `created_at`, `updated_at`, `deleted_at`)
- **`encrypted: { columns: ['value', 'notes'] }`** — Encrypt only named columns
- Values stored as `enc:<base64(iv + authTag + ciphertext)>` using AES-256-GCM
- Plaintext values pass through unchanged on read (migration-safe)
- Rendered files contain decrypted content (encryption is at the DB layer)

## Full API reference

See [API Reference — Entity Context types](./api-reference.md#entity-context-types) for complete type signatures.