# How Memory Works How EverOS turns a stream of messages into durable, searchable memory — the storage stack, the path layout on disk, the write→index→read pipeline, and the consistency guarantees. This is the narrative companion to the reference docs: see [storage_layout.md](storage_layout.md) for the exact file encoding, [architecture.md](architecture.md) for the layer boundaries, and [api.md](api.md) for the HTTP contract. ## Table of contents - [The storage stack](#the-storage-stack) - [Storage paths](#storage-paths) - [How a memory is born](#how-a-memory-is-born) - [Memory types & storage strategies](#memory-types--storage-strategies) - [The cascade daemon](#the-cascade-daemon) - [The Offline Memory Engine (OME)](#the-offline-memory-engine-ome) - [Consistency model](#consistency-model) - [Zero external services](#zero-external-services) - [Operating it](#operating-it) ## The storage stack Three embedded pieces, each owning what it is best at. Markdown is the **source of truth**; the other two are **derived and rebuildable**. | Layer | Backed by | Holds | Rebuildable? | |---|---|---|---| | **Markdown + YAML frontmatter** | plain `.md` files | the memory content itself — the only portable, human-editable asset | — (it *is* the truth) | | **SQLite** (`aiosqlite`) | `.index/sqlite/*.db` | system state, audit log, the cascade queue, the boundary buffer, OME engine state | ✅ from markdown | | **LanceDB** (Arrow) | `.index/lancedb/*.lance` | vector + BM25 + scalar columns for retrieval | ✅ from markdown | !!! note "The one rule that follows from this" Delete the entire `.index/` directory and **no memory is lost** — it rebuilds from the `.md` tree. There is no separate "export"; the markdown *is* the export. (How to trigger a rebuild: [Operating it](#operating-it).) ## Storage paths The default memory root is **`~/.everos/`** (override with `EVEROS_ROOT` env var or `--root` on the CLI). Configuration lives inside the memory root as `everos.toml` (generated by `everos init`). Memory is partitioned by **`/`** *before* the user-visible directories, so different `(app, project)` spaces never share a directory or cross in search. The reserved id `"default"` materialises as `default_app` / `default_project` on disk (so a default space stays visually distinct from a user-named one). ``` ~/.everos/ ← memory root (EVEROS_ROOT) ├── default_app/ ← ("default" → default_app) │ └── default_project/ ← ("default" → default_project) │ ├── users/ ← user-visible (source of truth) │ │ └── / │ │ ├── user.md single-file (profile) │ │ ├── episodes/ │ │ │ └── episode-.md daily-log append │ │ └── .atomic_facts/ daily-log (hidden) │ │ └── atomic_fact-.md │ ├── agents/ │ │ └── / │ │ ├── .cases/ daily-log (hidden) │ │ │ └── agent_case-.md │ │ └── skills/ skill-named dir │ │ └── skill_/SKILL.md (+ references/ scripts/) │ └── knowledge/ ← shared / global │ ├── .index/ ← system-managed, rebuildable (gitignore) │ ├── sqlite/ │ │ ├── system.db state / audit / cascade queue (md_change_state) / buffer / LSN │ │ ├── ome.db Offline Memory Engine state │ │ ├── ome.aps.db APScheduler jobstore (split to avoid lock contention) │ │ └── ome.db.lock OME single-engine guard (portalocker) │ └── lancedb/ │ └── .lance/ one Arrow table per kind │ ├── ome.toml ← user-editable OME strategy overrides (hot-reloaded) └── .tmp/ atomic-write staging ``` !!! warning "Differences from older PRD-era docs" The index dir is **`.index/`** (dot-prefixed), not `_index/`. The cascade queue and LSN/audit state live in **SQLite** (`system.db`, table `md_change_state`) — there is no `.cascade.log` / `.manifest.json` file in the current implementation. The `/` nesting is real and always present (`default_app/default_project` for the default scope). There is no command literally named `reindex`, but `everos cascade rebuild` rebuilds the index from markdown (see [Operating it](#operating-it)). The path manager is [`MemoryRoot`](../src/everos/infra/persistence/memory_root.py); every path above is a property on it. `MemoryRoot.ensure()` creates the runtime dirs (`.index/{sqlite,lancedb}/`, `.tmp/`); user-visible dirs appear on first write. Config files (`everos.toml`, `ome.toml`) are created by `everos init`. ## How a memory is born A message does not become memory immediately — it accumulates, a boundary is detected, an LLM extracts a cell, writers persist markdown, and the index catches up asynchronously. ``` POST /add ──▶ unprocessed_buffer (SQLite) ← messages accumulate per (session, app, project) │ ├─ boundary detector trips ─┐ POST /flush ─────────┤ (or you force it) │ one LLM call │ ▼ │ extract MemCell ──▶ memcell row (SQLite) │ │ │ ┌──────────────┴───────────────┐ │ ▼ ▼ │ UserMemoryPipeline (sync) AgentMemoryPipeline (events only) │ writes episode .md NOW emits AgentPipelineStarted ▼ │ │ (response returns once md is on disk) │ ▼ ▼ ┌─────────────────── Offline Memory Engine (OME) ───────────────────┐ │ async strategies write derived .md: │ │ atomic_facts · summary · profile · agent cases · skills │ └───────────────────────────────┬──────────────────────────────────────┘ ▼ cascade daemon watches the .md tree ▼ md_change_state queue (SQLite, durable) ▼ rebuild LanceDB rows ──▶ searchable ``` - **`/add`** appends messages to a per-`(session_id, app_id, project_id)` buffer and returns `accumulated` (or `extracted` if the boundary tripped on this call). See [api.md](api.md). - **`/flush`** forces the boundary now (one extraction LLM call), used at the end of a chat/agent run. - **Boundary detection** (`memory/extract/boundary/BoundaryDetection`) asks the model, in one call over the buffered window, where finished episodes end. It also judges whether the trailing segment is already a complete episode: if so, that segment is closed and extracted on the same `/add`; if it is still open (the conversation may go on), it stays buffered until a later `/add` cuts it or `/flush` closes it. An unusable reply is tried three times; a session whose calls keep failing three times in a row has its buffer moved to a `quarantine` track so the session can continue. - Deployments can append requirements to the boundary, episode, profile and reflection prompts with [custom instructions](custom_instructions.md). - Episode markdown is written **synchronously** — when `/flush` returns `extracted`, the episode file is already on disk. A memcell costs two calls: a short one names the language of its conversation, then the episode is extracted in that language and copied into the files of every user sender of the memcell, each copy with a placeholder summary (the episode's opening, cut and closed with `…`) that the OME replaces with a model-written preview shortly after. - **Language.** The language call answers Chinese, Japanese or English (any other language is written in English), judging the sentences the participants write to each other rather than pasted code or documents, names, or a few words of another language mixed in. The episode stores that language, and its summary and atomic facts are written in it; the profile takes the language of the first episode it is built from and keeps it, and a Reflection merge follows the profile's language (else the language most of the merged episodes share). When the language call fails or times out, the episode model picks the language and the episode stores the one its text is written in. - Everything else (atomic facts, profile, agent cases/skills) is produced **asynchronously** by the OME — see [the OME section](#the-offline-memory-engine-ome). - The **cascade daemon** turns every `.md` write into LanceDB rows so the content becomes searchable. ## Memory types & storage strategies Eight business memory kinds today, each user-, agent-, or globally owned, each picking one of three on-disk patterns: | Kind | Owner | Dir / file | Strategy | Produced by | |---|---|---|---|---| | **episode** | user | `episodes/episode-.md` | daily-log | extraction (sync) | | **atomic_fact** | user | `.atomic_facts/atomic_fact-.md` (hidden) | daily-log | OME | | **profile** | user | `user.md` | single-file rewrite | OME | | **agent_case** | agent | `.cases/agent_case-.md` (hidden) | daily-log | OME | | **agent_skill** | agent | `skills/skill_/SKILL.md` | skill-named dir | OME (clustering) | | **knowledge_document** | global | `knowledge///index.md` | knowledge tree | knowledge service | | **knowledge_topic** | global | `knowledge///_.md` | knowledge tree | knowledge service | The three strategies: | Strategy | Shape | Why | |---|---|---| | **Daily-log append** | `-.md`, one entry appended per memory | collapses thousands of per-entry files into one file per day | | **Single-file rewrite** | a fixed filename overwritten in place | for a single evolving document (a user/agent profile) | | **Skill-named dir** | one directory per skill | a skill is a richer unit (body + optional `references/` `scripts/`) | !!! note The single-file writer also supports `agent.md` / `soul.md` / `tools.md` / `behaviors.md`, but no shipped OME strategy produces those yet — today only `user.md` is written. Detailed frontmatter and entry-id encoding live in [storage_layout.md](storage_layout.md). ## The cascade daemon The cascade subsystem keeps LanceDB in sync with the markdown tree. It runs **in-process** with the server (a coroutine started by the app lifespan), not as a separate OS daemon. 1. A native filesystem watcher (`watchdog`: FSEvents on macOS, inotify on Linux) sees a `.md` create/modify. 2. The change is enqueued in the **`md_change_state`** table (SQLite) — durable, so a crash mid-sync replays on restart. 3. A worker drains the queue at **entry-level** granularity: it diffs the file, re-embeds only changed entries (keyed by `content_sha256`), and upserts the LanceDB rows. Because markdown is the source of truth, **editing a file directly is fully supported** — open an episode in VSCode / Obsidian / Vim, change an entry, save, and the daemon re-indexes just that entry. Operate the queue with `everos cascade` ([Operating it](#operating-it)); deeper runbook in [cascade_runbook.md](cascade_runbook.md). ## The Offline Memory Engine (OME) Most memory kinds are **not** extracted on the request path — they are derived later by the OME, an in-process async strategy engine. When extraction carves a MemCell, it emits events; OME strategies pick them up and write their markdown when ready. Each memcell's episode is announced once (`EpisodeExtracted`, listing every owner's copy), so work done on the episode runs once and is shared by its owners: - `extract_atomic_facts` — single-sentence facts from an episode, extracted once and written under every owner of the episode - `summarize_episode` — the episode's display summary, written in the episode's language over the placeholder it was stored with - `split_episode_by_owner` — hands each owner's copy of an episode to the profile as an event of its own (no model call) - `extract_user_profile` — the aggregated `user.md`, one run per owner - `extract_agent_case` — a reusable agent trajectory (only when the cell is substantive enough; thin trajectories are skipped by design, and the `agent_case_skipped` log line says which check skipped them). With `[agent_memory] case_gate_enabled`, an LLM judgement of whether the work was completed correctly decides instead - `extract_agent_skill` — clusters related cases into a named skill - `trigger_profile_clustering` — groups a user's episodes into topic clusters, which agentic search and Reflection read (the episode is embedded once for all of its owners) - `trigger_skill_clustering` — triggers agent skill clustering - `reflect_episodes` (cron, default off) — offline memory consolidation. Merges fragmented episodes within a cluster into a single coherent narrative, re-extracts atomic facts, and deprecates the originals. Enable via `ome.toml`. Strategies are configurable without a code change via **`ome.toml`** at the memory root (hot-reloaded within ~2 s). Example — turn two off: ```toml [strategies.summarize_episode] enabled = false [strategies.extract_user_profile] enabled = false ``` Turning the profile off, also turn off `split_episode_by_owner`: it only feeds the profile. OME keeps its own state in `.index/sqlite/ome.db` (run records, counters) and its scheduler jobstore in `.index/sqlite/ome.aps.db` (split so the sync APScheduler writer and the async OME writer never contend for one file lock). ### Reflection (offline consolidation) The `reflect_episodes` strategy runs on a cron schedule (default `0 2 * * 1` — every Monday at 02:00, disabled by default). It selects clusters with multiple members, calls the LLM to merge their episodes into a single narrative, writes the merged episode to markdown, re-extracts atomic facts, and deprecates the originals via `deprecated_by`. The merged episode uses `parent_type=cluster` and `session_id=None`. Enable it in `ome.toml`: ```toml [strategies.reflect_episodes] enabled = true ``` !!! tip "Implication for clients" After `/flush` returns `extracted`, the **episode** is queryable soon (once cascade indexes it), but **atomic facts / profile / agent cases** appear only after their OME strategy runs — typically seconds later — and the episode's `summary` is its placeholder until `summarize_episode` replaces it. Poll / retry if you need them immediately. ## Consistency model Two paths, two guarantees: | Path | Guarantee | Detail | |---|---|---| | **Write** (`/add`, `/flush`) | **strong** | the episode `.md` is on disk before the call returns `extracted`; never blocks on LanceDB | | **Read** (`/search`, `/get`) | **eventual** | reads LanceDB, which lags md by the cascade processing time — sub-second typically, up to ~10–15 s under load | So a `/search` immediately after the `/flush` that produced a record may miss it. The markdown is durable regardless; index lag never loses data. If you need read-your-write, retry with backoff (the running server is the only index writer; `everos cascade sync` is for when no server is running). Integrity is anchored by a few invariants (details in [storage_layout.md](storage_layout.md)): the frontmatter `id` / `entry_id` is the immutable join key; `content_sha256` decides whether an entry needs re-embedding; an LSN watermark (in `system.db`) orders rebuilds; the durable `md_change_state` queue is the replayable audit trail. ## Zero external services No database server, message broker, or vector service to run. Vector ANN, full-text BM25, and scalar filtering all execute inside the **embedded LanceDB** engine in one query; SQLite is a local file. The whole stack is a single directory you can copy, back up, or check the user-visible parts of into git. !!! note There is no automatic "grep over markdown" search fallback today — if the LanceDB index is unavailable, rebuild it from markdown (it is derived and disposable) rather than relying on a degraded search path. ## Operating it The CLI ([cli.md](cli.md)) is intentionally small: | Command | What it does | |---|---| | `everos init` | generate starter config files (`everos.toml` + `ome.toml`) | | `everos server start` | run the HTTP API (cascade + OME start with it) | | `everos cascade status` | queue / LSN summary | | `everos cascade sync` | drain the cascade queue now (force md → LanceDB) | | `everos cascade fix` | list failed rows / re-enqueue retryable ones | | `everos cascade rebuild` | rebuild the whole index from markdown (drift / corruption recovery) | !!! warning "There is no `everos reindex` or `everos flush`" - **Reindex** = the index is rebuildable from markdown. To rebuild the whole index, run `everos cascade rebuild` — it drops the LanceDB tables and re-indexes from md, re-populating even entries the queue already marked `done` and preserving un-extracted buffered messages. (A bare `rm -rf /.index/lancedb` is **not** enough: the cascade queue still shows those files `done`, so the scanner skips them and the index comes back empty.) For an incremental catch-up, use `everos cascade sync`. - **Flush** is an HTTP endpoint (`POST /api/v2/memory/flush`), not a CLI command — it forces *extraction* of the session buffer, which is a different thing from forcing *index sync* (`cascade sync`). ## References - [storage_layout.md](storage_layout.md) — exact file encoding, frontmatter chassis, entry-id format, atomic-write semantics - [architecture.md](architecture.md) — DDD layers and dependency rules - [api.md](api.md) — the HTTP contract (`/add` `/flush` `/search` `/get`) - [cascade_runbook.md](cascade_runbook.md) — operating the sync queue