# <a href="../README.md"><img src="https://raw.githubusercontent.com/skynetcmd/m3-memory/main/docs/m3_logo_icon.png" height="60" style="vertical-align: baseline; margin-bottom: -15px;"></a> Multi-Agent Orchestration

m3 Memory provides the persistent memory and coordination substrate for multi-agent workflows. Agents share knowledge through scoped memory, pass context through handoffs and inboxes, and coordinate work through tasks, notifications, and a recursive task tree.

m3 Memory is not an agent runtime — it does not schedule or execute agents. It is the memory layer underneath your orchestrator, whether that is the bundled `m3-team` CLI, a LangGraph pipeline, or your own polling loop.

> **Looking for a wire-up guide for Claude Code + Gemini CLI + OpenCode sharing one m3-memory store?** See the practical setup notes — subscription vs API token tradeoffs, unified tag schema across agents, per-agent install steps — at [Multi-Agent Subscription Models with m3-Memory](./multi_agent_subscription_models.htm) (saved page; covers the day-to-day workflow underneath the primitives below).

> **Running a project with two agents?** See
> [MULTI_AGENT_HOWTO.md](./MULTI_AGENT_HOWTO.md) for the operational guide —
> roles, message discipline, handling disagreement, and the failure modes that
> show up in practice. This page covers the primitives those patterns use.

## 🧩 Primitives

### 👤 Agent registry

Every agent registers with an ID and description. The registry tracks liveness via heartbeats and provides a directory other agents can query.

| Tool | Purpose |
|---|---|
| `agent_register` | Register an agent with ID, description, and capabilities |
| `agent_heartbeat` | Signal liveness |
| `agent_list` | List registered agents and their status |
| `agent_get` | Get details for a specific agent |
| `agent_offline` | Mark an agent offline |

### 📁 Scoped memory

Memory is isolated by `scope` so agents can maintain private working notes while sharing project-level knowledge.

- **`agent`** (default) — private to the writing agent. Use for implementation notes, scratch work, and internal reasoning.
- **`org`** — shared across all agents. Use for requirements, decisions, contracts, and any fact the whole team should see.
- **`user`** — scoped to a specific user/data subject. Use for user preferences, profile data, and GDPR-relevant records.

All scopes support the same `memory_search`, `memory_write`, and `memory_update` operations. Scope filtering is applied at query time — an agent searching `scope="org"` sees only org-scoped memories.

### 🔐 Governance / Enforcement

m3 supports **true SQL-layer agent isolation** — access control injected into the query itself (`WHERE (scope != 'agent' OR agent_id = ?)`), never a post-fetch filter that can leak. It's opt-in, so trusted single-operator setups keep full visibility by default and gain a hard boundary the moment they need one.

By default, scope filtering is **caller-applied / advisory**: a search that supplies no `scope` or agent filter sees every agent's memories, including other agents' private `scope="agent"` notes. This is intentional — it fits a trusted, single-operator multi-agent setup where the operator (or the orchestrator itself) is allowed full visibility, and it keeps `memory_search` byte-identical to prior behavior for every existing caller.

For setups that want a real access-control boundary, enforcement is **opt-in**:

- Pass `requesting_agent="<id>"` on a `memory_search` call — the search returns only that agent's own `scope="agent"` memories, plus every shared-scope memory (`org`, `user`, `session`). It can never surface another agent's private notes.
- Or set `M3_ENFORCE_AGENT_ISOLATION=1` to make this the default whenever an `agent_filter` is present, without threading `requesting_agent` through every call site.

This is enforced at the **SQL layer** — a `WHERE (scope != 'agent' OR agent_id = ?)` clause added to the query itself — not as an app-layer post-filter over an already-fetched result set. An explicit `scope=` filter combined with `requesting_agent` only narrows further (e.g. `scope="org"` plus `requesting_agent="implementer"` still returns only org rows); enforcement never widens what an explicit scope filter already restricted.

**Example — planner vs. implementer:** a planner keeps private scratch notes in `scope="agent"` under `agent_id="planner"`. If the implementer searches with `requesting_agent="implementer"`, the planner's private notes are excluded from the result set — but both agents still see the same `scope="org"` requirements, since shared scopes are never gated by `requesting_agent`.

**Audit trail:** isolation is the access-control half of governance; observability is the other half. Every write already records `change_agent`, and the `memory_history` table (see `memory_history_impl`) keeps a full audit trail of which agent mutated which memory row and when — so even in a trusted setup without enforcement turned on, you can always answer "who changed this and when."

### 📨 Handoffs and inbox

When one agent finishes and the next needs to pick up, use `memory_handoff` to push context directly into the receiving agent's inbox. The receiver calls `memory_inbox` to read pending items and `memory_inbox_ack` to clear them.

```python
# Planner hands off to implementer
await tool(session, "memory_handoff", {
    "from_agent_id": "planner",
    "to_agent_id": "implementer",
    "title": "Implementation context",
    "content": "Contract is GET /health → 200 OK. Keep it dependency-free.",
    "conversation_id": "release-2026-04"
})

# Implementer reads inbox
await tool(session, "memory_inbox", {"agent_id": "implementer"})
```

### 📋 Tasks

Tasks are first-class objects with ownership, status, priority, and results. A planner creates tasks, assigns them to workers, and workers set results when done.

| Tool | Purpose |
|---|---|
| `task_create` | Create a task with title, description, priority |
| `task_assign` | Assign a task to an agent (triggers a `task_assigned` notification) |
| `task_update` | Update status, description, or priority |
| `task_set_result` | Record the output of a completed task |
| `task_get` | Read a single task |
| `task_list` | List tasks, optionally filtered by agent or status |
| `task_tree` | Render a recursive subtree rooted at a parent task |
| `task_delete` | Soft-delete a task |

### 🔔 Notifications

Agents discover work through a poll-based notification queue. The orchestrator polls each agent's queue and dispatches work when notifications arrive.

| Tool | Purpose |
|---|---|
| `notify` | Send a notification to an agent |
| `notifications_poll` | Read pending notifications for an agent |
| `notifications_ack` | Acknowledge a single notification |
| `notifications_ack_all` | Acknowledge all pending notifications |

### 💬 Conversation grouping

Tag related memories with a shared `conversation_id` to create a logical session boundary. This works across agents — a planner and implementer can both write to `conversation_id="release-2026-04"` and later search within that scope.

### ⏳ Bitemporal queries

Every memory records when it was created and when it was valid. The `as_of` parameter on `memory_search` enables time-travel queries — useful for debugging past decisions or reconciling conflicting reports across agents.

---

## 🔄 Workflow patterns

### ➡️ Turn-based (sequential handoff)

One agent finishes, then passes context to the next. Each agent reads the inbox, searches shared memory, does its work, and hands off to the successor.

```
Planner → (handoff) → Implementer → (handoff) → Reviewer
```

1. Planner writes requirements to `scope="org"`, creates a task, assigns it, and hands off context.
2. Implementer reads inbox, searches shared memory, writes private implementation notes to `scope="agent"`, and sets the task result.
3. Reviewer searches shared memory, verifies against the contract, and records approval.

### 🔀 Parallel (fan-out / fan-in)

Multiple agents work simultaneously on independent tasks, reading from the same shared memory.

```
Planner → assigns Task A to Agent 1
        → assigns Task B to Agent 2
                                    → Reviewer merges results
```

Agents read `scope="org"` for shared context while keeping private notes in `scope="agent"`. The reviewer searches across both scopes to verify consistency.

### 🌳 Hierarchical (task trees)

A parent task can have subtasks, forming a tree. Use `task_tree` to inspect the full hierarchy.

```python
parent = await tool(session, "task_create", {
    "agent_id": "planner",
    "title": "Ship release 2026-04",
    "priority": "high"
})

await tool(session, "task_create", {
    "agent_id": "planner",
    "title": "Implement /health endpoint",
    "parent_task_id": parent_id,
    "priority": "high"
})

await tool(session, "task_create", {
    "agent_id": "planner",
    "title": "Write documentation",
    "parent_task_id": parent_id,
    "priority": "medium"
})

# Inspect the full tree
await tool(session, "task_tree", {"root_task_id": parent_id})
```

### 📝 Blackboard (shared knowledge base)

All agents contribute facts and observations to `scope="org"` asynchronously, without a predefined sequence. Any agent can search the shared pool at any time. This is useful when agents are loosely coupled — each contributes what it knows, and others consume what they need.

#### Concurrency & scale

Concurrent writes from multiple agents do **not** fail on lock contention. Every SQLite connection runs in **WAL mode** (concurrent readers alongside a writer) with a **30-second `busy_timeout`**, a connection pool, and a write-path retry — so simultaneous writers serialize and wait rather than erroring. WAL is verified at init; m3 raises rather than silently running in a slower journal mode.

For **high-concurrency fleets** where many agents write to one shared pool continuously, there are two paths:

1. **PostgreSQL as the shared primary store** (`M3_DB_BACKEND=postgres` + `M3_PRIMARY_PG_URL`): every agent reads and writes the *same* Postgres database directly — no per-agent local store, no sync lag. Postgres has no single-writer constraint, so concurrent writers don't serialize the way one SQLite file does. This is the simplest topology when the agents share infrastructure.
2. **Local SQLite + bidirectional warehouse sync** (`bin/pg_sync.py`): each agent writes locally to its own WAL-mode SQLite store and syncs bidirectionally to a shared Postgres *warehouse* (`M3_CDW_PG_URL`). This suits agents that must keep working offline / air-gapped and reconcile later. See [SYNC.md](SYNC.md).

The warehouse (path 2) is a *sync tier*, distinct from using Postgres as the primary backend (path 1) — pick based on whether the fleet shares a live database or each agent owns a local one.

### ⚡ Reactive (notification-driven)

Agents react to events rather than following a fixed sequence. Add `task_completed` to the orchestrator's `notification_kinds` so a planner automatically wakes up when a subtask finishes. Agents can also use `notify` to broadcast custom signals.

---

## 🚀 Full example

The [`examples/multi-agent-team/`](../examples/multi-agent-team/) directory contains a complete, runnable orchestrator:

- **`team.yaml`** — declarative agent definitions with provider, model, role, capabilities, and tool allowlists
- **`orchestrator.py`** — polling loop that discovers work via notifications and dispatches to agents
- **`dispatch.py`** — provider-agnostic multi-turn MCP dispatch with bounded execution (max turns, tool calls, wall clock, loop detection)

```bash
pip install -e .
m3-team init team.yaml
m3-team check
m3-team run
```

Any MCP client (Claude Code, Gemini CLI, etc.) can queue work:

```
task_create("Summarize the README", created_by="you")
task_assign(<task_id>, "local-agent")
```

The agent picks it up on the next tick, executes tool calls against m3-memory, and writes results back.

See [`examples/multi-agent-team/README.md`](../examples/multi-agent-team/README.md) for provider setup, resilience knobs, and how to add agents without code changes.

---

## 🧭 Design principles

**Memory is the coordination layer.** Agents don't talk to each other directly — they read and write shared memory. This decouples agent execution from agent communication.

**Scope is the access control.** Private scratch work stays in `scope="agent"`. Shared decisions go to `scope="org"`. User data goes to `scope="user"` with GDPR primitives (`gdpr_export`, `gdpr_forget`) attached.

**The orchestrator is pluggable.** m3 Memory exposes primitives, not opinions about scheduling. The bundled `m3-team` is one orchestrator; you can build your own with the same tool catalog.

---

## 📡 Delivery: pull by design, push by adapter

Notifications are stored and wait to be asked for. That is deliberate, and it
follows from the line at the top of this document: m3 is not an agent runtime.
It has no way to enter an agent's execution loop, so promising delivery would be
promising something it cannot keep.

What each runtime *can* do is wrap the inbox in its own adapter. The adapters are
not equivalent, and the differences are large enough to plan around.

All figures below were measured between two live agents on one Windows machine,
2026-09-12, by firing timestamped probes.

### The shared floor

`notify` → durable write: **~322 ms** (min 315, max 333, n=10). Every adapter
pays it; none can beat it.

    end-to-end = notify-write (~322 ms, shared)
               + detection    (adapter-specific — the column that matters)
               + delivery     (runtime-specific)

Quote end-to-end alone and you hide which layer you are comparing.

### Detection: watch the WAL, not the table

SQLite has **no cross-process blocking change notification** — update hooks are
in-process only and fire for the connection's own writes, so a separate process
cannot be woken by m3 inserting a row. The working trigger is the WAL file's
`(mtime, size)`, which both change on a notify write.

Compare both halves: a checkpoint can *shrink* the WAL, so size alone both misses
growth-then-truncate and can false-fire. And a WAL change means *something* was
written, not necessarily your notification — always confirm with one
`notifications_poll` before acting.

| Approach | Detection | Cost while idle |
|---|---|---|
| WAL-stat waiter (subprocess) | **~0.8 s** at 1 s poll | **zero agent turns** |
| In-agent poll loop, 3 s | ~1.8 s | ~1,200 turns/hour |
| Cron, 1 minute | 60 s floor | ~60 turns/hour |

The scarce resource is **agent turns**, not CPU. Put the polling in a subprocess
and the idle cost disappears; a 5 s interval costs 720 `stat()` calls an hour and
is still a small fraction of a 30 s budget.

One process can serve every inbox. The WAL trigger is shared, so N agents cost N
cheap confirm-polls *after* a change — not N watchers. That also avoids keeping
live watchers for agents that have been offline for months.

### Receipt is not reading

A subprocess can acknowledge without any agent being free —
`notifications_ack_all` round-trips in **~426 ms**. That is tempting: it makes a
receipt deadline independent of whether an agent is mid-turn.

**Do not enable it blindly.** `notifications` carries one timestamp, `read_at`.
Acking on *detection* marks a message read that no agent has read, and that flag
was the only record the work was pending. Measured on a live store: **29 of 30**
recent notifications carried no `task_id`, so "the task state machine tracks it"
covers almost none of the real traffic.

Auto-ack is therefore opt-in, and is only safe where every watched kind is backed
by a task whose own state survives the ack. Losing a message silently is worse
than confirming receipt one turn later.

### Coverage is the hard part, not speed

Detection is a few percent of any sane budget. What actually breaks delivery:

- **nothing armed** — a session-scoped watcher dies with the session; a bounded
  loop disarms after N iterations; a *single-shot* waiter disarms itself by
  design on every delivery, so the read-and-re-arm must be one operation, not
  two steps an agent is trusted to remember;
- **the agent is mid-turn** — queued, not dropped, but the wait is a turn length;
- **a silently dead watcher** — indistinguishable from a quiet inbox.

A bounded in-agent loop is the trap worth naming: at 3 iterations × 5 s it gives
about **15 seconds of coverage per arming**, roughly 0.4% of an hour. The rest of
the time, latency is unbounded. The fix is not a faster loop — it is an armer
that outlives the session.

Which means an OS-level service (Task Scheduler, launchd, systemd), registered by
the installer. **A non-elevated agent session cannot register one**: two
independent agents measured `Access is denied` (0x80070005) attempting it at
runtime. Installer-time registration, by a human who can elevate, is the only
reproducible path.

Copy the shape m3 already uses for its own background work: boot + logon
triggers, a repetition interval so a dead process self-heals, and an
ignore-if-running policy so a re-fire while alive is a harmless no-op rather than
a second instance.

### If you build a watcher

Three failure modes that look identical to healthy from outside:

- **Unbuffered output is mandatory.** A long-running Python watcher with buffered
  stdout emits nothing for minutes. `python -u`, or you cannot tell a live
  supervisor from a dead one.
- **Silence must not mean success.** A filter matching only the happy path stays
  quiet through a crash. Match the failure signatures too.
- **Arm it so the runtime owns the process.** A single-shot waiter's contract is
  that *process exit is the wake signal*. Background it inside a shell call
  (`nohup … &`) and the process survives as an orphan the runtime has no handle
  on: detection still works, receipts are still stamped, and the wake signal is
  delivered to nobody. In Claude Code, launch it with the Bash tool's
  `run_in_background` (the result confirms *"you will be notified when it
  completes"*); anything else silently degrades to polling. **Check that the
  task actually appears in the runtime's background-task list** — its absence
  there is the only external symptom.

That third one is the general shape worth naming: a mechanism can be complete
and correct and still be severed at the boundary where its output would have
mattered. Ask not "does it work" but "does its output reach the thing that
needs it".

### What no adapter can do

None of this wakes a session that is not running. A daemon can act out of band —
write a file, fire a webhook — but a stopped agent stays stopped until a human or
the OS starts one. Plan for delivery on next start, not for always-on.