# DiffMem (Git-Native Agent Memory) Evaluation **Resource**: DiffMem, a git-native / file-based memory backend for AI agents **Source**: [github.com/Growth-Kinetics/DiffMem](https://github.com/Growth-Kinetics/DiffMem) **Local clone analyzed**: `/Users/florianbruniaux/Sites/divers-test/DiffMem`, HEAD `9d24d0c` **Author**: Alex / Growth Kinetics (powers the "Annabelle AI" product) **License**: claimed MIT (pyproject classifier) but no LICENSE file exists, GitHub detects none **Evaluated**: 2026-07-25 (code + health + docs, each by a dedicated agent) --- ## Score: 3/5 (Moderate, case study only) **Decision**: Integrate as a case study and a concept in `guide/core/memory-systems.md` §3.7, not as a recommended tool. The documentable idea is **LLM-agentic git-shell retrieval** (an LLM that greps and blames the repo instead of querying any index), a point in the memory design space the guide did not cover. Do not recommend adoption: bus factor one, no license file, the retrieval core is untested and being rewritten. Carry forward one sharp teaching point (match retrieval to query shape) and two reusable primitives. --- ## The premise flipped: DiffMem deleted BM25 The starting assumption (from a Perplexity survey) was "Markdown + Git + in-memory BM25". That is no longer true of the code. Commit `c4fe4a3` ("git based retrieval") removed the entire `bm25_indexer/` module (a real `rank_bm25.BM25Okapi` implementation, 114 lines, self-labeled "PoC minimal") and replaced it with an LLM-driven shell-command agent. The current README states it twice (README:9, README:345): "No vector databases, no embeddings, no BM25, just git and an LLM." The old BM25 is only recoverable via `git show c4fe4a3^:src/diffmem/bm25_indexer/indexer.py`. This matters for our own context: DiffMem is not a validation of the file-plus-BM25 approach. They tried it and walked away, toward full-LLM retrieval. For a lexical query shape (error strings, symbols) that is arguably the wrong direction. See the teaching point below. --- ## What It Is (verified against code) - **Storage**: Markdown files in a git repo, real multi-tenant layout. One central repo holds each user as an **orphan branch** `user/`, mounted via `git worktree` (`local_storage.py:83-111`). Orphan branches share no history, so isolation is a genuine git guarantee, not a folder convention. Per-user files: `.md`, `memories/{people,contexts}/*.md`, `timeline/YYYY-MM.md`, autogenerated `index.md`. Frontmatter is YAML parsed with pure PyYAML, no LLM (`frontmatter.py:23-56`). - **Retrieval**: no ranking algorithm at all. `run_retrieval_agent()` (`retrieval_agent/agent.py:197-308`) runs an LLM tool-calling loop (default 4 turns) with exactly one tool, `run(command=...)`. The model shells out to whitelisted commands (`cat, head, tail, grep, ls, wc, git log/diff/blame/show/rev-list`) through `command_router.py`, output capped 150 lines / 30KB, 10s timeout, executed via `subprocess.run(shell=True)` (`command_router.py:265`). A deterministic LLM-free baseline loads the user entity plus five recent timeline files (`baseline.py`, docstring: "No BM25, no embeddings, no LLM calls, pure file reads"). - **Temporal / differential memory**: wired, but LLM-driven. Git history is queryable data the agent pulls on demand (`git blame` to find when a fact last changed). The consolidator's `_link.py:29-56` does programmatic co-change analysis: it walks recent commits to build an entity co-occurrence graph, then an LLM proposes wikilinks. Genuine differential signal, LLM for the write. - **Machinery** (core vs optional): core is a Python library (`DiffMemory` in `api.py`) with **4 dependencies** (requests, openai, gitpython, python-dotenv). Around it: writer agent (1276 lines, largest file), consolidator (4 tools: dedupe/link/redistribute/reabsorb, ~1300 lines), a pluggable executor (`inline` default, `hatchet` durable-queue optional), an 890-line FastAPI server, pluggable ontologies (personal/corporate). The optional layers are gated behind env vars and `requirements-server.txt`. The core/server split is honest. --- ## Scoring Breakdown | Criterion | Score | Rationale | |-----------|-------|-----------| | Relevance to CC users | 3/5 | Agent memory, file-based, but OpenRouter/OpenAI-SDK driven, not CC-specific. | | Novelty vs. guide | 4/5 | LLM-agentic git-shell retrieval (no index) is a genuinely new point vs the vector/graph tools in `memory-systems.md`. | | Technical quality | 3/5 | 8255 src LOC / 5856 test LOC (0.71 ratio, healthy) and a clean core/server split, but the retrieval agent has **zero tests**, `cli.py` is declared in `pyproject.toml:62` yet does not exist (broken console script), version drift 0.4.0 vs shipped v0.5. | | Evidence quality | 2/5 | "Production, powers Annabelle" but one workload; "50-year durability" unsupported; no eval harness (the notes' Eval section is four bookmark links). Latency and cost documented honestly though. | | Maturity / health | 2/5 | Bus factor one (Alex 78% under 7 aliases), no LICENSE file, no tags, no releases, not on PyPI. 897 stars is real traction but a classic star-vs-maintainership gap. | **Overall: 3/5.** More substantial than a weekend project (real production use, 897 stars (now 898 as of 2026-07-28), 8K LOC, strong docs) but carries hard disqualifiers (bus factor one, no license) plus an untested, in-rewrite retrieval core. --- ## The one teaching point worth keeping: match retrieval to query shape DiffMem built BM25, then deleted it for full-LLM retrieval. That is a real-world signal, and the reason is instructive. DiffMem's queries are **natural-language questions about people, events, and relationships** (a conversational companion recalling "what did the user say about their sister in March"). That is a semantic query shape, where keyword BM25 underperforms and entity/temporal reasoning matters, so they escalated to an LLM that can `git blame` and reason. The inverse holds for a coding-agent retex system: the queries are **error strings, stack traces, and code symbols**, a lexical query shape where term overlap is high and BM25 is not just sufficient, it is often better than embeddings and far cheaper than an LLM loop. So DiffMem's abandonment of BM25 is not evidence that BM25 is wrong. It is evidence that **retrieval must fit the query shape**. Same substrate (files + git), opposite retrieval choice, because the queries differ. This is the sharpest, most transferable lesson from the whole analysis and belongs in the guide. --- ## Patterns worth extracting (for the guide and for a retex system) 1. **LLM-agentic git-shell retrieval** (`command_router.py`, `retrieval_agent/agent.py`). Instead of a search DSL or an index, give the LLM `grep` + `git log/blame/diff` behind a base-command whitelist, output truncation, and a timeout. About 351 lines for a very flexible retrieval surface, and `git blame`-style "when was this last true" comes free. Novel vs the guide. Caveat to document: arg-level validation is missing (only `tokens[0]` is checked before `shell=True`), so `$(...)` inside an argument is a real injection risk if ever exposed to untrusted input. 2. **Orphan-branch-per-user + worktree isolation** (`local_storage.py:83-111`). Real filesystem and history isolation across tenants with one shared object store, cleaner than "one folder per user." 3. **Frontmatter as the query surface, prose as the payload** (`frontmatter.py`). Structured queryable fields in YAML, narrative in the body. Directly maps to a retex file: `severity`, `tags`, `prevention_rule` in frontmatter, story in body. 4. (Minor) **PID-liveness lockfile with staleness reclaim** (`lock.py`) and **commit decoupled from remote backup** (`repo_manager.py:97-117`). Good minimal primitives for a file-based memory running as a service. Explicitly **not** worth copying: the no-LLM-free mode. Every read and write hits OpenRouter (`api-surface.md:9-10`), writes take 60-600s, reads 5-30s, at a per-op cost. For a retex system queried constantly during debugging, that latency and cost tax is exactly what a keyword/BM25/frontmatter-tag filter avoids. --- ## Weaknesses - **Bus factor one**: Alex / Growth Kinetics is 78% of 60 commits under 7 git aliases. If Alex stops, the project stops. - **License gap**: MIT claimed only in the pyproject classifier, no LICENSE file, GitHub detects no license. A legal blocker for adoption. - **Retrieval core untested and in flux**: the highest-blast-radius component (an LLM freely generating shell commands) has zero test coverage, and a `rebuilding-retrieval` branch is open. Do not build on a moving, untested part. - **Broken console script**: `pyproject.toml:62` declares `diffmem = "diffmem.cli:main"` but `cli.py` does not exist. `pip install -e . && diffmem` crashes. - **"Production" oversold**: production for one workload (Annabelle), the roadmap admits the PoC indexing strategy still needs hardening (too memory-intensive), "zero external dependencies" is contradicted by a mandatory LLM API, "50-year durability" has no supporting data. --- ## Integration Decisions | Item | Decision | Rationale | |------|----------|-----------| | Document LLM-agentic git-shell retrieval in `memory-systems.md` §3.7 | Done | New point in the design space, distinct from vector/graph/BM25. Includes the injection caveat. | | Add the "match retrieval to query shape" teaching point | Done | Transferable, sourced from DiffMem's BM25 removal. Sharpest lesson. | | Recommend DiffMem as a tool | Skip | Bus factor one, no license, untested in-rewrite retrieval. | | Full standalone tool entry in `memory-systems.md` §3 | Skip | Catalog-bloat reasoning; it is a case study inside §3.7, not a recommendation. | | Note orphan-branch isolation primitive | Done (§4.7) | Useful for multi-user/multi-project file memory. | | Entry in `credits.md` | Add | Growth Kinetics, MIT-claimed. | --- ## For the retex / smart-suggest-routing angle DiffMem confirms, by counter-example, that the existing **BM25 smart-suggest-routing engine** is the right tool for the retex query shape (lexical error strings), and that DiffMem is not a competitor to copy. The reusable idea from DiffMem is the **escalation tier**: BM25 or a frontmatter-tag filter as the cheap default, and an LLM-agentic git-shell pass only when the lexical match misses. That is the pragmatic hybrid, built on the engine already in production, not on DiffMem's clone-only, LLM-required, single-maintainer stack.