Omni-SimpleMem: Unified Multimodal Memory for Lifelong AI Agents

Extending SimpleMem to multimodal — store, organize, and recall text, image, audio & video experiences with state-of-the-art accuracy.

Quick Start · Architecture · Results · Benchmarks · Config · Citation

Omni-SimpleMem Framework Overview

--- ## 🚀 Quick Start ### Installation ```bash git clone https://github.com/aiming-lab/SimpleMem.git cd SimpleMem/OmniSimpleMem pip install -e . ```

📦 Optional dependency groups

```bash pip install -e ".[all]" # Everything pip install -e ".[visual]" # Image/video (torch, transformers, CLIP) pip install -e ".[audio]" # Audio (soundfile, librosa) pip install -e ".[vector]" # FAISS vector search pip install -e ".[server]" # FastAPI REST server pip install -e ".[dev]" # Development (pytest) ```

### Basic Usage ```python from omni_memory import OmniMemoryOrchestrator, OmniMemoryConfig config = OmniMemoryConfig() config.embedding.model_name = "all-MiniLM-L6-v2" config.embedding.embedding_dim = 384 orchestrator = OmniMemoryOrchestrator(config=config, data_dir="./my_memory") # Store orchestrator.add_text( "User loves hiking in the Rocky Mountains every summer.", tags=["session_id:D1", "timestamp:2024-06-15"], ) # Query result = orchestrator.query("What does the user enjoy?", top_k=5) for item in result.items: print(item["summary"]) orchestrator.close() ```

🌐 REST API Server

```bash pip install -e ".[server]" export OPENAI_API_KEY=your_key_here python examples/api_server.py # Visit http://localhost:8000/docs ```

🔌 MCP Server (use Omni-SimpleMem from any MCP client)

Expose multimodal memory to Claude Desktop or any MCP client over stdio: ```bash python -m omni_mcp --data-dir ~/.omni_simplemem/mcp ``` Provides 12 tools — `omni_add_text` / `omni_add_image` / `omni_add_audio` / `omni_add_video` / `omni_add_document`, `omni_query` / `omni_answer`, plus namespace management. Media arguments accept local paths, `http(s)://`, Google Drive share links, `s3://` (S3 or MinIO) and `gs://`. Every tool takes an optional `namespace` giving each agent a fully isolated memory cluster. See [`omni_mcp/README.md`](omni_mcp/README.md) for setup, tool reference and configuration.

📝 More examples

- [`examples/quickstart.py`](examples/quickstart.py) — Basic text memory - [`examples/multimodal_memory.py`](examples/multimodal_memory.py) — Multimodal content - [`examples/api_server.py`](examples/api_server.py) — FastAPI REST server - [`omni_mcp/`](omni_mcp/) — MCP server (multimodal memory over stdio)

--- ## Highlights

📈 +411%
_{LoCoMo F1}

📈 +214%
_{Mem-Gallery F1}

⚡ 5.81 q/s
_{3.5x faster}

🧠 4 modalities
_{Text · Image · Audio · Video}

🔬 ~50 experiments
_{Fully autonomous}

📐 Full system diagram

``` Input (Text / Image / Audio / Video) │ ▼ ┌─────────────────────────────┐ │ Entropy-Driven Filtering │ ← CLIP / VAD / Jaccard └─────────────┬───────────────┘ ▼ ┌─────────────────────────────┐ │ Two-Tier Storage │ │ Hot: summaries, embeddings│ ← FAISS + BM25 index │ Cold: raw media (disk/S3) │ └─────────────┬───────────────┘ ▼ ┌─────────────────────────────┐ │ Hybrid Search │ │ Dense (FAISS) ∪ Sparse │ ← Set-union merging └─────────────┬───────────────┘ ▼ ┌─────────────────────────────┐ │ Pyramid Retrieval │ ← Token-budget aware │ Preview → Details → │ │ Evidence │ └─────────────┬───────────────┘ ▼ ┌─────────────────────────────┐ │ Knowledge Graph Traversal │ ← h-hop expansion └─────────────┬───────────────┘ ▼ LLM Answer ```

--- ## 📊 Results

LoCoMo — Overall F1

| Method | GPT-4o | GPT-4o-mini | GPT-4.1-nano | GPT-5.1 | GPT-5-nano | |:--|:--:|:--:|:--:|:--:|:--:| | Mem0 | 0.397 | 0.364 | 0.310 | 0.390 | 0.352 | | A-MEM | 0.394 | 0.357 | 0.216 | 0.385 | 0.348 | | MemGPT | 0.404 | 0.364 | 0.316 | 0.385 | 0.355 | | SimpleMem | 0.432 | 0.404 | 0.342 | 0.418 | 0.388 | | **Omni-SimpleMem** | **0.598** | **0.519** | **0.492** | **0.613** | **0.522** |

Mem-Gallery — Overall F1

| Method | GPT-4o | GPT-4o-mini | GPT-4.1-nano | GPT-5.1 | GPT-5-nano | |:--|:--:|:--:|:--:|:--:|:--:| | Mem0 | 0.298 | 0.291 | 0.268 | 0.270 | 0.283 | | A-MEM | 0.370 | 0.330 | 0.365 | 0.408 | 0.505 | | MemGPT | 0.435 | 0.398 | 0.360 | 0.425 | 0.388 | | SimpleMem | 0.535 | 0.498 | 0.518 | 0.538 | 0.522 | | **Omni-SimpleMem** | **0.797** | **0.749** | **0.780** | **0.810** | **0.787** |

### Efficiency

Efficiency Comparison

--- ## 🔬 Benchmarks Omni-SimpleMem is evaluated on two benchmarks: - [**LoCoMo**](https://github.com/snap-research/locomo) — 1,986 QA pairs across 10 conversations and 5 categories (Multi-hop, Single-hop, Temporal, Open-domain, Adversarial) - [**Mem-Gallery**](https://github.com/Mem-Gallery/Mem-Gallery) — 1,711 QA pairs from 240 dialogues across 9 multimodal categories ### Reproducing LoCoMo Results ```bash # 1. Download dataset git clone https://github.com/snap-research/locomo.git # 2. Set API key export OPENAI_API_KEY="your-openai-api-key" # 3. Run benchmark python benchmarks/locomo/run_locomo.py \ --data-path /path/to/locomo/data/locomo10.json \ --model gpt-4o -o ./locomo_results # Quick test (1 conversation, 20 QA pairs) python benchmarks/locomo/run_locomo.py \ --data-path /path/to/locomo/data/locomo10.json \ --max-conversations 1 --max-qa 20 ``` --- ## ⚙️ Configuration

Full configuration reference

```python from omni_memory import OmniMemoryConfig config = OmniMemoryConfig() # Embedding config.embedding.model_name = "all-MiniLM-L6-v2" # Local (384d) config.embedding.embedding_dim = 384 # LLM config.llm.summary_model = "gpt-4o-mini" config.llm.query_model = "gpt-4o" config.llm.temperature = 0.0 # Retrieval config.retrieval.default_top_k = 20 config.retrieval.enable_hybrid_search = True config.retrieval.enable_graph_traversal = True # Storage config.storage.base_dir = "./my_memory_data" # Unified model shortcut config.set_unified_model("gpt-4o") ```

See [`configs/`](configs/) for benchmark-specific YAML configurations. --- ## 🧪 Testing ```bash pip install -e ".[dev]" pytest tests/ -v ``` --- ## 📁 Package Structure

Click to expand

``` OmniSimpleMem/ ├── omni_memory/ # Core package │ ├── orchestrator.py # Central coordinator │ ├── app.py # FastAPI REST server │ ├── core/ # MAU, config, events │ ├── storage/ # FAISS vector store, cold storage, dedup │ ├── retrieval/ # Pyramid retriever, BM25, query processor │ ├── processors/ # Text, image, audio, video processors │ ├── triggers/ # CLIP visual & VAD audio triggers │ ├── knowledge/ # Knowledge graph & entity extraction │ ├── graph/ # Event management │ ├── parametric/ # Memory consolidation │ ├── routing/ # Query routing │ ├── evolution/ # Self-evolution │ └── utils/ # Embedding, model utilities, logging ├── configs/ # Benchmark YAML configs ├── benchmarks/ # LoCoMo & Mem-Gallery adapters ├── tests/ # 126 unit tests ├── examples/ # Usage examples ├── setup.py ├── requirements.txt └── LICENSE # Apache 2.0 ```

--- ## 🔧 Implementation Details | | Component | Details | |:--|:--|:--| | 🔎 | Dense retrieval | FAISS — all-MiniLM-L6-v2 (384d) or text-embedding-3-large (3072d) | | 🔤 | Sparse retrieval | BM25 via `rank_bm25` | | 🖼️ | Visual embeddings | OpenVision CLIP ViT-L/14 (768d) | | 🕸️ | Knowledge graph | In-memory, 7 entity types, 12 relation types | | ⚡ | Concurrency | Thread-safe (`RLock`), parallel workers | | 💾 | Storage | JSONL MAU persistence, FAISS index serialization | --- ## 🙏 Acknowledgments Omni-SimpleMem's architecture was discovered through [AutoResearchClaw](https://github.com/aiming-lab/AutoResearchClaw), an autonomous 23-stage research pipeline. The most impactful discoveries were bug fixes (+175%), prompt engineering (+188%), and architectural changes (+44%) — each exceeding all hyperparameter tuning combined. --- ## 📄 License

--- ## 📌 Citation ```bibtex @article{omnisimplemem2026, title = {{Omni-SimpleMem}: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory}, author = {Liu, Jiaqi and Ling, Zipeng and Qiu, Shi and Liu, Yanqing and Han, Siwei and Xia, Peng and Tu, Haoqin and Zheng, Zeyu and Xie, Cihang and Fleming, Charles and Ding, Mingyu and Yao, Huaxiu}, journal = {arXiv preprint arXiv:2604.01007}, year = {2026}, } ``` ---

_{Part of the SimpleMem family · Architecture discovered by AutoResearchClaw}