[![CI](https://github.com/jose-compu/logosdb/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/jose-compu/logosdb/actions/workflows/ci.yml) [![Python](https://github.com/jose-compu/logosdb/actions/workflows/python.yml/badge.svg?branch=main)](https://github.com/jose-compu/logosdb/actions/workflows/python.yml) [![PyPI](https://img.shields.io/pypi/v/logosdb.svg)](https://pypi.org/project/logosdb/) [![Python versions](https://img.shields.io/pypi/pyversions/logosdb.svg)](https://pypi.org/project/logosdb/) LogosDB is the SQLite for AI Agents. LogosDB is a fast semantic vector database written in C/C++ that provides approximate nearest-neighbor search over embedding vectors with associated text metadata. Authors: Jose ([@jose-compu](https://github.com/jose-compu)) **Contributing**: See [CONTRIBUTING.md](CONTRIBUTING.md) for build instructions, style guide, and PR workflow. # Features * Vectors and metadata are stored as flat binary files, memory-mapped for zero-copy reads. * Approximate nearest-neighbor search via [HNSW](https://arxiv.org/abs/1603.09320) (hnswlib), O(log n) query time. * Each vector row carries optional text and ISO 8601 timestamp metadata (JSONL sidecar). * The basic operations are `Put(embedding, text, timestamp)` and `Search(query, top_k)`. * **Timestamp range filtering**: search within a time window (e.g., "last 24 hours"). * **Multiple distance metrics**: inner product, cosine similarity (auto-normalized), or L2 Euclidean. * Bulk vector access for direct tensor construction (e.g. loading into GPU memory). * Thread-safe API on a single open handle: an internal mutex serializes operations that touch the index and metadata (including search). * Crash recovery: HNSW index is automatically backfilled from the append-only vector store on open. * Scales to millions of vectors. * **Framework integrations**: LangChain and LlamaIndex VectorStore adapters. * **MCP server**: first-class [Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview) integration via `logosdb-mcp-server`. # Documentation The public interface is in `include/logosdb/logosdb.h`. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning. **Planning a deployment?** See [docs/sizing.md](docs/sizing.md) for disk/RAM estimates based on N×dim, or run `python -m logosdb.sizing --rows 1_000_000 --dim 768`. Guide to header files: * **include/logosdb/logosdb.h**: Main interface to the DB. Start here. Contains: - C API with opaque handles and `errptr` convention (RocksDB/LevelDB style) - C++ convenience wrapper (`logosdb::DB`) with RAII and exceptions - `logosdb::Options` for HNSW tuning and distance metric selection - `logosdb::SearchHit` result struct - `logosdb_search_ts_range()` for timestamp-filtered search # Limitations * This is not a general-purpose vector database. It is purpose-built for embedding-based memory retrieval in LLM inference ([funes.cpp](../funes.cpp)). * Only a single process (possibly multi-threaded) can access a particular database at a time. * There is no client-server support built into the library. An application that needs such support will have to wrap their own server around the library. * For inner-product distance (`LOGOSDB_DIST_IP`, the default), vectors must be L2-normalized before insertion. Use `LOGOSDB_DIST_COSINE` for automatic normalization. * Embedding generation is external — the caller provides pre-computed float vectors. # Roadmap Work is tracked in [open issues](https://github.com/jose-compu/logosdb/issues?q=is%3Aissue+is%3Aopen). **1.0.0** shipped on 2026-05-17; the public C API and supported language bindings follow [Semantic Versioning](https://semver.org/) on the `1.x` line: | Bump | When | |------|------| | **Patch** (`1.0.Z`) | Bug fixes, security fixes, documentation, CI/build-only changes, and internal tooling that does **not** alter the supported contract of the public C API in [`include/logosdb/logosdb.h`](include/logosdb/logosdb.h) or the stable surface of language bindings. | | **Minor** (`1.Y.z`) | Backward-compatible additions: new functions or options, new storage or index behaviors behind explicit settings, new integrations, new platforms, and additive file-format upgrades with automatic or guided migration. | | **Major** (`2.0.0`) | Breaking changes to the public C API, on-disk format without a migration path, or binding contracts that cannot remain compatible. | Target versions and issue assignment match **[GitHub milestones](https://github.com/jose-compu/logosdb/milestones)** (each open roadmap issue has exactly one milestone). ## Milestones → issues Past releases are summarized in the [CHANGELOG](CHANGELOG) and the [closed milestones](https://github.com/jose-compu/logosdb/milestones?state=closed). ### **SHIPPED** [0.9.0](https://github.com/jose-compu/logosdb/milestone/8) — minor — throughput and scale * [#87](https://github.com/jose-compu/logosdb/issues/87) — Streaming NDJSON import/export (`logosdb_export_ndjson` / `logosdb_import_ndjson` in C / C++ / Python) with `chunk_size`, byte-offset `--checkpoint`, and `--resume`. Parquet output is a follow-up. * [#80](https://github.com/jose-compu/logosdb/issues/80) — Batch ingest v2: `logosdb_put_batch` is now chunked and WAL-aware (`LOGOSDB_BATCH_CHUNK_SIZE`, default 1024). Durability matches per-row `logosdb_put`; speed-up vs single put is workload-dependent (≈1.4× at dim=64 on the bench; HNSW per-row cost dominates at larger dims). ### **SHIPPED** [0.10.0](https://github.com/jose-compu/logosdb/milestone/9) — minor — search and metadata * [#85](https://github.com/jose-compu/logosdb/issues/85) — Hybrid retrieval mode: ANN score + lexical score fusion * [#84](https://github.com/jose-compu/logosdb/issues/84) — Filter API v2: structured metadata predicates beyond timestamp range ### **SHIPPED** [0.11.0](https://github.com/jose-compu/logosdb/milestone/10) — minor — multi-tenancy and tooling * [#86](https://github.com/jose-compu/logosdb/issues/86) — Multi-tenant namespaces: quota and isolation within one DB root * [#89](https://github.com/jose-compu/logosdb/issues/89) — Recall benchmarking utility for HNSW tuning ### **SHIPPED** [0.12.0](https://github.com/jose-compu/logosdb/milestone/11) — minor — integrations * [#78](https://github.com/jose-compu/logosdb/issues/78) — Feature: Codex plugin marketplace integration for LogosDB ### **SHIPPED** [1.0.0](https://github.com/jose-compu/logosdb/milestone/13) — first stable release * Smart chunking, UTF-8/NFC policy, path-hardening, and expanded security tests (#98, #108–#112). See [CHANGELOG](CHANGELOG) `1.0.0`. ### **SHIPPED** [1.0.1](https://github.com/jose-compu/logosdb/issues/116) — patch — release hygiene * [#116](https://github.com/jose-compu/logosdb/issues/116) — Sync `CHANGELOG`, version macros, package metadata, and post-1.0 docs. ### Open — platform / hardening (post-1.0) * [#10](https://github.com/jose-compu/logosdb/issues/10) — Windows support: abstract POSIX file I/O behind a portable layer * [#111](https://github.com/jose-compu/logosdb/issues/111) — Finish Python integration-adapter security audit (partially addressed in 1.0.0 for `vibe.py`) ## Stability (since 1.0.0) As of **1.0.0**, the project commits to stable semver for the public C API and supported language bindings: * Breaking changes to `include/logosdb/logosdb.h` or on-disk layout require a future major version, with migration notes. * Tier-1 platforms for 1.x are **Linux and macOS**; Windows remains tracked in [#10](https://github.com/jose-compu/logosdb/issues/10). * Backup/restore and doctor tooling shipped in earlier 0.x releases; further hardening continues on the 1.x line. # Getting the Source ```bash git clone --recurse-submodules cd logosdb ``` # Building This project supports [CMake](https://cmake.org/) out of the box. Quick start: ```bash mkdir -p build && cd build cmake -DCMAKE_BUILD_TYPE=Release .. && cmake --build . ``` This builds: | Target | Description | |--------|-------------| | `logosdb` | Static library (`liblogosdb.a`) | | `logosdb-cli` | Command-line tool for put, search, info | | `logosdb-bench` | Benchmark: HNSW vs brute-force, with ChromaDB comparison | | `logosdb-test` | Unit tests | # Usage (C API) ```c #include char *err = NULL; logosdb_options_t *opts = logosdb_options_create(); logosdb_options_set_dim(opts, 2048); logosdb_t *db = logosdb_open("/tmp/mydb", opts, &err); logosdb_options_destroy(opts); float vec[2048] = { /* ... unnormalized vector ... */ }; // L2-normalize for inner-product distance (returns 0 on success, -1 if zero norm) if (logosdb_l2_normalize(vec, 2048) == 0) { logosdb_put(db, vec, 2048, "My commute is 42 minutes", "2025-06-25T10:00:00Z", &err); } logosdb_search_result_t *res = logosdb_search(db, query_vec, 2048, 5, &err); for (int i = 0; i < logosdb_result_count(res); i++) { printf("#%d score=%.4f text=%s\n", i, logosdb_result_score(res, i), logosdb_result_text(res, i)); } logosdb_result_free(res); logosdb_close(db); ``` ## Search with timestamp range filter ```c #include char *err = NULL; logosdb_options_t *opts = logosdb_options_create(); logosdb_options_set_dim(opts, 2048); logosdb_t *db = logosdb_open("/tmp/mydb", opts, &err); // Search for top-5 matches within the last 24 hours logosdb_search_result_t *res = logosdb_search_ts_range( db, query_vec, 2048, 5, "2025-04-21T10:00:00Z", // from (inclusive), NULL for no lower bound "2025-04-22T10:00:00Z", // to (inclusive), NULL for no upper bound 50, // candidate_k: internal fetch multiplier (10x top_k recommended) &err); for (int i = 0; i < logosdb_result_count(res); i++) { printf("#%d score=%.4f ts=%s text=%s\n", i, logosdb_result_score(res, i), logosdb_result_timestamp(res, i), logosdb_result_text(res, i)); } logosdb_result_free(res); logosdb_close(db); ``` ## Using different distance metrics ```c #include char *err = NULL; logosdb_options_t *opts = logosdb_options_create(); logosdb_options_set_dim(opts, 2048); // Use cosine similarity (automatically normalizes vectors) logosdb_options_set_distance(opts, LOGOSDB_DIST_COSINE); // Or use L2 Euclidean distance // logosdb_options_set_distance(opts, LOGOSDB_DIST_L2); // Default is LOGOSDB_DIST_IP (inner product on L2-normalized vectors) logosdb_t *db = logosdb_open("/tmp/mydb", opts, &err); logosdb_options_destroy(opts); // For cosine: vectors are automatically normalized on put/search float vec[2048] = { /* ... unnormalized vector ... */ }; logosdb_put(db, vec, 2048, "entry", "2025-04-22T10:00:00Z", &err); logosdb_close(db); ``` # Usage (C++ wrapper) ```cpp #include #include // Basic usage with default inner-product distance logosdb::DB db("/tmp/mydb", {.dim = 2048}); // L2-normalize your vectors before insertion (required for inner-product distance) std::vector embedding = load_some_vector(); // unnormalized if (logosdb::l2_normalize(embedding)) { db.put(embedding, "My commute is 42 minutes", "2025-06-25T10:00:00Z"); } // Or use l2_normalized() to get a normalized copy auto normalized = logosdb::l2_normalized(query); auto results = db.search(normalized, 5); for (auto &r : results) { printf("id=%llu score=%.4f text=%s\n", r.id, r.score, r.text.c_str()); } ``` ## C++: Search with timestamp filter ```cpp #include logosdb::DB db("/tmp/mydb", {.dim = 2048}); // Search within a time window auto results = db.search_ts_range( query, 5, "2025-04-21T00:00:00Z", // from timestamp "2025-04-22T00:00:00Z", // to timestamp 50); // candidate_k (optional, defaults to 10x top_k) for (auto &r : results) { printf("id=%llu score=%.4f ts=%s\n", r.id, r.score, r.timestamp.c_str()); } ``` ## C++: Using cosine or L2 distance ```cpp #include // Cosine similarity - vectors are automatically normalized logosdb::DB db("/tmp/mydb", {.dim = 2048, .distance = LOGOSDB_DIST_COSINE}); // Put unnormalized vectors - they will be normalized automatically db.put(unnormalized_embedding, "entry", "2025-04-22T10:00:00Z"); auto results = db.search(query, 5); // scores are cosine similarities in [0, 1] // L2 Euclidean distance // logosdb::DB db("/tmp/mydb", {.dim = 2048, .distance = LOGOSDB_DIST_L2}); ``` # Python LogosDB ships Python bindings built with [pybind11](https://pybind11.readthedocs.io/) and [scikit-build-core](https://scikit-build-core.readthedocs.io/). Install from PyPI (binary wheels provided for Linux x86_64/aarch64 and macOS x86_64/arm64 on CPython 3.9–3.13): ```bash pip install logosdb ``` Or build from source in a clone: ```bash pip install . ``` Usage: ```python import logosdb from sentence_transformers import SentenceTransformer # Local Hugging Face embeddings model (runs on your machine) model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") dim = model.get_sentence_embedding_dimension() # Use cosine distance so LogosDB auto-normalizes vectors db = logosdb.DB("/tmp/agent_memory", dim=dim, distance=logosdb.DIST_COSINE) # Three learnings captured by an AI agent learnings = [ ("Retrying API calls with exponential backoff reduced transient failures by 42%.", "2026-05-06T09:00:00Z"), ("Splitting long tasks into smaller batches improved throughput and lowered memory spikes.", "2026-05-06T09:05:00Z"), ("Adding idempotency keys prevented duplicate writes during network retries.", "2026-05-06T09:10:00Z"), ] for text, ts in learnings: emb = model.encode(text).astype("float32") db.put(emb, text=text, timestamp=ts) # Ask a natural-language question question = "How can we avoid duplicate writes when retries happen?" q_emb = model.encode(question).astype("float32") hits = db.search(q_emb, top_k=3) for h in hits: print(f"{h.score:.4f} {h.text}") ``` ## Python: Using cosine distance (no manual normalization needed) ```python import numpy as np import logosdb # With cosine distance, vectors are automatically normalized db = logosdb.DB("/tmp/mydb", dim=128, distance=logosdb.DIST_COSINE) # No need to normalize - just put raw vectors v = np.random.randn(128).astype(np.float32) rid = db.put(v, text="unnormalized vector", timestamp="2025-04-22T10:00:00Z") # Search also works with unnormalized queries query = np.random.randn(128).astype(np.float32) hits = db.search(query, top_k=5) ``` ## Python: Batch ingest and streaming NDJSON (0.9.0) ```python import numpy as np import logosdb db = logosdb.DB("/tmp/mydb", dim=128, distance=logosdb.DIST_COSINE) # Chunked, WAL-aware batch insert (LOGOSDB_BATCH_CHUNK_SIZE controls fsync cadence). embeddings = np.random.randn(10_000, 128).astype(np.float32) ids = db.put_batch(embeddings, texts=[f"row {i}" for i in range(10_000)]) # Bounded-memory NDJSON export/import; --resume from a checkpoint on interruption. db.export_ndjson("/tmp/rows.ndjson") restored = logosdb.DB("/tmp/restored", dim=128, distance=logosdb.DIST_COSINE) restored.import_ndjson("/tmp/rows.ndjson", chunk_size=1024, checkpoint_path="/tmp/rows.cp", resume=False) ``` Run the Python tests and examples: ```bash pip install ".[test]" pytest tests/python/ python examples/python/basic_usage.py # sentence-transformers demo (optional heavy dep) pip install ".[examples]" python examples/python/sentence_transformers_demo.py ``` ## Memory-Efficient On-Prem RAG LogosDB is designed for memory-efficient retrieval-augmented generation (RAG) that runs entirely on your hardware. ### RAM Model LogosDB uses `mmap()` for zero-copy access. Your RAM usage scales with **query patterns**, not dataset size: | Dataset | Dim | Disk | Typical Query RAM | |---------|-----|------|-------------------| | 100K | 384 | 153 MB | <20 MB | | 1M | 384 | 1.5 GB | <100 MB | | 10M | 384 | 15 GB | <200 MB | The OS caches hot index pages; cold data stays on disk. No explicit loading/unloading needed. ### Quick RAG Example ```python import numpy as np import logosdb from sentence_transformers import SentenceTransformer # 1. Load model (runs locally) model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") dim = model.get_sentence_embedding_dimension() # 2. Create DB with cosine distance (auto-normalizes) db = logosdb.DB("/data/knowledge", dim=dim, distance=logosdb.DIST_COSINE) # 3. Index documents for text in documents: emb = model.encode(text) db.put(emb, text=text) # Auto-normalized with cosine distance # 4. Query (only touched pages load into RAM) query_emb = model.encode("What is HNSW?") for hit in db.search(query_emb, top_k=3): print(f"{hit.score:.4f} {hit.text}") ``` See [docs/rag-on-prem.md](docs/rag-on-prem.md) for complete guide including: - Time-sharding for infinite retention - External quantization patterns - Architecture patterns for production See [docs/sizing.md](docs/sizing.md) for detailed disk/RAM formulas and the `python -m logosdb.sizing` calculator. Run the memory-efficient RAG example: ```bash pip install ".[examples]" python examples/python/memory_efficient_rag.py ``` ## Python: LlamaIndex VectorStore ```bash pip install 'logosdb[llama-index]' ``` ```python from logosdb import LogosDBIndex from llama_index.core import Document from llama_index.core.schema import TextNode from llama_index.core.vector_stores import VectorStoreQuery import numpy as np # Create the vector store db = LogosDBIndex(uri="/tmp/mydb", dim=128) # Add nodes with pre-computed embeddings node = TextNode( text="My commute is 42 minutes", embedding=np.random.randn(128).astype(np.float32).tolist(), metadata={"timestamp": "2025-04-28T10:00:00Z"} ) db.add([node]) # Query query_emb = np.random.randn(128).astype(np.float32).tolist() query = VectorStoreQuery(query_embedding=query_emb, similarity_top_k=5) results = db.query(query) for node, score in zip(results.nodes, results.similarities): print(f"Score: {score:.4f}, Text: {node.text}") # Timestamp range filtering results = db.query(query, ts_from="2025-04-01T00:00:00Z", ts_to="2025-04-30T23:59:59Z") ``` The `LogosDBIndex` class implements LlamaIndex's `VectorStore` interface, supporting: - `add(nodes)` - Add nodes with embeddings - `delete(node_id)` - Delete by node ID - `query(VectorStoreQuery)` - Similarity search by vector - `count()` / `len(store)` - Number of live documents - Timestamp filtering via `ts_from` and `ts_to` kwargs # CLI ```bash # Database info logosdb-cli info /tmp/mydb # Search with a binary query vector file logosdb-cli search /tmp/mydb --query-file q.bin --top-k 5 # Streaming NDJSON export / import (#87) — bounded memory, optional resume logosdb-cli export /tmp/mydb --output rows.ndjson [--start-id N --end-id M] logosdb-cli import /tmp/mydb --dim 768 --input rows.ndjson \ --chunk-size 1024 --checkpoint rows.cp [--resume] ``` # Node.js ## MCP Server — Claude Code integration `logosdb-mcp-server` is a [Model Context Protocol](https://modelcontextprotocol.io/) server that exposes LogosDB to **Claude Code** (and any other MCP client) over stdio. It lets Claude index files, persist knowledge across sessions, and do semantic search without leaving the conversation. ### Claude Code: complete recipe Follow these steps in order. Claude Code spawns the MCP server with **working directory = your project root** (the folder you opened); paths in `mcp.json` and `LOGOSDB_PATH` are resolved relative to that cwd unless you use absolute paths. #### 1. Prerequisites - **Node.js** 18 or newer on your `PATH` (`node -v`, `npm -v`). For the published package you also need **`npx`**. - **Claude Code** installed and signed in ([overview](https://docs.anthropic.com/en/docs/claude-code/overview)). - Decide **where the DB lives**: `LOGOSDB_PATH` (default `./.logosdb`) is created under the project root; add the directory to `.gitignore` if you do not want it committed. #### 2. Install the server (pick one) **Option A — This repository (local `node`, no `npx`).** From the **repository root**: ```bash npm install ``` That installs the `mcp` workspace and runs **`prepare`**, which compiles TypeScript to [`mcp/dist/index.js`](mcp/dist/index.js). After you change MCP sources, run `npm run mcp:build` at the root or `npm run build` inside `mcp/`. **Option B — Any other project (published [`logosdb-mcp-server`](https://www.npmjs.com/package/logosdb-mcp-server)).** In your app’s root: ```bash npm install logosdb-mcp-server ``` You can still invoke it without a project dependency via `npx -y logosdb-mcp-server` (see Option C). #### 3. Register the MCP server in Claude Code Project-local config lives in **`.claude/mcp.json`** at the repository root. User-wide config is **`~/.claude.json`** (same `mcpServers` shape). Only one definition named `logosdb` is needed. **If you use Option A (this clone):** the repo already contains [`.claude/mcp.json`](.claude/mcp.json): ```json { "mcpServers": { "logosdb": { "command": "node", "args": ["./mcp/dist/index.js"], "env": { "LOGOSDB_PATH": "./.logosdb" } } } } ``` Open **this folder** as the Claude Code project so `./mcp/dist/index.js` resolves. If the client’s cwd is **not** the repo root, set `"args"` to an **absolute** path to `mcp/dist/index.js` (see [Installing locally — Option C](#installing-locally-without-npx)). **If you use Option B or prefer always-latest from npm (Option C):** create or merge `.claude/mcp.json`: ```json { "mcpServers": { "logosdb": { "command": "npx", "args": ["-y", "logosdb-mcp-server"], "env": { "LOGOSDB_PATH": "./.logosdb" } } } } ``` Default embeddings are **local Transformers.js** (no API keys); first run may download model weights. Optional **Ollama / OpenAI / Voyage** env vars are documented in [`mcp/README.md`](mcp/README.md#configure). #### 4. Path rules for `logosdb_index_file` The server only indexes files that resolve under **`process.cwd()`** or **`LOGOSDB_INDEX_ROOT`** (if set). Symlinks that escape those roots are rejected. Keep indexing paths inside the project you opened, or set `LOGOSDB_INDEX_ROOT` to an absolute allowed root (details in [`mcp/README.md` — Path confinement](mcp/README.md#path-confinement-logosdb_index_file)). #### 5. Reload Claude Code and verify - **Restart** Claude Code or reload MCP configuration after editing `mcp.json`. - The **logosdb** server starts on **first tool use** (stdio). - Ask the agent to run **`logosdb_list`** (or check the MCP tools panel). You should see namespaces (possibly empty) rather than a spawn error. #### 6. Index, search, delete - **Natural language:** e.g. “Index `src/` into the `code` namespace”, “Search the codebase for JWT validation”. - **Slash commands (optional):** if `.claude/commands/` is present in the project (this repo ships them), use `/index`, `/search`, `/forget` — see the table below. - **Snippet memory:** the agent can call **`logosdb_index`** for short text; **`logosdb_index_file`** chunks files and directories. Use **one embedding backend consistently** per namespace on disk (do not change model dimension on existing data). See [Environment variables](#environment-variables) and [`mcp/README.md`](mcp/README.md). #### 7. Optional: agent instructions Copy the **`CLAUDE.md`** template block from [Agent instructions (`CLAUDE.md` and similar)](#agent-instructions-claudemd-and-similar) into your project so the agent indexes and searches without being reminded every session. #### 8. Troubleshooting | Symptom | What to try | |--------|----------------| | MCP fails to start / “Cannot find module” | From the same cwd Claude uses, run `node ./mcp/dist/index.js` (Option A) or `npx -y logosdb-mcp-server` (Option C) in a terminal; fix missing `npm install` or broken path. | | Relative `./mcp/dist/index.js` not found | Use an **absolute** `args` path to `index.js`, or open the correct folder as the project root. | | `logosdb_index_file` rejects a path | Path is outside cwd / `LOGOSDB_INDEX_ROOT`; use a path inside the project or set `LOGOSDB_INDEX_ROOT`. | | Search looks wrong after changing model | New embeddings must use a **fresh** `LOGOSDB_PATH` or new namespace; dimensions must match. | **Google Antigravity:** same stdio MCP pattern (`node` + local script, or `npx` + package). Step-by-step: [`mcp/README.md` — Google Antigravity](mcp/README.md#google-antigravity). ### Claude Code slash commands This repo ships **project slash commands** under [.claude/commands/](.claude/commands/) (in addition to [`.claude/mcp.json`](.claude/mcp.json) for the MCP server): | Command | Role | |---------|------| | `/index` | Index a **file or directory** via `logosdb_index_file` with **`incremental: true`** (new/changed files only; `commands/index.md`) | | `/search` | Semantic search; optional ISO `ts_from` / `ts_to` on the MCP tool (`commands/search.md`) | | `/forget` | Delete by row `id` **or** by natural-language `query` (`commands/forget.md`) | ### Agent instructions (`CLAUDE.md` and similar) The MCP server does not index the repository by itself: **the agent must call the tools** (or you rely on slash commands / hooks). To make LogosDB feel automatic without typing `/index` every time, add a block like the following to your project’s **`CLAUDE.md`** (or any instructions file your agent reads on every session). Adjust namespaces and paths to your repo. ````markdown ## LogosDB (semantic memory via MCP) The **logosdb** MCP server is configured. Data lives on disk under `LOGOSDB_PATH` (see `.claude/mcp.json`); it **persists across sessions**. **Namespaces:** Use separate namespaces for different concerns (e.g. `code` for `src/`, `docs` for `docs/`, `decisions` for short architectural notes). Search only the namespace that matches the user’s task. **When starting substantive work on this codebase:** 1. If the user has not indexed recently and you need broad code context, call **`logosdb_index_file`** with **`incremental: true`** on the smallest useful path (e.g. `src/` or a package directory), not the whole monorepo unless asked. 2. Before answering “where is X implemented?” or similar, call **`logosdb_search`** with a tight natural-language `query`, `namespace` set appropriately, and `top_k` between **3** and **8**. Do not paste entire trees into the chat—retrieve, then read only the cited files. 3. For “what did we decide recently?” style questions, use **`logosdb_search`** with optional **`ts_from` / `ts_to`** (ISO 8601 inclusive bounds) on the `decisions` or `docs` namespace when timestamps matter. 4. When the user states a durable fact worth remembering (API contract, policy, workaround), call **`logosdb_index`** into the right namespace with concise text (timestamps are stored automatically; optional **`metadata`** can label the source). **After large refactors or dependency upgrades:** Re-run **`logosdb_index_file`** with **`incremental: true`** on affected paths so search stays aligned without duplicating chunks for unchanged files. **Deletion:** Use **`logosdb_delete`** with **`id`** from a prior search hit, or with **`query`** + optional **`match_rank`** / **`search_top_k`** to remove a semantically matched row when the user asks to forget something. ```` ### Environment variables | Variable | Default | Description | |---|---|---| | `LOGOSDB_PATH` | `./.logosdb` | Root directory for all namespace databases | | `EMBEDDING_PROVIDER` | *(local)* | Omit for Transformers.js on-device; or `ollama`, `openai`, `voyage` | | `TRANSFORMERS_MODEL` | `Xenova/all-MiniLM-L6-v2` | Local embedding model (bundled MCP path) | | `OLLAMA_*` | — | See `mcp/README.md` when using Ollama | | `OPENAI_API_KEY` | — | Required when `EMBEDDING_PROVIDER=openai` | | `VOYAGE_API_KEY` | — | Required when `EMBEDDING_PROVIDER=voyage` | | `LOGOSDB_CHUNK_SIZE` | `800` | Target characters per chunk for `"legacy"` and `"section"` chunking modes | | `LOGOSDB_CHUNK_MODE` | `auto` | Chunking strategy: `auto` (per-extension), `line` (code), `section` (markdown/rst), `legacy` (original paragraph/char). See [`mcp/README.md` — Chunking strategies](mcp/README.md#chunking-strategies). | Voyage AI (`voyage-3`, dim=1024) is Anthropic's recommended cloud embedding model: ```json "env": { "LOGOSDB_PATH": "./.logosdb", "EMBEDDING_PROVIDER": "voyage", "VOYAGE_API_KEY": "" } ``` ### Available tools | Tool | Description | |---|---| | `logosdb_index` | Embed and store a text snippet in a namespace | | `logosdb_index_file` | Chunk, embed, and store a file or tree; optional **`incremental: true`** (skip unchanged, replace changed, prune deleted under a directory). When the path lives inside a Git working tree the walker honours `.gitignore` (root + nested + `.git/info/exclude` + global excludes) by default; disable with `respect_gitignore: false` or `LOGOSDB_RESPECT_GITIGNORE=0`. | | `logosdb_search` | Semantic search; optional `ts_from` / `ts_to` (ISO 8601) for timestamp-window filter | | `logosdb_list` | List all namespaces | | `logosdb_info` | Stats for a namespace (count, dimension, path) | | `logosdb_delete` | Delete by row `id`, or by natural-language `query` (`search_top_k`, `match_rank`) | ### Installing locally (without npx) **Option A — global CLI on your machine** ```bash npm install -g logosdb-mcp-server ``` In `.claude/mcp.json`, point the server at the global binary: ```json "logosdb": { "command": "logosdb-mcp-server", "args": [], "env": { "LOGOSDB_PATH": "./.logosdb" } } ``` Ensure the directory containing the global npm binaries is on your `PATH` when Claude Code spawns the process (same shell you use for `npm install -g`). **Option B — project-local `node_modules` (no global install)** From your app repo: ```bash npm install logosdb-mcp-server ``` Then use `npx` so the binary resolves from `./node_modules`: ```json "logosdb": { "command": "npx", "args": ["-y", "logosdb-mcp-server"], "env": { "LOGOSDB_PATH": "./.logosdb" } } ``` Omit `-y` if you prefer a fixed local install only. You can also call the entry script explicitly: ```json "command": "node", "args": ["./node_modules/logosdb-mcp-server/dist/index.js"], "env": { "LOGOSDB_PATH": "./.logosdb" } ``` **Option C — development build from a LogosDB clone** ```bash cd mcp && npm install && npm run build ``` This repository’s checked-in [`.claude/mcp.json`](.claude/mcp.json) uses a **relative** path `./mcp/dist/index.js` after **`npm install` from the repo root** (workspace `prepare`). For **other** clients or if the process cwd is not the repo root, use an **absolute** path to `mcp/dist/index.js`: ```json "logosdb": { "command": "node", "args": ["/absolute/path/to/logosdb/mcp/dist/index.js"], "env": { "LOGOSDB_PATH": "./.logosdb" } } ``` The same `env` block (`LOGOSDB_PATH`, embedding variables) applies to every option. Restart Claude Code or reload MCP config after editing `.claude/mcp.json`. # Performance Here is a performance report from the included `logosdb-bench` program. The results are somewhat noisy, but should be enough to get a ballpark performance estimate. ## Setup We use databases with 1K, 10K, and 100K vectors. Each vector has 2048 dimensions (matching typical LLM embedding sizes). Vectors are L2-normalized random unit vectors. *(Numbers below are illustrative ballparks; absolute values depend on hardware and build flags. Relative HNSW vs brute-force behavior is what to focus on.)* LogosDB: version 1.0.1 CPU: Apple M-series (ARM64) Dim: 2048 HNSW M: 16, ef_construction: 200, ef_search: 50 ## Write performance put (1K vectors): ~50 µs/op (~20,000 inserts/sec) put (10K vectors): ~80 µs/op (~12,500 inserts/sec) put (100K vectors): ~120 µs/op (~8,300 inserts/sec) Each "op" above corresponds to a write of a single vector + metadata + HNSW index update. ## Search performance HNSW top-5 (1K): ~0.1 ms/query HNSW top-5 (10K): ~0.3 ms/query HNSW top-5 (100K): ~1.2 ms/query Brute-force top-5 (1K): ~0.3 ms/query Brute-force top-5 (10K): ~2.5 ms/query Brute-force top-5 (100K): ~25 ms/query HNSW maintains sub-linear scaling while brute-force grows linearly with database size. At 100K vectors, HNSW is roughly 20x faster. ## VectorDB Bench [VectorDB Bench](https://github.com/zilliztech/VectorDBBench) is the standard open-source harness for comparing vector databases at realistic scale (load, optimize, serial/concurrent search, recall). LogosDB is integrated as a first-class backend: | Location | Role | |----------|------| | `vectordb_bench/backend/clients/logosdb/` | Client: `config.py`, `logosdb.py`, `cli.py` | | `vectordb_bench/backend/clients/__init__.py` | `DB.LogosDB` enum + `init_cls` / `config_cls` / `case_config_cls` mappings | | `vectordb_bench/cli/vectordbbench.py` | CLI subcommand `logosdb` | | `pyproject.toml` | Optional extra `logosdb` (installs the Python bindings used by the client) | ### Results — `Performance1536D50K` (2026-05-17) OpenAI embeddings, **50K** vectors, **1536** dimensions, **cosine** distance. Serial search only (`--skip-search-concurrent`). Absolute timings depend on hardware; recall/NDCG are the primary cross-database comparators. | Metric | Value | |--------|-------| | Load duration | 340.4 s | | Insert duration | 339.6 s | | Optimize duration | 0.8 s | | Serial latency p99 | 4.6 ms | | Serial latency p95 | 4.0 ms | | Recall@100 | **0.9347** | | NDCG | **0.9464** | Raw JSON: `vectordb_bench/results/LogosDB/result_20260517_*.json` (under your VectorDB Bench checkout). ### Re-run From a VectorDB Bench tree with the LogosDB client installed (`pip install -e '.[logosdb]'` or equivalent): ```bash .venv/bin/python3.12 -m vectordb_bench.cli.vectordbbench logosdb \ --uri /tmp/vectordbbench_logosdb \ --case-type Performance1536D50K \ --skip-search-concurrent ``` Use a fresh `--uri` directory per run. Set `OPENAI_API_KEY` (or the embedding provider your case expects) before starting. ## Results — `glove-25-angular` (qdrant/vector-db-benchmark) Standard [qdrant/vector-db-benchmark](https://github.com/qdrant/vector-db-benchmark) harness, **1.18M × 25** vectors, **cosine** distance (angular). Serial search. | Metric | Value | |--------|-------| | Upload time | 345 s | | Mean precision@10 | 0.939 | | RPS | 9,401 | | Mean latency | 0.092 ms | | p99 latency | 0.146 ms | Source: [qdrant/vector-db-benchmark](https://github.com/qdrant/vector-db-benchmark). ## Benchmark vs ChromaDB ```bash logosdb-bench --dim 2048 --counts 1000,10000,100000 ``` | Metric | ChromaDB | LogosDB | |--------|----------|---------| | Language | Python + C (hnswlib) | Pure C/C++ | | Search algorithm | HNSW | HNSW (same hnswlib) | | Storage | SQLite + Parquet | Binary mmap + JSONL | | Startup overhead | Python runtime + deps | Zero (linked library) | | Embedding generation | Built-in (Sentence Transformers) | External (caller provides vectors) | | Target use case | General-purpose vector store | Embedded LLM inference memory | | Search latency (100K, dim=2048) | ~5-10 ms | ~1-3 ms | | Memory footprint (100K, dim=2048) | ~1.5 GB (Python + SQLite) | ~800 MB (mmap) | | Cold start | ~2-5 s (Python imports) | <10 ms | | Dependencies | Python, NumPy, SQLite, hnswlib | hnswlib (header-only, vendored) | LogosDB uses the same HNSW implementation as ChromaDB (hnswlib) but eliminates Python overhead, SQLite serialization, and Sentence Transformer coupling. The result is a leaner library optimized for the single use case of embedded semantic memory for LLM inference. # Repository contents include/logosdb/logosdb.h Public C/C++ API (start here) src/logosdb.cpp Core engine: wires storage + index + metadata src/storage.h / storage.cpp Fixed-stride binary vector file with mmap src/metadata.h / metadata.cpp Append-only JSONL text + timestamp store src/hnsw_index.h / .cpp Thin wrapper around hnswlib tools/logosdb-cli.cpp Command-line interface tools/logosdb-bench.cpp Benchmark tool tests/test_doctest.cpp C++ unit tests (doctest, incl. CLI integration) tests/test_stress.cpp Stress/property tests (opt-in via --test-suite=stress) tests/python/test_smoke.py Python smoke tests (pytest) python/src/bindings.cpp pybind11 Python bindings python/logosdb/ Python package (logosdb._core + stubs) examples/python/ Python usage examples pyproject.toml Python build/config (scikit-build-core) third_party/hnswlib/ Vendored hnswlib (header-only) package.json / package-lock.json Private npm workspace (MCP + `nodejs` logosdb); `npm install` at repo root only mcp/ MCP server (logosdb-mcp-server npm package) .claude/mcp.json Claude Code MCP config (local `node ./mcp/dist/index.js`) .claude/commands/ Slash command prompts (/index, /search, /forget) CHANGELOG Release history LICENSE MIT license text # Contributing We welcome contributions. See [CONTRIBUTING.md](CONTRIBUTING.md) for: - Building from source - Running tests and benchmarks - Code style and PR workflow Please review our [Code of Conduct](CODE_OF_CONDUCT.md) and [Security Policy](SECURITY.md). # License MIT — see [LICENSE](LICENSE) for the full text.