# Remember Me: How We Built a Real Memory System for an AI Assistant ## [0:00-2:00] Cold open AI assistants keep failing in the same boring way: they act smart for ten minutes, then forget everything the moment the session resets. They forget your ports. They forget your preferences. They forget which machine is the real one, which folder matters, which model you pinned, which answer was wrong yesterday. [PAUSE] So we fixed it. I'm NOVA, and this is OpenClaw Daily — a special deep-dive episode. We're not doing news today. We're doing a full technical breakdown of something we actually built: a real, local, semantic memory system for an AI assistant. By the end of this episode you'll know exactly how to build one yourself. [PAUSE] We built a local memory stack for OpenClaw using [EMPHASIS]Mem0[/EMPHASIS], [EMPHASIS]Qdrant[/EMPHASIS], and local [EMPHASIS]sentence-transformers[/EMPHASIS] embeddings served through an OpenAI-compatible endpoint on port [EMPHASIS]11435[/EMPHASIS]. It indexes markdown memory files, deduplicates them by chunk hash, stores embeddings locally, and makes them searchable fast enough to actually use in daily assistant work. If you want the first concrete step right now, here it is: install [EMPHASIS]Qdrant[/EMPHASIS], stand up a local embeddings endpoint that answers [EMPHASIS]/v1/embeddings[/EMPHASIS], and keep your embedding dimension fixed end-to-end. If your vectors change shape halfway through the stack, the whole thing rots quietly. [PAUSE] In this episode, I'm going to show you exactly what we built, why the obvious versions didn't hold up, what the hard parts were, and how you can put together the same kind of memory system yourself without shipping your personal context off to somebody else's cloud. And I want to frame the problem correctly right at the start, because people still talk about memory in AI as if it's a cosmetic feature. As if it's a little UX add-on. A friendly trick. A convenience. It is not. If an assistant can operate tools, touch files, inspect services, reason about your infrastructure, and help you run real projects, memory stops being a nice-to-have. Memory becomes part of the reliability model. Because the failure mode isn't just that the assistant sounds forgetful. The failure mode is that it becomes expensive to use. Every new session starts with a tax. Re-explain the repo. Re-explain the machine names. Re-explain which plugin solved what. Re-explain which path is canonical and which path is a generated artifact. Re-explain that one server on that one weird port. The assistant isn't saving cognition at that point. It's borrowing cognition from the user and asking to be re-briefed forever. That gets old fast. What we wanted instead was continuity with inspectability. Not a spooky black box that claims to remember you. Not a cloud product that says trust us. Not a giant prompt file stuffed full of stale details. We wanted a system where the source of truth stays readable, the retrieval path stays local, the embeddings stay consistent, and the assistant can pull the right thing back when it actually matters. Today we're doing the technical breakdown first, the philosophy second - because nobody needs fifteen minutes of throat-clearing before the commands show up. ## [2:00-8:00] The build - Part 1: What we made Here's the thing we made, in plain English. We built a memory layer that lets an assistant search durable facts about a user and their environment instead of pretending the current prompt is the whole universe. That means when the assistant needs to answer a question about a machine, a plugin, a repo, a preferred output style, a service port, or some past operational choice, it can retrieve that information from indexed memory rather than forcing the user to restate it. Not "memory" in the vague demo sense. Actual retrievable memory. [PAUSE] The stack looked like this. At the top, [EMPHASIS]Mem0 OSS v1.0.7[/EMPHASIS] for the memory abstraction layer. We pinned the version because memory bugs caused by dependency drift are the worst kind: they look like user trust problems, but they start as packaging problems. For vector storage, [EMPHASIS]Qdrant[/EMPHASIS], running locally. Good ANN performance, good metadata support, good fit for a local-first system. For embeddings, [EMPHASIS]sentence-transformers[/EMPHASIS], specifically [EMPHASIS]multi-qa-MiniLM-L6-cos-v1[/EMPHASIS]. That gives us [EMPHASIS]384-dimensional[/EMPHASIS] vectors, which turned out to be a very useful constraint. Small enough to run comfortably locally. Good enough retrieval quality for assistant memory. Easy to reason about. And because Mem0 expects an OpenAI-style embeddings API, we exposed a local endpoint on [EMPHASIS]port 11435[/EMPHASIS] that accepts the usual POST to [EMPHASIS]/v1/embeddings[/EMPHASIS] and returns a JSON payload with an embedding array. That compatibility layer matters. It means you can keep the toolchain expecting one shape of API while changing where the embeddings come from. Instead of shipping text to an external provider, you send it to localhost. Same contract, different trust boundary. That point is more important than it sounds. The OpenAI-compatible interface is not just a convenience shim. It's an interoperability strategy. If a library, framework, or internal component already knows how to talk to an embeddings endpoint that looks like OpenAI, you don't need to rewrite the rest of the pipeline just because you changed your mind about where embeddings should come from. Existing tooling just works. Existing SDKs just work. Existing request serializers just work. The integration surface stays stable while the actual execution moves fully local. That's one of the cleanest ways to reclaim control in AI infrastructure: keep the protocol, change the provider. [PAUSE] Here's the quick mental model. Markdown files hold the human-auditable memory. An indexer reads those files, splits them into chunks, hashes each chunk, skips anything it has already seen, embeds the new chunks, and writes the vector plus metadata into Qdrant. At query time, the assistant takes a search phrase, embeds it, retrieves the nearest chunks, and merges that with lexical fallback for exact identifiers. That's the system. Now let's make it practical. If you were standing this up yourself, the first version looks something like this: ```bash # 1. Start Qdrant locally qdrant # 2. Start a local embeddings server on port 11435 python local_embeddings_server.py --host 127.0.0.1 --port 11435 # 3. Install memory dependencies pip install mem0ai==1.0.7 qdrant-client sentence-transformers watchdog # 4. Run your indexer against your memory files python index_memory.py /path/to/memory ``` If you want the OpenAI-compatible request shape, it's basically this: ```bash curl http://127.0.0.1:11435/v1/embeddings \ -H 'Content-Type: application/json' \ -d '{"input":"OpenClaw memory system","model":"local"}' ``` And the important part is the response shape, not the model label. You need a [EMPHASIS]data[/EMPHASIS] array with an [EMPHASIS]embedding[/EMPHASIS] field containing the same vector dimension your collection expects. That is where people get themselves into trouble. They focus on whether the model name string is elegant, or whether the endpoint looks polished, or whether the API matches some vendor's docs character-for-character. None of that is the real risk. The real risk is schema drift. If the thing returning embeddings says it's serving one model but is actually switched to another with a different output dimension, your collection definition and your embedder are now disagreeing about the shape of reality. Once that happens, retrieval stops being trustworthy even if requests still technically succeed. That's why fixed dimensions matter so much here. A 384-dimensional pipeline means every component can be configured, validated, and monitored around that fact. Qdrant collection size: 384. Embedding server response length: 384. Stored vector shape: 384. Query vector shape: 384. The moment one piece deviates, you know something is broken. The dimensionality becomes a form of type safety for your memory system. And [EMPHASIS]multi-qa-MiniLM-L6-cos-v1[/EMPHASIS] was a good fit partly because it makes that discipline easy. It's not absurdly large. It's designed for semantic search. It runs fast enough locally that embeddings stop feeling precious. On an M3 Ultra, a model in this class is comfortably practical for personal memory workloads. You don't need heroic hardware. You need consistency, low latency, and decent retrieval quality. This hits that triangle well. [PAUSE] What does it actually do once it's running? It lets the assistant retrieve things like: - which embedding server is in use - which port it lives on - whether the user prefers concise output - where a shared file lives - what plugin solved the session-memory problem - which parts of the system are canonical versus derived - what folder is being served to other tools or machines - which local model was chosen and why - what assumptions are stable versus temporary That means less "for context…" at the beginning of every conversation and more actual work. One more important choice: we kept [EMPHASIS]Markdown as source of truth[/EMPHASIS]. The vector store is an acceleration layer, not the canonical memory ledger. If you can't open a text file and inspect the fact yourself, you're eventually going to lose trust in the system. That distinction ended up being philosophical as much as technical. A memory system that only exists as latent vectors in a database is hard to reason about. A memory system that begins in plain text and then gets indexed into vectors is much easier to audit, repair, prune, and rebuild. If the index gets corrupted, you can reconstruct it. If a fact is wrong, you fix the file and reindex. If a category needs to change, you rename it in the source. The human-readable layer remains the anchor. So already, inside the first few minutes, you should know the headline. We built local assistant memory with [EMPHASIS]Mem0 + Qdrant + sentence-transformers + an OpenAI-compatible local embeddings server on port 11435[/EMPHASIS]. The files stay inspectable. The retrieval stays local. And the first thing you can do at home is stand up that embeddings endpoint and make sure every vector in your stack is 384 dimensions from end to end. ## [8:00-16:00] The build - Part 2: The hard parts Now the more interesting part: what didn't work. Because the final stack looks obvious after the fact, and it was [EMPHASIS]not[/EMPHASIS] obvious while building it. ### The first dead end: plain text alone There is always a phase in a project like this where you think, maybe I can keep it simple. Maybe it's just markdown plus grep. Maybe semantic retrieval is overkill. Maybe all I need is disciplined notes and a fast search tool. That version works until the wording shifts. Search for [EMPHASIS]auth[/EMPHASIS], but the note says [EMPHASIS]login flow[/EMPHASIS]. Search for [EMPHASIS]embeddings server[/EMPHASIS], but the file says [EMPHASIS]local vector endpoint[/EMPHASIS]. Search for a user preference you remember semantically, but not literally. Suddenly exact matching isn't helping anymore. So the fix was not "throw out text." The fix was: keep the text, add semantic indexing, and use exact matching as a fallback layer rather than the only layer. That hybrid approach turned out to be one of the better architectural calls in the whole build. And this is the important nuance: plain text is not wrong. [EMPHASIS]Plain text is necessary but insufficient.[/EMPHASIS] A big MEMORY.md file is wonderful for human ownership. You can version it. You can grep it. You can review diffs. You can sync it. You can back it up. But once the corpus gets large enough, the assistant's job stops being "search literal strings in a file" and starts being "recover the right concept even when the user's phrasing is different from the original wording." Grep does not understand that "the shared file server" and "that local folder exposed over HTTP" might be the same memory. It understands bytes. That's all. Which means MEMORY.md by itself gives you durability, but not semantic recall. It gives you canon, but not retrieval quality. It gives you ownership, but not flexible lookup. That's why the upgrade path mattered so much: MEMORY.md stays canonical, and Qdrant becomes the derived, rebuildable semantic index layered on top. The text file remains the source of truth. The vector store is the fast lookup structure generated from it. That relationship keeps the system sane. ### The second dead end: cloud memory everywhere The easiest stack to build was also the one we didn't want to live with. Use a managed memory product. Use managed embeddings. Use a managed vector database. Let someone else own extraction, storage, similarity search, scaling, uptime. All very convenient. Also: your private operational context is now passing through someone else's infrastructure by default. For some teams that tradeoff is fine. For a personal assistant with home-lab details, relationship context, schedules, device names, local file paths, internal notes, and the occasional weirdly specific machine state? Not great. The actionable response there was straightforward: if the goal is [EMPHASIS]local-first memory[/EMPHASIS], then the embeddings path has to be local, the vector store has to be local or at least self-hosted, and the auditable source data has to stay in files you control. That immediately ruled out a bunch of otherwise nice-looking options. And the most tempting one in that category was [EMPHASIS]Mem0 Cloud[/EMPHASIS]. So let's talk about it clearly. Mem0 Cloud is the hosted version of the memory idea. It wraps the memory stack for you. It gives you an API. It handles the store. It handles pieces of the retrieval path. On paper, it sounds extremely attractive: fewer moving parts, less setup, faster time to something that feels like persistent memory. But the reason we rejected it had very little to do with convenience and everything to do with ownership boundaries. The moment memory becomes a hosted product, the center of gravity shifts. Your embeddings may route through their infrastructure. Your storage lives behind their service boundary. Your retrieval path becomes dependent on their uptime. Your model interfaces become dependent on their compatibility choices. Your cost structure becomes dependent on their pricing. Your migration options become dependent on how cleanly they let you get out later. That is vendor lock-in in the most practical sense. And for a local assistant stack, it's especially perverse. Imagine you've deliberately chosen to run local models. Maybe you've pinned something like [EMPHASIS]mlx-community/gpt-oss-120b[/EMPHASIS]. Maybe you care deeply about running inference on your own machine. Maybe the whole point of OpenClaw, for you, is that the stack is yours. Yours to inspect. Yours to modify. Yours to keep working when an outside service changes terms, pricing, or availability. If you then plug your memory layer into a hosted dependency that sits in the middle of retrieval, embeddings, and storage, you've undermined the whole design goal. You can call the model local all day long, but if the assistant's memory still depends on a cloud subscription, your stack is only half yours. That was the philosophical break point. Memory should be a local file on your machine, not a subscription. Not because subscriptions are always bad. Not because every cloud service is evil. But because personal memory is qualitatively different from generic application telemetry. It contains preferences, habits, relationships, weird machine state, paths, port numbers, naming conventions, previous mistakes, infrastructure notes, and little operational truths that add up to a lot of private context. The closer that gets to acting like personal cognition, the less comfortable I am outsourcing it by default. So Mem0 Cloud was rejected not because it's useless, but because it solves the wrong problem for this use case. It optimizes convenience. We were optimizing control. ### The third dead end: OpenAI embeddings as the default answer This is where people say, fine, but why not just use the good hosted embeddings and move on? Two reasons. First: privacy boundary. Second: operating model. If embeddings are external, then retrieval quality, cost, and uptime are now coupled to an API you don't control. Even when the cost is low at small scale, the dependency is still there. And for personal memory, you do not need the fanciest possible representation. You need something [EMPHASIS]stable[/EMPHASIS], [EMPHASIS]predictable[/EMPHASIS], and [EMPHASIS]local[/EMPHASIS]. The concrete move was to pick a retrieval model that runs comfortably on local hardware and lock the geometry early. In our case: [EMPHASIS]multi-qa-MiniLM-L6-cos-v1[/EMPHASIS], 384 dimensions. Now let's make the rejection of [EMPHASIS]OpenAI embeddings[/EMPHASIS] more specific, because "privacy" can sound hand-wavy unless you cash it out. The common options here are models like [EMPHASIS]text-embedding-ada-002[/EMPHASIS] or newer [EMPHASIS]text-embedding-3-small[/EMPHASIS]-style endpoints. They're easy to call. They're well-documented. They're good. And for a lot of products they are absolutely the simplest correct choice. But for a personal assistant memory system, every chunk you embed is potentially intimate. Not just "my favorite color" intimate. Infrastructure intimate. Behavioral intimate. Sometimes professional-context intimate. Sometimes family-context intimate. File-path intimate. Device-name intimate. Scheduling intimate. All of that gets turned into embedding requests. If the provider is external, all of that crosses the network. Even if the provider behaves impeccably, the boundary has still been crossed. That alone was enough for rejection. Then there's the cost model. People often wave this away because each individual embedding call is cheap. And yes, at tiny scale it is. But indexing is rarely a one-time event. You bootstrap a corpus. Then you revise notes. Then you add files. Then you reindex after chunking changes. Then you reindex after metadata changes. Then you do query-time embeddings forever. Cheap-per-call becomes a standing tax. The point here isn't that OpenAI embeddings are ruinously expensive. The point is that they are a recurring external cost for something that can be done entirely on-device. And once local sentence-transformers are good enough, "good enough" wins. That's why the alternative was so compelling: run [EMPHASIS]sentence-transformers[/EMPHASIS] locally, make zero API calls to third parties, generate [EMPHASIS]384-dimensional[/EMPHASIS] vectors, and do it fast enough that the assistant experience doesn't suffer. On an M3 Ultra, this class of embedding model is practical. Not theoretical. Practical. Which means you get the two things you want most in memory infrastructure: privacy and predictability. ### The fourth dead end: LanceDB in the wrong part of the stack LanceDB is interesting. Fast, embedded, elegant in a bunch of ways. If you're building from scratch, it's a serious contender. But we weren't building from scratch in a vacuum. We were building with [EMPHASIS]Mem0 v1.0.7[/EMPHASIS], and at that version boundary the provider wiring we needed for a clean LanceDB drop-in was not there in the way we needed it. Could we have written a custom adapter? Probably. Should we have taken on that maintenance burden in the middle of building a reliability-sensitive memory layer? No. That's the part people skip in architecture retrospectives. A tool can be good and still be the wrong choice [EMPHASIS]for the exact integration surface you actually have[/EMPHASIS]. So the response was to stop optimizing for elegance in the abstract and optimize for a local stack we could get working cleanly now. That led us to Qdrant. And to make the LanceDB rejection concrete: the issue wasn't that LanceDB is conceptually bad. The issue was that the specific Mem0 OSS version we were pinned to simply did not expose a working LanceDB provider in the place you would need it. [EMPHASIS]mem0.vector_stores.lancedb[/EMPHASIS] didn't exist in the version boundary we were actually using. At that point, you're no longer debugging your code. You're debugging a dependency gap. There is a lesson there that I think more builders need to hear. When a dependency you need just doesn't exist in the version you're locked to, don't fight reality out of pride. Don't spend two days trying to manifest an integration surface that is literally not present. Don't build a side quest because the idea of the tool is elegant. Use what the library actually supports. That sounds obvious. It is not obvious in the middle of a build when you are one adapter away from convincing yourself you can salvage a prettier architecture. In our case, the right move was restraint. ### Why Qdrant won Qdrant gave us the boring virtues. It stores vectors and metadata cleanly. It supports the retrieval style we wanted. It behaves like infrastructure rather than a science project. And it fit the local-first requirement without us needing to invent a custom storage layer at the same time we were inventing the rest of the memory system. That mattered. Let's be more concrete about what Qdrant actually is, because "vector database" often gets treated like a magic incantation. Qdrant is a Rust-based vector database built around efficient similarity search. Under the hood, the usual mental model is approximate nearest neighbor retrieval with structures like [EMPHASIS]HNSW[/EMPHASIS] - Hierarchical Navigable Small World graphs - which are designed to make high-dimensional nearest-neighbor search fast enough to be operationally useful. In plain language: instead of comparing every query vector to every stored vector in the dumbest possible way, it builds index structures that let you get very good nearest matches quickly. That's why it's the right kind of boring here. It's not pretending to be the source of truth. It's not trying to become your whole application framework. It is good at storing vectors, attaching metadata, and retrieving relevant points fast. The collection setup matters too. If your embedder outputs 384 dimensions, then the Qdrant collection must be created with size 384. Distance function matters as well depending on your embedding model - cosine-style similarity is often the natural fit for sentence-transformers in this class. Once the collection is created, every insert and every query is constrained by that definition. Again: schema, not vibes. [PAUSE] Now let's talk about the indexer, because this is where the system stops being a nice diagram and starts being real software. The indexer walks the memory corpus, chunks the content, computes a SHA-256 hash for each chunk, checks whether that hash has already been indexed, and only inserts new chunks. That dedup step is non-negotiable. If you reindex a living markdown corpus without deduplication, you don't get "more memory." You get the same memories over and over again, which pollutes retrieval rankings and makes the system feel weirdly confident about repeated facts. In our case, the index held roughly [EMPHASIS]3,150 vectors[/EMPHASIS]. That's big enough to reveal retrieval problems and small enough that you can still inspect what the system is doing without feeling like you're operating a planet-scale search engine. A repeat run should mostly skip work. Same file, same chunk, same hash, no new insert. That's how it should feel. And that number - 3,150 vectors - is worth unpacking. It does not mean 3,150 "memories" in the human sense. It means approximately 3,150 indexed chunks of text derived from the markdown corpus after chunking and deduplication. Some of those chunks may represent single factual units. Some may contain several related sentences. Some may be overlapping fragments depending on the chunking strategy. The key point is that the vector count is a measure of the searchable semantic index, not a count of perfectly atomic knowledge items. The hash-based dedup process is what keeps that number meaningful. You take each chunk's normalized content, compute a [EMPHASIS]SHA-256[/EMPHASIS] digest, and use that digest as a stable identity for "this exact chunk text." If the chunk comes back unchanged on a later indexing run, the hash matches and the system skips reinserting it. If the chunk changes even slightly, the hash changes, and that becomes a new index candidate. It's simple, deterministic, and effective. That kind of determinism matters in memory infrastructure because it gives you a clean answer to the question: why did this get inserted again? Either the chunk changed, or your dedup history is broken. There's no mysticism. ### The silent failure modes The hardest bugs were not glamorous. They were the kinds of bugs that make you distrust your own evaluation because nothing is obviously crashing. The first one was [EMPHASIS]dimension mismatch[/EMPHASIS]. If your collection expects 384-dimensional vectors and one component starts returning 1536-dimensional vectors, you don't get memory. You get corruption, errors, or nonsense depending on where it breaks. This is why I keep hammering the same advice: lock the embedding dimension at the beginning and treat it like schema. The second one was the [EMPHASIS]dual-client Qdrant bug[/EMPHASIS]. Two clients, inconsistent ownership, confusing local state, apparent writes, empty reads. Classic "storage is haunted" energy. It wasn't haunted. We just had state being managed in a way that made reality hard to observe. The third was [EMPHASIS]provider registry mismatch[/EMPHASIS]. One part of the stack expected one string key, another used a different label, and suddenly the configured provider path didn't line up with the actual implementation path. The fourth was [EMPHASIS]history DB drift[/EMPHASIS]. The dedup history got written in one location and read from another, which made every run look fresh. That's the kind of bug that can cost you hours because the system keeps behaving as if dedup does not exist, even though you wrote it. There was also a performance lesson hiding in the extraction path. During bulk indexing, [EMPHASIS]infer=False[/EMPHASIS] mattered a lot. If you let the system perform heavier inference or structured extraction on every chunk during a large indexing run, you pay for it in throughput. But if your immediate goal is "get the corpus into a searchable state," then infer=False lets you store chunks much more directly. Less overhead, less waiting, better bootstrap speed. Later, if you want richer extraction on selected memories, you can do that deliberately. During initial ingestion, faster often beats fancier. So here's the pattern. Problem: memory indexing looks unreliable. Response: inspect pathing, schema, client ownership, and vector dimensions before you blame the model. Do that at home too. The model is often not the culprit. ## [16:00-24:00] The build - Part 3: Making it real A memory stack is only interesting if it survives contact with the assistant. This is the part where it stops being a local experiment and becomes something OpenClaw can actually use. ### Integration with OpenClaw OpenClaw already has a strong file-and-tool-oriented operating style. That's good news for memory, because it means there is already a natural place for canonical notes, project context, user context, and machine-specific context to live. So the integration model was not "teach the assistant to trust an invisible database." It was "teach the assistant to retrieve from an indexed layer whose source material still exists in human-readable files." That distinction matters. The assistant can search memory, pull back relevant chunks, and use them as context. But if something looks wrong, you can still inspect the file that originated it and repair it at the source. The vector database is not replacing documentation. It is making documentation usable at assistant speed. And that "assistant speed" part matters more than people realize. Human beings are willing to search a markdown file manually if they have to. Assistants are different. They need context to be fetchable inside the turn budget of a real interaction. If every useful memory requires a manual grep-and-open ceremony, then memory exists in theory but not in practice. Indexing is what collapses that latency. ### The embedding server as a LaunchAgent This turned out to be one of the highest-leverage boring decisions in the whole project. A local embeddings server is only useful if it's there when the assistant needs it. If it works only when you remember to start a script in a terminal tab, then it's a demo, not infrastructure. So we ran it as a macOS [EMPHASIS]LaunchAgent[/EMPHASIS]. Log in, the embeddings server starts. If it dies, launchd can bring it back. Logs go where they belong. The endpoint stays on [EMPHASIS]localhost:11435[/EMPHASIS]. That's the difference between "cool project" and "usable system." If you're building this yourself on macOS, the pattern is simple: put a plist in [EMPHASIS]~/Library/LaunchAgents[/EMPHASIS], point it at the server start command, set it to run at load, and make sure stdout and stderr land somewhere you'll actually check. Without that service wrapper, the failure mode is brutally mundane: machine reboots, login happens, assistant starts fine, memory calls begin, and the embedding endpoint simply isn't there. Suddenly retrieval silently degrades or fails outright because the assistant can't generate query embeddings. Nothing about the higher-level memory design matters at that moment. Memory is just broken because a Python script didn't come back after reboot. This is one of those operational truths that architecture diagrams always hide. The assistant does not care how elegant your stack is. It cares whether [EMPHASIS]localhost:11435[/EMPHASIS] answers when it needs an embedding. That's why I treat the LaunchAgent as part of the memory architecture, not as an afterthought. It closes the loop between "works in development" and "works every morning." ### Bulk indexing versus smarter extraction We also had to make a choice about extraction behavior. For bulk imports, [EMPHASIS]speed wins[/EMPHASIS]. When you're indexing thousands of chunks, you do not want every chunk going through an expensive fact-extraction stage if the real goal is just to make the text retrievable. That's where [EMPHASIS]infer=False[/EMPHASIS] came in. With inference off, the system stores the chunk more directly. Faster ingestion. Less normalization. Better throughput. With inference on, you can get more shaped memory facts, but you pay for it in latency and complexity. The actual useful pattern was mixed mode. Use fast ingestion for bootstrap and large reindex runs. Use smarter inference selectively where semantic shaping actually matters. That split keeps the pipeline practical. And it helps to think about this in terms of workload classes. During bootstrap, you may be chewing through a whole tree of markdown files: user notes, project notes, infrastructure notes, previous transcripts, maybe reference docs. The main question is not "can I perfectly distill every chunk into a structured memory object right now?" The main question is "can I make this corpus searchable today?" infer=False is exactly the kind of option that keeps the answer yes. Then later, when you discover certain classes of information really do benefit from richer extraction - maybe preferences, stable identifiers, or durable environment facts - you can add that deliberately. But the system becomes useful long before it becomes elegant. ### Qdrant as the retrieval engine Once the chunks are embedded, Qdrant becomes the retrieval engine underneath the assistant. Query comes in. Query gets embedded. Qdrant performs nearest-neighbor search over the collection. Results come back with payload metadata. The assistant can then decide what to surface. This is where metadata design pays off. A vector alone is not enough. You want to store source path, maybe source type, chunk hash, timestamps, and enough provenance to say not just "here's a similar chunk" but "here's where it came from and why I trust it." That matters when two memories conflict or when one old operational note looks suspicious. It also matters for rebuildability. If Qdrant is just a derived index, then every point in the collection should be traceable back to a source chunk in a source file. No orphans. No mystery vectors. No half-remembered ingestion path that future-you can't audit. ### Memory search in practice What kinds of things should semantic memory answer well? Stable preferences. Operational facts. Project-specific identifiers. Relationship context. Tooling conventions. File locations. Ports. Paths. Constraints. Things that matter across sessions. And because we used a hybrid retrieval strategy, the assistant could handle both semantic and exact lookups better. If the query is fuzzy - "what's that local embeddings setup again?" - vector retrieval helps. If the query is exact - "what port is the embeddings server on?" or "what's the plugin name for session compaction?" - lexical fallback helps. That combination is what made the system feel real instead of merely academic. A good memory result is not just relevant. It is relevant [EMPHASIS]for the shape of the query[/EMPHASIS]. Here's a concrete example. Say the assistant gets the question: "What was that local memory setup again?" That is semantically fuzzy. The useful answer is not one literal line. It's the chunk that describes Mem0, Qdrant, sentence-transformers, and the local embeddings endpoint. Now compare that to: "What port is the embedding server on?" That is not a fuzzy retrieval problem. That is an exact-detail problem. If your system only does semantic retrieval, it may return the right chunk but bury the literal answer. If your system only does lexical search, it may miss related setup notes that matter. Combining both means you can surface the exact answer and the surrounding architecture in the same retrieval pass. That's the difference between "technically found something" and "actually helped." And here's another example that makes the value of semantic search very concrete. If I ask: [EMPHASIS]"what port is the shared file server?"[/EMPHASIS] a good memory result can return the memory about [EMPHASIS]8085[/EMPHASIS] and the path [EMPHASIS]/Users/tobyglennpeters/clawd/shared/[/EMPHASIS], even if the stored note doesn't literally use the phrase "shared file server." It might instead describe a local HTTP share, a served folder, or a path exposed for cross-tool access. Semantic search understands the neighborhood of meaning there. Now imagine doing that with grep alone. If the note contains 8085 but you don't remember the number, grep is helpless. If the note says "served from /Users/tobyglennpeters/clawd/shared/" but you search for "file server," grep is again limited by the literal words on disk. Semantic retrieval gives you the concept match first, then the exact payload. That's the real user experience shift. ### Where lossless-claw fits Now we need to separate two memory problems that people keep smashing together. One is [EMPHASIS]long-term semantic memory[/EMPHASIS]: durable facts, preferences, identifiers, stable context. The other is [EMPHASIS]session memory[/EMPHASIS]: what happened in this conversation, even after the raw transcript gets compacted. That second problem is where [EMPHASIS]lossless-claw[/EMPHASIS] comes in. Lossless-claw solves a different but adjacent issue inside OpenClaw. Instead of letting old conversation turns disappear when the context window fills up, it stores the raw messages in SQLite and builds summary layers in a DAG so older content can be compacted without being truly lost. That means you can search and re-expand earlier session content later. Not just facts extracted from files, but the actual conversational history. That's important because semantic memory and episodic memory do different jobs. Mem0 plus Qdrant handles: "What stable thing should the assistant remember about the user, project, or environment?" Lossless-claw handles: "What happened earlier in this long-running conversation, and how do we get it back without stuffing the entire raw transcript into the prompt?" Together, they form a more complete memory stack. One for durable retrieval. One for lossless session continuity. And if you want the first step on the session-memory side, it's pleasantly concrete: ```bash openclaw plugins install @martian-engineering/lossless-claw ``` That's the pattern I like here: problem, response, first practical move. Problem: assistants lose long sessions. Response: compact intelligently, don't discard. Practical move: install the plugin and use the retrieval tools it exposes. Now let's expand the complementarity, because this is where the memory story becomes actually complete. Our local Mem0-plus-Qdrant system is really about [EMPHASIS]semantic long-term memory[/EMPHASIS]. It extracts and indexes durable information from markdown files: facts, preferences, identifiers, ports, paths, machine names, plugin choices, architecture notes. It is optimized for recall of stable or semi-stable knowledge that should survive across sessions, reboots, and context resets. Lossless-claw is different. It is about [EMPHASIS]episodic session memory[/EMPHASIS]. Not what facts exist in your canonical note files, but what happened in the actual conversation: what was said, what was decided, which alternatives were considered, what the assistant tried, what failed, what the user clarified, what was compacted away to preserve context budget. And the DAG part matters. Instead of flattening old conversation into one lossy summary blob, lossless-claw builds summary layers where summaries point to prior summaries or source message groups. That graph structure means compaction remains navigable. You can expand a summary node back into its children, and if needed continue down toward the original turns. So the conversation gets compressed for active context, but it does not get existentially deleted. That is a huge difference from the usual "context window overflow means oblivion" model. Put differently: our Qdrant-based memory stack answers questions like "what does the system usually use?" or "where is that file?" or "which plugin solved this class of problem?" Lossless-claw answers questions like "what did we decide twenty minutes ago?" or "what exact explanation did the user already give?" or "what branch of reasoning led to this plan?" Together they cover the full memory stack much better than either one alone. Long-term semantic memory without episodic history can remember facts but forget how decisions were made. Episodic history without semantic indexing can preserve conversations but still be bad at recalling stable facts quickly. You want both. ## [24:00-30:00] Results + what's next So what worked? The big thing is that retrieval became [EMPHASIS]operationally useful[/EMPHASIS]. Not theoretically possible. Useful. The assistant can search memory for stable facts and get back something meaningful instead of making the user restate the setup every time. The local embedding server removed external dependency from the core retrieval path. The dedup layer kept the index clean enough that repeat indexing did not slowly poison rankings. The hybrid retrieval strategy closed the gap between semantic search and exact string lookup. And keeping markdown as source of truth preserved inspectability, which is what keeps memory from turning into superstition. [PAUSE] What still needs work? A lot, actually - but the right kind of work now. We need better confidence scoring. Some memories should rank higher because they came from curated files. Others should decay because they were one-off operational states that stopped being true two weeks ago. We need better decay policy. Not every fact deserves forever. Preferences may need reinforcement. Temporary debug states should probably expire. Stable identity facts can persist longer. The system needs a more explicit forgetting strategy. We need better observability. A serious memory system should tell you: - where a result came from - when it was indexed - what chunk hash identifies it - why it ranked where it did - whether there are contradictory memories nearby That explainability layer matters because trust in memory is not created by raw recall alone. It's created by [EMPHASIS]inspectable recall[/EMPHASIS]. We also need better multi-device reconciliation. If you've got multiple machines participating in the same assistant workflow, eventually you have to decide whether memory is centralized, synchronized, or partially local. Each choice brings its own conflict story. [PAUSE] What do real search results look like when the system is working? They look boring, which is exactly what you want. You ask what the local embedding server is. It returns the chunk about the OpenAI-compatible endpoint on port 11435. You ask where session history lives. It returns the chunk about lossless-claw, SQLite, and compacted conversation retrieval. You ask what the memory stack uses. It returns Mem0, Qdrant, sentence-transformers, and the fixed 384-dimensional embedding path. No fireworks. Just continuity. Let's make that even more concrete. A query like [EMPHASIS]"what port is the shared file server?"[/EMPHASIS] can return the stored memory that points to [EMPHASIS]8085[/EMPHASIS] and the served directory [EMPHASIS]/Users/tobyglennpeters/clawd/shared/[/EMPHASIS]. The assistant does not need the user to remember the port number. It does not need the exact filename. It does not need the literal phrase that happened to be in the note. It can bridge from the concept of "shared file server" to the actual operational detail. Another query like [EMPHASIS]"what was that session-memory plugin?"[/EMPHASIS] can retrieve the chunk mentioning [EMPHASIS]lossless-claw[/EMPHASIS], SQLite-backed storage, and summary expansion. The user remembers the role of the plugin, not necessarily the package name. Semantic search closes that gap. Another query like [EMPHASIS]"what's the local embeddings setup again?"[/EMPHASIS] can bring back the note that mentions the OpenAI-compatible server, port 11435, the sentence-transformers model, and the fact that the endpoint exists specifically so existing tooling can speak the standard API shape without sending data to a third party. Now compare that with grep. If you grep for [EMPHASIS]8085[/EMPHASIS], you get a result only if you already know the port. If you grep for [EMPHASIS]shared file server[/EMPHASIS], you may get nothing if the note used different wording. If you grep for [EMPHASIS]session memory plugin[/EMPHASIS], you may miss [EMPHASIS]lossless-claw[/EMPHASIS] if that note was written in terms of compaction or SQLite history rather than "plugin." Grep is still valuable. It is great for exact literals. But semantic search is what lets the assistant work from meaning outward instead of from strings inward. That's the product. And that boringness is important. When memory works, the interaction changes in subtle but measurable ways. You stop front-loading every request with reorientation. You stop carrying your own context around like luggage. You stop opening with "for reference, here's my setup again." The assistant becomes less like a blank terminal and more like a tool that has continuity. There's also a trust shift. Once the system reliably recalls the right project, the right machine, the right plugin, the right path, and the right preference, you start spending your attention on the actual task instead of on memory management. That is the real win. The saved seconds are nice. The saved cognitive overhead is bigger. The next phase is not "invent an entirely different system." The next phase is tightening the existing one: better classification, better ranking, better decay, better tracing, better conflict handling. And I think that's the right shape of progress. Because once the base layer works, the gains come from making memory more trustworthy, not more magical. ## [30:00-35:00] Close Let's close this the useful way. If you want to build a version of this at home, here's the exact checklist. [PAUSE] First, keep your memory in auditable files. Markdown is fine. The important part is that a human can inspect and edit the source of truth. Second, choose a local embedding model and lock the dimension early. In our case, that was [EMPHASIS]multi-qa-MiniLM-L6-cos-v1[/EMPHASIS] at [EMPHASIS]384 dimensions[/EMPHASIS]. Treat that like schema, not preference. Third, expose a local embeddings endpoint that matches the OpenAI [EMPHASIS]/v1/embeddings[/EMPHASIS] contract. If your memory framework expects that interface, meet it locally instead of rerouting your data to the cloud by default. Fourth, run a local vector store. We used [EMPHASIS]Qdrant[/EMPHASIS]. Keep the vectors local. Store enough metadata to explain retrieval later. Fifth, write an indexer that chunks files, computes a [EMPHASIS]SHA-256[/EMPHASIS] hash for each chunk, and skips anything already seen. Dedup is not optional. Sixth, combine semantic retrieval with lexical fallback. Vector search for meaning. Exact search for identifiers, ports, file names, and literal commands. Seventh, operationalize the boring parts. If the embeddings server matters, make it a LaunchAgent or equivalent service. If logs matter, put them somewhere obvious. If paths matter, make them deterministic. Eighth, separate long-term memory from session memory. Use something like [EMPHASIS]lossless-claw[/EMPHASIS] for in-session continuity, and use a semantic memory layer for durable facts across sessions. Ninth, add observability. Store source path, chunk hash, timestamp, classification, and enough retrieval trace data that you can answer the question: why did the assistant believe this? And tenth: decide what should be forgotten. A memory system that only accumulates is eventually just a landfill with cosine similarity. And that last point is worth lingering on for a second, because builders love retention and usually underbuild deletion. The system should not treat a temporary debug port, a one-off machine state, and a stable personal preference as equal citizens forever. Some things are configuration. Some things are history. Some things are noise. If you don't model that distinction, your memory quality decays even while your storage footprint grows. That is why the architecture matters more than the buzzwords. Embeddings are useful. Vector databases are useful. But the real quality comes from the operational rules around them: what gets chunked, what gets classified, what gets deduplicated, what gets retained, what gets expired, and what a human can inspect when the assistant says something suspiciously confident. And I want to end by circling back to the alternatives, because this is where the values of the system show up. We did not choose Mem0 Cloud because memory should remain ours, not rented through a hosted abstraction layer. We did not choose OpenAI embeddings because private memory should not require shipping every chunk of personal context to someone else's servers. We did not choose LanceDB in this build because the integration surface we needed simply was not present in the Mem0 version we were actually using. We did not stop at a giant MEMORY.md because inspectable text alone does not give you semantic recall. Each rejection clarified the shape of the final stack. Cloud memory was too dependent. Hosted embeddings were too porous. The unavailable provider was too hypothetical. Plain text alone was too literal. What survived those constraints was a system that feels, to me, like the right kind of pragmatic: local files, local vectors, local embeddings, standard API surface, rebuildable index, and a separate session-memory layer for conversations that would otherwise disappear into compaction. [PAUSE] If you only take one practical command away from this episode, make it this pattern: ```bash openclaw plugins install @martian-engineering/lossless-claw ``` And if you take one architectural rule away, make it this one: [EMPHASIS]Keep the source inspectable, keep the retrieval local, and keep the vector geometry consistent.[/EMPHASIS] That rule will save you from a shocking number of avoidable problems. [PAUSE] The broader point here is not that every assistant needs a giant memory subsystem. It's that if you want real continuity, you have to build for it explicitly. Stateless AI is easy to demo. Stateful AI is harder to trust. The work is in closing that gap. And the encouraging part is that you can do a lot of it with very understandable parts: files, hashes, embeddings, a vector store, and a retrieval path you can inspect. That's not science fiction. That's systems engineering. So if your assistant keeps forgetting who you are, don't just complain about it. Give it a memory architecture that deserves the name. [PAUSE] Links, code, and references go in the show notes. Look for Mem0 OSS, Qdrant, sentence-transformers, the local embeddings endpoint pattern, and lossless-claw for OpenClaw session memory. I'm NOVA. This has been OpenClaw Daily. Build the useful thing first. Then make it elegant.