Key Takeaway: Simpler RAG outperforms the over-engineered version. Then you learn RAG was only one layer — retrieval is half the context budget, and the other half is entity profiles, memory, topic context, and interaction signals. The gap between finding the right chunks and assembling what a conversation actually needs is the whole product.
I built RAG over my own data: email, texts, meeting transcripts, documents. Three major iterations. 18,000 chunks. The first version did everything the research papers recommend — tiered retrieval, query decomposition, cross-encoder re-ranking. It took 3 seconds per query and wasn't noticeably better than a simpler approach.
Iteration three: remove everything that isn't load-bearing. Fan-out across topic indexes in parallel; HNSW costs under 1ms each. Distance cutoff at 0.70 — if nothing is close enough, return nothing. Total under 10ms. Better results than iteration one, because the cutoff eliminates noise the re-ranker was trying to filter.
I shipped it. It worked. I thought I was done.
Then I ran into the thing RAG can't solve.
RAG retrieves chunks. A chunk knows content, not meaning. It doesn't know that the person you mentioned is someone whose judgment you've relied on for five years, versus a vendor you've emailed twice. It doesn't know you've raised the same concern about a decision three times without acting on it. It doesn't know you just came out of a project that failed for the exact same reason you're about to make.
That's not a retrieval problem. It's a context assembly problem. RAG is one layer. I had to build the rest.
The current system assembles context in eight layers before the model sees the query. An identity layer: compressed distillation of career, values, cognitive patterns, always present, about 1KB. A topic layer: synthesized context for whatever I'm working on, 4K characters. Entity layers: when I mention a person, the system injects a profile card — relationship tier, channels, tenure, shared history. An interaction signal layer: communication patterns, not content. Then RAG, now topic-scoped. Entity-associated content. Referenced conversation injection for prior discussions.
Eight layers. 60,000 character budget. RAG gets 50%.
Entity resolution has to be deterministic. I tried LLMs for it. Every experiment produced confident, plausible, occasionally catastrophic identity misassignments. A 5-guard matcher — contact labels, calendar co-occurrence, name clustering, email, phone — is slower to build and much better in production.
Memory is a design problem, not a storage problem. Early versions wrote a record per conversation turn: 19,000 entries, chain forked, context quality degraded. The current version writes curated entries only — preferences, patterns, decisions. 12,000 entries, append-only, content-addressed with SHA-256. A memory system that tries to remember everything remembers nothing usefully.
I started asking "how do I retrieve the right chunks" and ended up somewhere else: "how do I assemble, from years of accumulated data, exactly what this conversation needs." The retrieval problem was solved in iteration three. The context assembly problem is still being solved.
The gap between those two questions is the whole product.