“What semantic is good at is providing similar content, but not necessarily relevant content. Whereas this seems to provide more relevant content — which is really important to anti-hallucination.” — Lou
Session context: 2026-07-16_Mastermind — Jay asked what the next upgrade past a Karpathy-style wiki would be, assuming the answer was “add RAG.” Lou pushed back on the assumption.
Core Idea
The default upgrade path for a growing knowledge base is treated as settled: files → wiki → vector database. When retrieval gets hard, add embeddings. Lou’s argument is that the industry is quietly walking this back, and the distinction that explains why is similar is not relevant.
Semantic search returns chunks whose embeddings sit near your query in vector space. That is a similarity ranking, and similarity is a decent proxy for relevance right up until it isn’t — at which point it confidently hands the model plausible, adjacent, wrong material. Which is precisely the failure that matters, because retrieved-but-irrelevant context is the raw material of a confident hallucination. The model doesn’t know the chunk was merely nearby.
Two data points Lou put on the table (both worth verifying before you rebuild anything on them):
- Anthropic reportedly dropped RAG internally, finding that indexing over documents and running grep-style fuzzy search outperformed semantic retrieval — Lou recalled the figure as roughly 4×. (verify: the 4× claim and its source; Lou attributed it to “Czerny and the boys” and offered to find the article.)
- Pinecone — a vector database company — shipped a wiki-style product positioned as the first level of retrieval, ahead of embeddings. When the vendor whose business is embeddings tells you to try something else first, that’s a signal worth reading.
The cost argument compounds the accuracy one. Lou: “Now you don’t have to ingest stuff, you don’t have to convert it, you don’t have to embed it, you don’t have to do the BM25, you don’t have to do the retrieval ranking and all that expensive stuff. You’re just doing a basic search.” No ingestion pipeline, no re-embedding when documents change, no chunk-size tuning, no staleness, much lower latency. The simpler architecture is also the more accurate one — which is rare enough to be worth acting on.
The proposed stack is grep over a graph: a small hot cache the agent can scan in full, plus a linked knowledge base (Lou is moving toward an OKF — open knowledge format — folder rather than a bespoke wiki, on the grounds that it transports to other systems, not just other harnesses). The graph gives the agent something semantic search never provides: a way to move deliberately — follow this link, check that backlink — rather than teleporting to whatever was nearest in vector space.
Lou’s caveat, and it’s a real one: the retrieval pattern that serves inference over your own memory is not obviously the pattern that serves a legal or financial semantic database. The claim here is scoped to agent memory, where the corpus is yours, the volume is modest, and relevance beats recall.
Practical Application
If you’re about to add a vector layer, don’t — yet. Run the cheap experiment first:
- Take ten real questions you’d want your knowledge base to answer.
- Point an agent at the raw folder with grep and glob and no index at all.
- Score the answers against what you’d hoped a semantic layer would give you.
Most people find the gap is smaller than the pipeline they were about to build. If the folder is too big to scan, that’s an argument for structure — a hot cache, better links, tighter frontmatter — before it’s an argument for embeddings.
Coaching question: “Am I adding a semantic layer because retrieval is failing, or because a vector database is what a serious knowledge base is supposed to have?”
Related Insights
- Insight - The Three-Layer Knowledge Architecture — Keyword, Graph, and Semantic Retrieval — the three retrieval modes; this reorders their priority and argues most people never need the third.
- Insight - The Hot Cache, Wiki, Semantic Memory Stack for AI-First Workflows — the prior architecture, which treats the semantic archive as the natural third tier. This insight contradicts that sequencing for agent memory.
- Insight - RAG Is Raw Material, Not Answers — Design for the Right Retrieval Architecture — established that RAG returns material, not answers; this goes further, questioning whether RAG should be the default retrieval layer at all.
- Insight - Multi-Pass Retrieval Turns Shallow Searches Into Strategic Intelligence — traversal over one-shot lookup; the graph is what makes traversal possible.
Evolution Across Sessions
This is a genuine reversal, and it’s marked as one. Insight - The Hot Cache, Wiki, Semantic Memory Stack for AI-First Workflows (2026-05-21) presented the three-tier stack with a Pinecone-style semantic archive as the long-term tier, and instructed readers to “pick a semantic layer that integrates with the wiki.” Ten weeks later Lou’s recommendation is: do the hot cache, then the graph, then measure — and probably stop. The contradicts: link is deliberate; both pages stay, because the earlier one is still right about access patterns being distinct, and wrong only about which pattern earns the third tier. Future sessions should resolve this properly — ideally against the actual Anthropic finding rather than a recalled 4×, and against this vault, which at 238 insights is now a real test corpus for grep-over-graph.
Source
- 2026-07-16_Mastermind (Lou — semantic vs. relevant, grep-over-graph, OKF direction; Jay Drobez — raised the RAG upgrade question)