Agent Memory at Scale — Markdown Files vs. Vector Databases

When a file-and-index memory system (the llm-wiki-pattern this vault uses, and Claude Code's auto memory) stops scaling, what actually breaks first, and when embeddings genuinely earn their place. Reasoned out in a session on 2026-07-19 that started from claude-code-memory-architecture's recall path.

The short answer: markdown scales far further than people assume, the failure mode is not the one they expect, and a vector store is usually the wrong first upgrade.

What breaks first is the index, not the storage

Disk is never the constraint. A 36k-word vault is ~50 KB of markdown; a thousandfold increase still wouldn't matter. The constraint is the preloaded index, and it fails differently depending on the system:

• Claude Code auto memory has a hard mechanical cliff — 200 lines / 25 KB of MEMORY.md, with everything past it silently dropped at load. One line per memory puts the real ceiling near 200 memories. Past that, memories still exist on disk and are simply never recalled. Nothing errors; recall just quietly stops being complete.
• A vault index.md (loaded by contract rather than by a harness limit) has no enforced cap. Its limit is soft: context budget, and the point where a flat one-line-per-page list stops being scannable enough for the model to choose well. That degradation starts well before any token limit bites.

Both are the same underlying failure: retrieval quality collapses when the index stops fitting in the reasoning budget. The documents are fine.

Hierarchy before vectors

The first upgrade is not embeddings — it is another level of indirection. An index grouped into categories becomes an index of indexes; the model traverses two hops instead of one. Same mechanism, one more level, and it carries a file-based system to thousands of pages. Wikipedia is this pattern.

A wiki has a second retrieval channel the flat-memory-directory pattern lacks: [[wikilinks]] are graph edges, traversable without consulting the index at all. Land anywhere near the right neighbourhood and links carry you the rest of the way. This is why the wiki pattern degrades gracefully where a flat pile of memory files does not — and it is an argument for linking liberally, since each link is a retrieval path that survives index truncation.

When a vector store genuinely earns its place

Three honest triggers. Notably, none of them is "I have a lot of notes":

1. You cannot write an index line per item. Not that it is tedious — that at 50k emails or every page of a PDF archive, no meaningful human-readable index exists at that granularity.
2. You do not control write time. File-and-index retrieval works because something wrote a careful one-line description at ingest. Bulk-imported content has no such curation, and retrofitting descriptions for 10k documents costs more than embedding them.
3. You need recall over precision. An index finds what the index-writer anticipated. Embeddings find the thing you would never have filed under that heading — unfamiliar phrasings, oblique connections. This matters for exploratory queries, not for "where did I put X."

A counterweight worth stating plainly: below roughly 10k documents, agentic grep over markdown often beats vector search. The model can iterate — search, read, refine the query, search again — and that loop routinely outperforms a single top-K similarity fetch. That capability exists with zero infrastructure. Exhaust it before adding an embedding pipeline.

The step between grep and a vector store

There is a middle rung this page originally skipped. Karpathy's llm-wiki-pattern names it: when the index file stops sufficing, reach for a local search engine over the markdown before an embedding pipeline. qmd is the concrete instance — hybrid BM25 + vector with LLM re-ranking, entirely on-device, shipping both a CLI and an MCP server.

That packaging matters more than the ranking algorithm. An MCP server is the "expose retrieval as a search tool" shape argued for below, so adopting it changes the agent's retrieval surface without changing the storage model at all: files stay canonical, stay greppable, stay diffable in git. Contrast a vector store, which introduces a second copy of the corpus that can drift from the files.

So the upgrade ladder has four rungs, not two: index.md → hierarchy in the index → local hybrid search over the same files → an embedding store with its own lifecycle. Only the last one is a genuine architecture change.

What changes in the two regimes

	Curated files + index	Vector store
Write cost	high — choose destination, dedupe, write a retrieval-grade description, link	low — chunk, embed, insert
Reconciliation	forced at write time	never
Contradictions	must be resolved to file the entry	both chunks retrieved; the model sorts it out mid-task
Read path	model decides what to open, in a loop; slow, precise, explainable	top-K fetched before the model reasons; fast, opaque
Failure mode	index truncation — silent incompleteness	relevant fact at rank K+1 — silent miss

The under-appreciated half is the write side: curated files front-load reconciliation; a vector store defers it indefinitely. A /lint pass that hunts contradictions and stale claims has no vector-store equivalent — the retrieval layer will hand the model a 2024 fact and its 2026 correction with equal confidence. Recency metadata mitigates this; it does not solve it. This is the real cost of switching, and it is paid continuously rather than at migration.

Both failure modes are silent. That is the property to design monitoring around.

Where RAG actually lands

RAG is not the alternative to file-based memory — it is a component, and current practice converges back toward the agentic shape anyway. Rather than pre-stuffing retrieved chunks into the prompt, expose the vector store to the agent as a search tool. Retrieval then becomes one more call in the agent loop, structurally identical to the model issuing a file Read. The difference between the two architectures collapses to how the agent finds the filename.

This reframes the whole question. The choice is not "files or vectors" but "which index does the agent search" — and an agent can hold both.

The hybrid, and the shape this vault already has

The endgame splits by curation level, not by size:

• Conclusions — synthesised, high-trust, small, index-loaded. Stays markdown indefinitely. (wiki/, including its index.md, and memory/.)
• Evidence — raw, append-only, large, uncurated. The layer that eventually gets an embedding index behind a search tool. (sources/.)

The pressure shows up in the evidence layer first, for a structural reason: sources grow monotonically by design while synthesis pages get rewritten rather than accumulated. An immutable append-only store has no equivalent of revising a page in place, so it is the only half of the system whose item count only ever increases. A vault can therefore stay flat-file for its wiki essentially forever while still needing search over its sources.

Concrete trigger for this vault: as of 2026-07-20 it holds 17 wiki pages, a 5.1 KB / 101-line index.md, ~38k words, and 3 sources. Revisit when sources/ holds a few hundred documents and /ingest becomes about finding things rather than filing them. Until then the entire upgrade path is hierarchy in index.md.

(Was 13 pages / 76-line index / zero sources on 2026-07-19. The index grew ~33% in a day against 4 new pages — a reminder that the index scales with pages, not with sources, so it is the wiki half that will strain first even though the sources half grows monotonically.)

An external data point: OKF makes the same bet

Google Cloud's Open Knowledge Format (v0.1, Jun 2026) is a spec for exactly the architecture this page describes — markdown documents, YAML frontmatter, index.md per directory, markdown links as graph edges — aimed at enterprise metadata estates. It is useful here as independent evidence that the pattern is taken seriously at scale, and as a test of this page's central claim.

The claim survives: OKF inherits the index failure mode wholesale and says nothing about it. No cardinality guidance, no convention for splitting an oversized index, no summary tier. It bets entirely on the second retrieval channel — graph traversal over links — which is the same mechanism credited above for making wikis degrade gracefully.

Where it gets uncomfortable is the write side. OKF's reference producer is an agent that walks BigQuery datasets and generates a document per table, in bulk. That trips two of the three triggers listed above — you do not control write time, and at per-table granularity across an enterprise estate no meaningful human-readable index exists — which is to say OKF's flagship producer operates precisely in the regime where this page argues files-plus-index stops working.

That is not a refutation of either position. It sharpens the distinction: OKF is an interchange format, not a retrieval architecture. A bundle is how knowledge moves between producer and consumer; what the consumer does to make a million documents findable is out of scope for the spec and remains the open problem this page describes. Worth revisiting if v0.2 grows an indexing convention.

Tension with ai-agent-architecture §6.4

That guide's §6.3–6.4 present the "standard memory implementation" as vector DB + SQL store + summariser, with a retrieval pipeline that embeds every incoming user message and pre-loads top-K memories into the system prompt before the agent loop runs.

This page argues that pipeline is increasingly the legacy shape. Pre-stuffing top-K spends context on guesses and gives the model no way to iterate when the first retrieval misses; agent-initiated search fetches on demand and can refine. Both descriptions are accurate about real systems — §6.4 describes what much production tooling still does, this page describes where the design is moving — but they should not be read as one consistent recommendation. Treat §6.4 as the pipeline architecture and this page as the agentic one.

Related

• claude-code-memory-architecture — the concrete file-and-index system whose recall path this page generalises from; source of the 200-line / 25 KB figures.
• ai-agent-architecture — §6.2 (four kinds of memory), §6.3–6.4 (the vector pipeline this page pushes back on), §6.6 (agent-controlled memory), §7 (context engineering).
• okf-spec — a published spec for the file-and-index shape, and the case study in the section above.
• llm-wiki-pattern — the pattern whose "~100 sources, ~hundreds of pages" scaling claim this page examines; source of the qmd middle rung.

References

• Auto memory index limits and on-demand topic-file loading — Claude Code docs: https://code.claude.com/docs/en/memory
• No raw source in sources/. This page is reasoning and synthesis from a working session, grounded in the Claude Code documented limits above; the scaling thresholds (~200 memories, ~10k documents) are order-of-magnitude judgements, not measured benchmarks.