"AI-ready" is supposed to mean something. My first test said otherwise.

For about a year I built knowledge bases by hand across a handful of projects: SaaS products, consulting work, internal tools, each its own shape. Two of them are the ones this piece follows: a SaaS product (Field Read, formerly called Ansir AI) and a small-business consulting practice. I ruled out Obsidian and NotebookLM too: locking the whole thing into one tool's ecosystem seemed too rigid for how differently these projects would grow. So the knowledge base stayed plain markdown: specs, decisions, half-finished plans, reference notes, scattered across whatever folder structure each project happened to grow.

My project folders had turned into the kind of desk where books pile into towers: don't touch, I know where everything is. That's not a complaint about how much material there was. It's the real problem: a knowledge base only one person can navigate isn't a knowledge base. It's a memory palace, and memory palaces don't survive the project outgrowing the person who built them.

This is the story of what happened when I tried to hire some help.

Building a library

I was already building something like a library, long before I had a name for it. Each project grew its own sprawl of markdown, and I learned, slowly, what actually helped a model reason about a codebase and what was just noise, closer to editing than prompting. It worked, in that I could find things, and the model usually answered correctly once pointed at the right file.

It didn't scale. As a project grew, so did what I had to hold in my head: which file replaced which, which reference doc was still true, whether two documents about the same feature actually agreed. Linking the files by hand was slow, and easy to skip under deadline pressure. Managing the shape of the material (not just whether a fact was right, but whether a file was still the right kind of document for what it had become) was the part nobody warns you about. It was a burden long before anything broke.

Hiring a librarian

I wasn't the only one circling this problem. In April, Andrej Karpathy named the pattern in his "LLM Wiki" gist: agents don't get bored, don't forget to update a cross-reference, and can touch fifteen files in one pass, so the upkeep that kills most human-run wikis stops being a burden once an agent does the bookkeeping. When Google Cloud published the Open Knowledge Format spec this June (a vendor-neutral markdown-plus-YAML standard, one required type field per concept, otherwise unopinionated), it read like a job posting for exactly the role I needed filled. It was, more or less, a standard version of the structure I'd already been building by hand.

I moved this project's knowledge base onto OKF mid-development, expecting a clean win. Before calling it done, I ran a blind test to check. What follows is what that test found: two places where the librarian I'd hired had quietly mishandled part of the collection, and how I caught both.

The setup

The material: 53 raw markdown files, no machine-readable metadata, one real cross-link in the whole set. I curated it into an OKF bundle: frontmatter (type/title/description/tags/status) on every file, a directory laid out by subject, a proper entry point. Along the way: 6 files deleted, 24 archived, 12 new reference docs pulled from finished plans.

Before claiming any win, I ran a blind test: a three-way comparison, not the easy two-way "old vs. new" version. Fresh agents, each with no prior knowledge of the project, answered the same nine questions against three conditions:

1. Pruned-raw: the 47-file raw directory, after the 6 deletions, with no other curation.
2. OKF bundle: the full curated structure, with an index pointer telling an agent where to start.
3. Unpruned-raw: the original 53-file set, restored from a backup just to run this test, then removed again right after.

All three were scored against ground truth: the actual, verified state of the live codebase, not just internal consistency.

First failure mode: pruning deleted the answer to a quality query

The unpruned original won the first round. Quality-query accuracy: unpruned-raw 4.4/5, pruned-raw 4.2/5, OKF 4.0/5. The least-processed set scored highest, and it wasn't noise spread across nine questions. One file caused almost all of it.

design-app-theme.md, one of the six files deleted during pruning, was the only document in the original set that answered "what is the visual direction for the app theme": the in-product theme (a teal at 191 70% 28%, Plus Jakarta Sans, an 8-point spacing grid), a different thing from the marketing-site theme the bundle does cover. Both the pruned-raw run and the OKF run answered that question with some version of "no dedicated app-theme doc exists, only marketing direction." That answer was true of the curated set. It was not true of the original docs. My own notes, written before this pass, had filed the gap under "marketing design files, out of scope." That was wrong. The file wasn't about marketing. It was about the product itself, and nothing in curation had extracted or archived it the way I'd handled other at-risk content. It had just vanished.

The uncomfortable lesson: a deletion and an extraction look the same from outside the active file set. In both cases, the file is gone. Only one of them keeps the information, and nothing flags the difference until something goes looking for it, in this case a blind test run against the unpruned original.

I fixed it, not just noted it. Same method I used everywhere else: read the file's claims, check them against the live code. Every checkbox in design-app-theme.md held: the color tokens, the font, all five components, all wired into the pages the file claimed. I extracted it into the bundle with a note recording that check, instead of restoring it as-is.

Second failure mode, one phase later: a completed plan kept in the wrong shape

I thought that was the last thing curation would get wrong quietly. One phase later, with the bundle close to sign-off, I ran a different check: not another blind test, but a manual audit of every active file against a rule the project had already set twice over: a finished plan gets archived, with a present-tense reference pulled out in its place, not left in the active set still wearing its checkboxes.

One file slipped through: plan-marketing-disclaimers.md, the only type: Playbook file still active in the bundle, all four phases checked [x]. It had already passed the blind test. Nobody had checked it against the shape question, only the content question.

Checking its claims against the live homepage code (the same method I used everywhere else) found one checked-off item was flatly false. Phase 4 claimed a "prices subject to change, as of [date]" footnote had been added near the competitor-pricing comparison. It hadn't. The only "subject to change" language anywhere in the code is an unrelated clause buried in the Terms of Service page. Everything else in the plan (the trademark notice, the claim-substantiation footnote, the footer layout) checked out exactly as described. One checkbox, out of roughly twenty, was wrong.

This is the same failure as design-app-theme.md, in a different form. Both times, a file's real state and its stated state had drifted apart, and nothing in the process was set up to notice, because "is this file accurate" and "is this file the right shape" are different questions. The blind test had already scored this bundle well. It has no opinion on whether a Playbook file should still exist; it can only score the answers a fresh agent gets from whatever's there. Catching this took a different kind of pass: one that checks the bundle against its own precedent, not against ground truth alone.

I archived the plan and extracted reference/ref-marketing-disclaimers.md in its place: present-tense, frontmatter intact, and honest about the stale claim instead of quietly dropping it, the same treatment ref-stack.md got for its own gap, below. Then I did what I'd now done three times running whenever a fix changed bundle content: re-ran the full nine-question test, instead of hand-adjusting the one score it affected.

The numbers moved three times before I trusted them

First pass: the bundle behind on quality against the unpruned original, ahead on the one question raw search simply couldn't answer. A mixed picture, not a win. After closing the design-app-theme.md gap and adding the "Deferred" section to ref-stack.md (below): quality ties the unpruned baseline exactly. After the shape audit caught the false checkbox in plan-marketing-disclaimers.md: quality beats the original outright, not just matches it.

Every one of those moves came from a specific, named fix, checked against the live code, followed by a full re-test instead of a manual score change. Three times, I chose to re-test instead of re-grade. Twice in a row, re-testing caught a second, different bug that re-grading the same run would never have found.

The efficiency column (accuracy score divided by tool calls) climbs cleanly through every OKF round and beats both raw baselines throughout, because tool-call count is the most reliable number in this project. The same calculation using bytes read instead of tool calls doesn't hold; it flips the story: the round with the best accuracy scores comes out worst on a bytes-based ratio, for reasons that have nothing to do with the content fixes (see eval-numbers-summary.md's "Why tool calls, not bytes" section for the full reasoning). The plain version: the CLAUDE.md index pointer cuts the cold start (no exploring with ls or find before the agent starts work), which is a tool-call story, not a byte-count one. Any combined ratio should rest on the number that's actually reliable.

Raw search resurfaced a stale claim: curation actually got this one right

One question asked for the app's full technical stack: infra, auth, payments, monitoring. The raw-baseline agent, in both the pruned and unpruned runs, grepped its way to a spec file listing Sentry and Lighthouse CI under a "Confirmed integrations" table, no qualifier, and reported both as part of the current stack. The bundle's curated reference/ref-stack.md left both out, correctly: they've since been deferred, not shipped.

My first read of this was backwards. I assumed the curated file had dropped real facts, since it says less. It hadn't. The raw spec files simply hadn't been updated, and grep can't tell a stale claim from a current one when the source states it flatly. Curation, done well, holds a judgment call raw search can't make on its own.

But the bundle didn't say why it left out Sentry and Lighthouse; it just left them out silently, which is its own kind of gap. I fixed that with a short "Deferred" section in ref-stack.md, explaining both tools are paused while the team gets the marketing site live, and flagging that the older specs' "confirmed" language reflects planning intent, not shipped state. A curated file should record what it left out on purpose, and why, not just what it includes. I used the same fix later on the disclaimers reference doc, for its own stale claim.

A doc contradiction nothing else caught

Answering a question about tracked analytics events, the OKF agent found a live contradiction between two reference files: ref-analytics.md labels six events "planned, not yet instrumented," while ref-tracking.md calls the same work phase-complete. Neither raw run, pruned or unpruned, found this. Both answered from whichever file they read first, with no reason to check a second one.

That's a value an accuracy score doesn't fully capture: checking documents against each other, not just looking up facts in one. It's also a different kind of bug than staleness or deletion: two documents that were both true when written, that drifted apart without either one becoming individually "wrong."

The forcing function, and its three blind spots

Writing a one-line description: <what an LLM should use this file for> for every file is a surprisingly useful exercise, even with no AI reading it. On its own, it caught a Playbook file elsewhere in this project whose checkboxes were all marked done for a homepage redesign that, on inspection, was never wired into the live page.

But testing it against real gaps, instead of just trusting it worked, turned up three places it has nothing to say:

1. It doesn't ask what a file left out. ref-stack.md's content was correct but undocumented as a deliberate exclusion until I fixed it by hand. The exercise asks "what is this file for," not "what did you leave out on purpose, and why."
2. It doesn't cover files that never make it into the curated set. It only runs on files that survive into the bundle. A file deleted outright, like design-app-theme.md, never gets asked the question at all.
3. It checks content, never shape. plan-marketing-disclaimers.md answered "what is this for" fine, and even scored well on a blind test. What it missed was a fourth question it never asks: is this the right shape for an active document, given how every file like it has already been handled? Checking content and checking shape turned out to be genuinely separate jobs, not two views of the same one.

What I'd tell someone about to do this

Don't claim "the AI gives better answers across the board": a fresh agent with raw grep access ties or beats a curated bundle on plain lookups once you account for outright deletions. Even after two rounds of fixes, the eventual win was earned across three test cycles, not assumed on the first one.

Do claim: cheaper retrieval, reliably fewer tool calls, and a likely edge at catching stale or contradictory claims; a structure that makes doc-to-doc drift visible instead of silent; and a process that checks both "what got deleted" and "what's still shaped wrong" as separate claims needing their own proof, because passing one test, even a strict one, isn't the same as being done.

Treat every deletion in a pruning pass as its own claim needing proof, separate from the "what is this file for" exercise. A file disappearing and a file being absorbed elsewhere look the same from outside the active set, and only one of them is safe. And once a test passes, run a second, different pass after it: one that checks every active file's shape against your own precedent, not against ground truth. Nothing else asks that question, and here, it was the only thing that caught the second bug.

Refined and ready

The whole process above is now a Claude Code Skill called okf-migrate: curate, check every claim against the real system where one exists, audit shape against precedent, run a blind test, and re-test instead of re-grade. It runs the same checks I ran by hand here: it detects whether you're curating an existing pile of docs, writing docs from a bare codebase with none, or checking an existing bundle for drift. It grades its own curation work with a fresh agent that never saw the session, because an agent that already knows where a bundle's gaps are will quietly route around them instead of showing them. That's the same blind spot this whole piece kept finding in my own manual process.

It has no dependencies (Python standard library only), is MIT-licensed, and is built to be extended, not used as a fixed product. Drop it into .claude/skills/ and it runs, no install step. It works with Claude only, for now. I want it working with other models and other agent tools next, because the problem it solves was never specific to one AI vendor: docs drifting quietly out of sync with what's true, invisible until an agent gives a wrong answer from them. It's on GitHub as okf-migrate.