Knowledge Compiler: an agentic knowledge system

A tool that uses an agent to find related notes, propose merges, updates or new entries, and keep a growing knowledge base coherent.

Engineering notes·4 min read

Knowledge Compiler starts from a practical problem: notes are easy to capture and hard to maintain. As a knowledge base grows, related ideas scatter, old notes go stale, and contradictions hide in the pile. The tool treats each raw note as source material: an agent searches the existing base, finds related concepts and blocks, drafts a merge, update or new-entry proposal, and stops for human approval before anything is written.

A note is an input to be compiled into knowledge, not a finished document to be filed and forgotten.

The challenge

A knowledge base decays in predictable ways as it grows. There's soon too much of it to hold in your head, so you stop revisiting it. Older notes quietly go out of date without anyone noticing. The same topic ends up scattered across a dozen half-finished notes, and every so often a new note flatly contradicts an old one. On top of that, every time you add something, reconciling it against everything you've already written is slow and tedious, so most people simply don't, and the base rots.

Agentic workflow

The fix is to hand that reconciliation to an agent. When you add a raw note, which the system calls a source, the agent runs a small, guarded tool loop to work out how that note should land in the base. It can load the raw source (get_source), resolve the note's concepts against the concept index with optional fuzzy matching (lookup_concepts), and search the existing knowledge for related material (search_blocks), since the base is stored as blocks rather than whole notes. When it needs more detail it can read a specific block or its edit history (get_block and get_block_history), and it can check that a draft is genuinely grounded in the source before committing to it (verify_grounding).

The loop only ends when the agent reaches a decision. Two terminal tools close it out: draft_proposal classifies the outcome as a merge, an update or a new entry and drafts the change, while finish_without_proposal stops cleanly when a complete proposal can't be made.

What keeps this reliable is that the order is enforced rather than hoped for. Each tool carries a guard, so nothing runs before the source is loaded, and a block can't be fetched unless an earlier search actually surfaced it, which stops the agent acting on an id it made up. Just as important, whatever the loop produces is only a proposal. It never writes to the knowledge base directly. You review the merge, update or new entry and decide what to apply, so the agent does the tedious work of finding what's related and drafting the change while you keep authorship and the final call.

The plumbing underneath is deliberately boring. The agent runs on the OpenAI Agents SDK, behind a clean-architecture Express and TypeScript backend where routes stay thin, services hold the business logic without touching raw SQL, and repositories own the database queries. Data lives in PostgreSQL, and the client is React with Vite.

Hybrid retrieval and eval

Answering a question reuses the same knowledge through a different path. Retrieval is hybrid and runs mostly at the block level: a query fans out to several retrievers at once and their results are combined. A Postgres full-text search acts as a baseline fallback. BM25 lexical ranking, through ParadeDB, handles keyword relevance. For meaning rather than wording, the query is embedded and matched by cosine similarity through pgvector. There is also a note-level match against the concept index, though that one is currently dormant, because the query-side concept resolver isn't wired up yet, so it contributes no candidates for now.

From there the pipeline is fixed. The separate rankings are merged with Reciprocal Rank Fusion (k=60), then expanded one hop across approved note links to pull in directly connected notes. That produces the top N blocks, each carrying its citation, which are handed to an answerer running an exact-question scoping prompt so the reply stays tied to what was actually retrieved.

Evaluation falls naturally out of this shape. Because every answer is assembled from cited blocks, and because the compile-time agent's drafts have to pass a grounding check, the system is built to be judged on whether an answer is genuinely supported by its sources rather than merely plausible.

Future work

There's plenty I still want to do. The most concrete item is wiring up the query-side concept resolver so the dormant concept-index retriever actually contributes candidates at query time. Beyond that I want the agent to get sharper at spotting genuine contradictions and at proposing a right-sized merge instead of over-merging, to expose the concept graph so the base can be browsed by concept rather than only searched, and to make the review step fast enough that accepting or rejecting a batch of proposals is effortless.

Conclusion

An AI-assisted knowledge system earns its keep by doing the reconciliation you would otherwise skip. It finds what's related by concept and proposes what to merge, update or add, then it stops and lets you decide. The agent proposes and the human approves, and that division of labour is what lets a knowledge base keep growing without quietly falling apart.