AgenticRAG: Letting LLMs Hunt for Evidence. Here's the Full Agentic Upgrade
I think you know that most RAG systems still work like this: ask a question, grab some chunks, stuff them into a prompt, and pray
One shot. No second chances
Fine for easy stuff. Falls apart the second your question gets messy, the kind where you'd open five tabs and rethink your search terms twice
I dug into how the newer agentic RAG systems actually work under the hood, then built the whole thing as real, runnable code. An actual harness with a retrieval backend, a tool-using agent loop, and a token budget that forces itself to summarize before it blows up. Copy the whole thing
The Problem In One Line
Classic RAG finishes searching before it starts thinking. The model never gets a say in its own search. It just reacts to whatever got handed to it
Agentic RAG flips that. The model runs the search itself, in a loop, deciding when it has enough
The 4 Piece Architecture:
1\ Loop – model decides each round: search more or answer now
2\ Search – broad, multiple rewritten queries at once
3\ Find + Open – targeted digging inside specific documents
4\ Summarize – forced checkpoint before context blows up
Below is the full build. Two files. Runs locally against a sample corpus, swap the corpus for your real index and you're production ready
One honest note before you copy anything: treat every number here as a dial, not a rule
File 1: the retrieval backend
This is the "dumb" part your agent controls
In production you'd swap this for Elasticsearch, a vector DB, whatever you already run. The interface matters. Every result gets a stable reference ID the moment it's first seen, and find/open only ever take that ID, never a raw file path
That's what keeps citations traceable across a long run
I tested this before writing a single line of the agent on top of it. Ran it against a sample revenue doc and an unrelated OCR runbook. Search correctly ranked the revenue doc first with a score of 1.87 versus 0.06 for the unrelated doc. find pulled every line mentioning "annual" with exact line numbers. open returned a clean numbered window. If your retrieval layer can't do this much on its own, fix that before you touch the agent loop
File 2: the agent harness
This is the part that actually decides when to search, what to open, and when it has enough. It runs on Claude's tool use, tracks its own token spend, and forces itself to checkpoint before it blows the context window
How to actually run this
You'll see live cycle-by-cycle output showing token spend and which tool the model reached for. That visibility alone is worth building this even if you never ship it, watching an agent choose search versus find versus open on a real question teaches you more about your own document structure than any dashboard will
The 6 box checklist for adapting this to your own stack
The one test that exposes a fake "agentic RAG"
Ask it: "Give me the exact reference_id and location for that last claim."
Can't produce one? It's not doing agentic retrieval, it's just pattern matching on whatever got stuffed in the prompt.
Before you build any of this, run the gut check
Only worth the extra complexity if all of these are true:
If you're running a simple FAQ bot for "what are your business hours," skip all of this. You'd be burning tokens on a Ferrari to buy milk..
The Real Takeaway
It's a systems decision: split the model's job (deciding what to search and when to stop) from the search stack's job (actually running the query). Once separated, you can upgrade either one on its own, swap the backend, tune the cycle cap, change the summarization trigger, without touching the other half.
Open question worth sitting with: how much of the gain here is the model being smart, versus just giving it more time, more tool calls, and a better policy for using them?
Worth testing before you assume you need the biggest model on the market
Save this so you don't lose it
Follow @beamnxw for more technical posts :)
