Agentic RAG

Introduction

One of my first articles becoming a software engineer. I outlined the reason I began pivoting into a software career. At the time, CRUD applications were the standard. Now that I have been in the field for years. I noticed a new shift. There is no such thing as an AI Engineer, there is a software engineer and there is a person out of a job. Every software engineer needs to know AI development along with all the other things they once had to know. This Blog will be talking about one of those processes called RAGRetrieval-Augmented Generation: fetch relevant documents at query time and put them in the prompt so the model can answer from them.

RAG stands for Retrieval-Augmented Generation. Which fetches relevant documents at query time and puts them in the prompt so the model can answer from them. The version that actually works well in practice is the agentic kind.

This post builds up from embeddings to a basic RAG pipeline to the agentic loop, with a demo for the two parts that are hardest to picture.

Why retrieval at all

A model's weights are frozen (unfortunately - call back to the end of this article tokens, weights, and temperature). It knows what was in its training data and nothing after. It doesn't know your company's PTO policy, your API's error codes, or what changed in last week's deploy.

You could fine-tune, but that's slow, expensive, and the model still hallucinates details. The simpler fix is to put the relevant text directly in the prompt. The model is good at reading; the hard part is deciding what to put there. That's retrieval.

Embeddings

To find relevant text you need a way to measure "relevant." Keyword search is brittle: "vacation days" won't match a document that says "PTO accrual." Embeddings fix this.

An embedding model turns a piece of text into a vector, a list of a few hundred to a few thousand numbers, such that texts with similar meaning end up with similar vectors. "How many vacation days do I get?" and "PTO policy: 20 days per year" land close together even though they share no words.

Similarity between two vectors is usually cosine similaritythe cosine of the angle between two vectors: 1 means they point the same way, 0 means unrelated, -1 means opposite, which ignores length and only looks at direction.

Below is a toy knowledge base: eleven chunks from an imaginary company handbook, embedded into two dimensions so you can see them. Pick a query or click anywhere to drop your own. The three nearest chunks are what a retriever would hand to the model.

Notice the clusters. The embedding model has never seen these specific docs, but it knows that "PTO" and "parental leave" are the same kind of thing, so they end up neighbors. That's the whole trick.

Basic RAG

With embeddings, the simplest pipeline is:

1. Index. Split your documents into chunks (a few hundred tokens each), embed every chunk, and store the vectors in a database that can do nearest-neighbor search.
2. Retrieve. Embed the user's question with the same model. Find the top-k closest chunks.
3. Generate. Stuff those chunks into the prompt along with the question and let the model answer.

question = "How many vacation days do I get?"
q_vec = embed(question)
chunks = vector_db.search(q_vec, top_k=3)

prompt = f"""Answer using only the context below.

Context:
{"\n\n".join(c.text for c in chunks)}

Question: {question}"""

answer = llm(prompt)

This works surprisingly well for simple questions and it's where most teams start. It's also where most teams get stuck, because it has a structural weakness.

Where one-shot RAG breaks

The pipeline retrieves once, using the user's exact words, and then has to answer with whatever came back. Each of those is a problem:

• Multi-hop questions. "Can I expense the hotel for the on-call offsite?" needs the travel policy and the offsite attendance rules. One query, phrased as the user phrased it, lands somewhere between them and retrieves neither cleanly.
• Vocabulary mismatch. The user says "prod is down," the runbook says "declare an incident." Embeddings help but don't fully close the gap.
• No way to say "not enough." If the retrieved chunks are irrelevant, the model either admits it or, more often, confidently answers from the wrong context.
• Fixed k. Three chunks is too many for a lookup and too few for a summary.

Agentic RAG

The fix is to stop treating retrieval as a preprocessing step and make it a tool the model can call, as many times as it needs, with queries it writes itself.

The loop looks like this:

1. Plan. Read the question, figure out what facts are needed.
2. Retrieve. Write a focused search query for one of them. Run it.
3. Evaluate. Read what came back. Does it answer the sub-question? Is anything still missing?
4. Repeat steps 2 and 3 until the plan is satisfied, or give up after a budget.
5. Answer, citing the chunks that were actually used.

Same question, both pipelines, side by side. Step through each one.

The one-shot version retrieves with the whole question, gets a mushy mix of half-relevant chunks, and produces an answer that's technically grounded but doesn't resolve the question. The agentic version splits the question, writes two sharper queries, notices after the first that it's not done, and comes back with a cited answer.

What the agent actually is

There's no special architecture. "Agentic" just means the model is in a loop with tools. In practice the skeleton is:

def agentic_rag(question, max_steps=6):
    messages = [system_prompt, user(question)]
    for _ in range(max_steps):
        response = llm(messages, tools=[search_tool])
        if response.tool_call is None:
            return response.text          # model decided it can answer
        results = vector_db.search(response.tool_call.query, top_k=3)
        messages.append(response)         # the model's own query
        messages.append(tool_result(results))
    return llm(messages + [user("Answer with what you have.")]).text

The model sees its previous searches and their results in the conversation, so each new query can build on the last. The system prompt tells it to search before answering, to break compound questions apart, and to say when it can't find something. That's it.

Things that matter in practice

• Chunking is most of the quality. Chunks that split mid-sentence or mid-table retrieve badly. Chunk on structure (headings, paragraphs, functions) and include a title or breadcrumb in each chunk's text.
• Hybrid search. Combine vector similarity with plain keyword search (BM25) and merge the rankings. Embeddings miss exact identifiers like error codes and ticket numbers; keywords catch them.
• Reranking. Retrieve 20 chunks cheaply, then have a small model score each one against the query and keep the top 5. Big precision win for little cost.
• Cite or it didn't happen. Make the model quote which chunk each claim came from. It cuts hallucination and makes failures debuggable.
• Cap the loop. Agents that can search forever will. Give them a step budget and a way to say "I couldn't find this."
• Log every query the agent wrote. The queries are the best signal for what's missing from your index.

Conclusion

Embeddings turn meaning into geometry, so "relevant" becomes "nearby." Basic RAG uses that once, with the user's words, and hopes. Agentic RAG hands the search to the model, lets it ask several sharper questions, and lets it notice when it doesn't have enough yet. The code difference is a loop and a tool definition. The quality difference, on anything beyond a single lookup, is the whole product.