RAG and fine-tuning architectures — from vanilla RAG to agentic RAG, with the measurement.
The whiteboard needs a wider screen — here are the notes in order.
This board reads left to right as one argument: problem → cheapest fix → cheap wins → expensive wins → measurement.
Only climb a step once you have measured the one below it. Jumping to agentic RAG without ever trying a different chunk size is putting the most expensive fix on top of the cheapest problem.
The default order: Prompt → RAG → Advanced RAG → Agentic → Fine-tune → Distill. Every step sits on the measured version of the one before it.
Everything a model knows is frozen in its training data. So it cannot do three things at once: be current, know your domain, cite a source.
RAG in one sentence: put what the model does not know in front of it before it answers. Retrieval-augmented generation — retrieve first, then generate.
Fine-tuning is also an answer, just not the first one: every new fact means labelling, cleaning and GPU time again. For a small corpus that is overkill.
The shortest split: RAG is for knowledge, fine-tuning is for form. Baking facts that change weekly into weights means retraining every week.
| RAG | Fine-tune | |
|---|---|---|
| Knowledge lives | outside the model | in the weights |
| Updating it | easy — reindex | hard — retrain |
| Inference cost | higher, context is full | lower, prompt is short |
| Context limit | yes, the window | none |
| Citing sources | natural | impossible |
| Good at | facts, fresh data | tone, schema, behaviour |
Vanilla RAG is two separate pipelines, and not confusing them is half the job: one runs offline and builds the index, the other runs on every request.
No router, no multi-step planning, no feedback. The model retrieves once and answers once. The real work is setting the contract between retrieval and generation: what you hand the model, and what you expect back.
The reference baseline — a working, plain, deliberately unclever vanilla RAG service. This is the zero point every measurement is compared against.
temperature = 0 is not an accident: during evals the variance in an answer should come from retrieval quality, not from sampling.
When naive RAG breaks it looks like a model bug from the outside. It is almost always a retrieval bug.
More retrieved tokens does not mean better performance. Raising top-k usually costs precision and makes the answer worse.
Tuning chunk size is the cheapest measurable win — but the thing that matters is not size, it is the boundary. A 512-token chunk cut through the middle of an idea is worse than a clean 1024.
| Chunk | Gain | Loss |
|---|---|---|
| Small (128–256) | high precision | context is severed |
| Medium (512–1024) | the balance for most corpora | — |
| Large (2000+) | context intact | blurred embedding, noise |
There is no single global chunking strategy. A contract, a table and a chat log do not survive the same splitter — the real work is classifying document type at ingest.
Metadata is context you can inject into every chunk: year, document title, page, section, owner. Raw semantic search confuses these at low precision.
The 2021 inside the question is not a meaning, it is a filter. Vector
similarity sees the 2020 and the 2021 document as nearly identical.
Dense vectors capture meaning and miss exact matches: error codes, SKUs, person names, version numbers. BM25 is good at exactly those.
Ask for chunk ids in the answer. A RAG system that cites nothing does not remove hallucination — it just makes it invisible.
Still unresolved for me: where the retrieval unit should sit relative to the embedding unit.
Options I keep circling:
Suspicion: the right answer is document-type dependent, which means the real work is classifying documents at ingest, not tuning one global strategy.
Intuition: embedding a big block of text is a bad idea. The vector blurs and the one sentence the question asks about drowns in noise.
The fix: embed the small thing, hand the big thing to the answer.
| Retriever | hit rate | MRR |
|---|---|---|
| Base (plain chunk) | 0.796 | 0.605 |
| Chunk references | 0.892 | 0.740 |
| Metadata references | 0.916 | 0.747 |
Source: LlamaIndex advanced-retrieval benchmarks. The absolute numbers move with the corpus; the ordering usually holds.
The sentence a user types is not a sentence written for search. Touching the query before retrieval is the biggest win available without touching the index.
Every transform is another LLM call — latency plus tokens. Measure first: how many of your queries are actually ambiguous? If most are one-shot, put this behind a router.
Vanilla RAG is one shot: if it retrieved the wrong thing, it cannot recover. What actually raises quality is turning retrieval into a closed loop.
Agentic RAG is not a fixed "retrieve and generate" pipeline; it is an orchestrator in which the model runs its own search process.
| Traditional (reactive) | Agentic (proactive) | |
|---|---|---|
| Flow | one way, deterministic | cyclical, dynamic |
| Searches | one | as many as needed |
| Data quality | never questioned | evaluated |
| Missing info | goes unnoticed | triggers a new strategy |
It does not work without a solid middle column. Without metadata, hybrid search and reranking, the agentic layer only makes bad retrieval expensive.
Agentic RAG is not one architecture, it is a family of patterns. Do not build them all — add one at a time, knowing which failure it fixes.
Agentic RAG wins on accuracy and costs real engineering effort in latency, tokens and failure surface. The decision is that trade.
| Dimension | Vanilla RAG | Agentic RAG |
|---|---|---|
| Latency | low, ~1–2 s | high and variable |
| Token cost | low, one query | high, history re-read each round |
| Complexity | a simple pipeline | state management + tool wiring |
| Accuracy on hard tasks | low, context breaks | high, cross-synthesis |
| Failure surface | narrow: search and generate | wide: planning, tools, loops |
If you cannot measure the accuracy gain, the latency and tokens you pay are real and the gain is an assumption.
An agent is a loop with a stopping condition. Most of the engineering is in the stopping condition.
A large context window is a budget, not a bucket. Everything you put in it competes for the model's attention and for your latency target.
Measure before trimming. Half the "context is too big" problems are really one tool returning an unpaginated JSON blob.
Fine-tuning is for form, not for facts. Burning knowledge that changes weekly into weights means retraining on every change.
Do not break the order: Prompt → RAG → Fine-tune → Distill. Reaching for a fine-tune before measuring RAG is renting GPUs without knowing the problem.
Training from scratch is almost no product team's job. In practice the choice is which parameter-efficient method.
| Method | What it does | When |
|---|---|---|
| Full SFT | updates every weight | rarely — costly, forgetful |
| LoRA | low-rank adapter on a frozen model | the default starting point |
| QLoRA | LoRA over a 4-bit base | fitting onto one GPU |
| DPO | aligns on preference pairs | you have good/bad answer pairs |
| GRPO | RL against a reward function | correctness is programmatically checkable |
| Distillation | copies a big model into a small one | cost and latency pressure |
An adapter does not replace retrieval. A fine-tuned model still invents the document it never saw — just in a nicer format.
The least-discussed way to improve retrieval: fine-tune the embedding, not the model. On a corpus full of domain jargon this is the cheapest win there is.
A general-purpose embedding model has never seen your company's acronyms, part numbers or internal vocabulary. Synthetic question generation closes that gap without a single hand-written label.
Before measuring RAG end to end, measure it in isolation: for a given question, were the retrieved chunks the right ones? Answer quality depends on that answer.
Input is a question, output is the ground-truth document ids that answer it. Write 20 by hand, then grow it by generating synthetic questions from chunks.
| Metric | What it tells you |
|---|---|
| Hit rate | is the right chunk in the top-k at all |
| recall@k | how many of the relevant chunks came back |
| MRR | how high the first correct result sits |
| NDCG | is the order right too — the metric for reranking |
Doing RAG without a benchmark is turning knobs without knowing which change improved what.
Isolated evals audit retrieval; end-to-end evals audit the answer the user sees. Both are needed, neither substitutes for the other.
| Metric | The question it asks |
|---|---|
| Faithfulness | is every claim present in the shown context |
| Answer relevancy | does the answer address the question |
| Context precision | how much of the retrieved context was needed |
| Context recall | how much of the needed context arrived |
Prompt tuning without evals is just vibes with extra steps. The eval does not need to be sophisticated; it needs to exist before the prompt changes.
| Kind | Count | What it catches |
|---|---|---|
| Golden | 10 | regressions on the happy path |
| Adversarial | 5 | prompt injection, refusal bait |
| Boring | 5 | empty input, huge input, wrong language |
Use one when the answer is genuinely open-ended, and then: