experiment · growing
Retrieval eval, in isolation
Hit rate, recall@k, MRR, NDCG. "Was the right thing retrieved" comes before answer quality.
Before measuring RAG end to end, measure it in isolation: for a given question, were the retrieved chunks the right ones? Answer quality depends on that answer.
Rendering diagram…
The dataset
Input is a question, output is the ground-truth document ids that answer it. Write 20 by hand, then grow it by generating synthetic questions from chunks.
| Metric | What it tells you |
|---|---|
| Hit rate | is the right chunk in the top-k at all |
| recall@k | how many of the relevant chunks came back |
| MRR | how high the first correct result sits |
| NDCG | is the order right too — the metric for reranking |
Doing RAG without a benchmark is turning knobs without knowing which change improved what.