experiment · growing
End-to-end eval
Faithfulness, answer relevancy, context precision and recall. Label-free and labelled modes.
Isolated evals audit retrieval; end-to-end evals audit the answer the user sees. Both are needed, neither substitutes for the other.
Question
RAG pipeline
Answer + context
Evaluator
| Metric | The question it asks |
|---|---|
| Faithfulness | is every claim present in the shown context |
| Answer relevancy | does the answer address the question |
| Context precision | how much of the retrieved context was needed |
| Context recall | how much of the needed context arrived |
Two modes
- Label-free. No reference answer: faithfulness, relevancy, tone, toxicity. Cheap enough to run on every commit.
- Labelled. Ground-truth answers exist: correctness, coverage. Expensive to write, and the sharpest catcher of regressions.