A retrieval-augmented generation system, usually shortened to RAG, can give the right answer for the wrong reason. It can retrieve the wrong passage, ignore the passage it retrieved, and still land on a correct-looking answer because the language model already knew it. It can also retrieve exactly the right passage and still hallucinate a detail that was never in it. If you only measure whether the final answer matches a reference answer, you will miss both failures, and you will fix the wrong component when performance drops.
This article explains how to evaluate RAG as a system with two distinct jobs: finding the right evidence, and using only that evidence to answer. The distinction matters because the fixes are different. Bad retrieval needs better chunking, embeddings, or ranking. Bad generation needs better grounding, prompting, or abstention. An end-to-end score blends them into one number that tells you neither.
The two jobs of a RAG system
A RAG pipeline has a retriever and a generator.…