GenAI Course · Module 9 · Session 36

RAG Evaluation

⏱ 13 min read🎯 Advanced🖼 2 diagrams🧪 1 live demo🐍 Python code

“It looks good in the demo” is not evaluation. Before a RAG system goes live, you need numbers: is it finding the right chunks, and is it answering faithfully from them? In this session, let us build a test set and compute real metrics.

In this session

Building a golden question–answer dataset
Retrieval metrics: hit rate, MRR, context precision and recall (with real numbers)
Generation metrics: faithfulness, groundedness, answer relevancy
RAGAS and DeepEval in code

01Build a golden dataset

A golden dataset is a list of real questions with the correct answer and the correct source chunk. It is your exam paper for the RAG system.

Start with 100–300 questions from real user logs and from subject experts.
For each: the question, the expected answer, and the relevant chunk id(s).
Include hard cases: vague questions, Hinglish, numbers, questions with no answer in the documents.
Keep it versioned; re-run it after every change to chunking, embeddings, prompts or models.

golden.jsonl

{"question": "What is the penalty for breaking an FD early?", "answer": "The interest rate is reduced by 1%.", "relevant": ["fd_p4"]}
{"question": "Do senior citizens get extra interest?", "answer": "Yes, an extra 0.50% on all tenures.", "relevant": ["fd_p2"]}
{"question": "What is the home loan rate for NRIs?", "answer": "NOT_IN_DOCS", "relevant": []}

02The evaluation loop

Every change to chunking, embeddings, prompts or models goes round this loop before release.

03Retrieval metrics

Metric	Question it answers
Hit rate@k	Was the right chunk anywhere in the top k?
MRR (Mean Reciprocal Rank)	How high was the first right chunk? (1st → 1, 2nd → 0.5, 3rd → 0.33)
Context precision	Of the retrieved chunks, how many were actually relevant?
Context recall	Of all the relevant information, how much did we retrieve?

retrieval_metrics.py

def hit_rate_and_mrr(results, relevant, k=5):
    hits, rr = 0, 0.0
    for q, ranked in results.items():
        top = ranked[:k]
        if relevant[q] in top:
            hits += 1
            rr += 1 / (top.index(relevant[q]) + 1)
    n = len(results)
    return hits / n, rr / n

results = {                                   # what the retriever returned, best first
    "FD penalty?":        ["fd_p4", "fd_p1", "sav_p2"],
    "Senior citizen rate?": ["fd_p2", "fd_p4", "loan_p1"],
    "Min balance?":       ["fd_p1", "sav_p3", "sav_p2"],
    "KYC documents?":     ["card_p1", "loan_p2", "fd_p9"],
    "Card block steps?":  ["card_p7", "card_p3", "kyc_p1"],
}
relevant = {"FD penalty?": "fd_p4", "Senior citizen rate?": "fd_p2", "Min balance?": "sav_p3",
            "KYC documents?": "kyc_p2", "Card block steps?": "card_p3"}
hr, mrr = hit_rate_and_mrr(results, relevant, k=3)
print(f"Hit rate@3 = {hr:.2f}   MRR = {mrr:.2f}")

Hit rate@3 = 0.80 MRR = 0.60

Demo · How we got MRR = 0.60

04Generation metrics

Metric	Question it answers	Catches
Faithfulness / groundedness	Is every claim in the answer supported by the retrieved context?	hallucinations
Answer relevancy	Does the answer actually address the question?	off-topic or evasive answers
Answer correctness	Does it match the golden answer?	wrong facts

These are usually scored by an LLM judge that compares the question, the retrieved context and the answer.

Faithfulness checks the answer against the retrieved context, not against the model’s memory.

05RAGAS and DeepEval

You don’t have to write the LLM judges yourself. Two popular open-source libraries:

ragas_eval.py

# pip install ragas datasets      (API shown for ragas 0.1.x; newer versions use EvaluationDataset)
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

data = Dataset.from_dict({
    "question":     ["What is the penalty for breaking an FD early?"],
    "answer":       ["The interest rate is reduced by 1%."],
    "contexts":     [["Breaking an FD early reduces the interest rate by 1%."]],
    "ground_truth": ["The interest rate is reduced by 1%."],
})
print(evaluate(data, metrics=[faithfulness, answer_relevancy, context_precision, context_recall]))

deepeval_test.py

# pip install deepeval
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

case = LLMTestCase(
    input="What is the penalty for breaking an FD early?",
    actual_output="The interest rate is reduced by 1%.",
    retrieval_context=["Breaking an FD early reduces the interest rate by 1%."],
)
metric = FaithfulnessMetric(threshold=0.8)
metric.measure(case)
print(metric.score, metric.reason)

Make it a habit: run the golden set in CI on every change and block a release if faithfulness or hit rate drops. LLM judges can be wrong too, so spot-check a sample by hand.

06Summary

A golden dataset (question, answer, relevant chunks) is your exam paper.
Retrieval: hit rate, MRR, context precision and recall.
Generation: faithfulness, answer relevancy, correctness, usually via an LLM judge.
RAGAS and DeepEval automate the metrics; run them on every change.

07Quick quiz

1. The first relevant chunk is ranked 2nd. Its reciprocal rank is…

2. Faithfulness measures…

3. Why include questions with no answer in the docs?

4. Hit rate@3 = 0.80 means…

5. When should you re-run the golden set?

08References

Created by Ajeetkumar