Evaluating RAG in production

A RAG pipeline can look great when it only has to answer five demo questions.

Then real users show up. Documents change. Someone asks a question nobody planned for. A metadata filter quietly removes the right page. The final answer still sounds confident, so the failure is easy to miss.

Two things make this much easier to deal with:

Evaluation tells us whether the system is good.
Tracing shows what happened on one request.

An eval can tell us an answer was wrong. A trace can tell us the correct chunk landed at rank eight, while only the top five chunks reached the model.

Now it is not just “the AI gave a bad answer.” It is a retrieval setting we can test.

A RAG answer has several failure points

A normal request moves through more than one system:

A bad answer can start anywhere in that path.

Maybe the document was never indexed. Maybe the right chunk was found but ranked too low. Maybe the model got the right evidence and ignored it. The answer can even be right while the citation points to the wrong source.

If we treat all of this as one black box, every fix turns into another prompt tweak. A better split is:

Retrieval
Did the system find the required evidence?
 
Generation
Did the model use that evidence well?

Those are different problems, so they need different checks.

The evaluation set

An eval set is just a collection of questions with the behaviour we expect. People call it a golden set, benchmark, or eval dataset. Same idea.

A small case might look like this:

{
  "question": "Can I get a refund after 30 days?",
  "relevant_chunk_ids": ["refunds-2"],
  "reference_answer": "No. Refunds are only available within 14 days.",
  "should_refuse": false
}

The name does not matter. What matters is keeping the same cases around and rerunning them after changing chunk size, embeddings, retrieval, reranking, prompts, or models.

The best cases usually come from real failures, not a neat list written before launch:

Over time, the eval set becomes a list of mistakes the system should not repeat.

Retrieval metrics

To measure retrieval, we first need to know which chunks count as relevant for each question.

Labelling them takes work, but without those labels we are mostly guessing whether a retrieval change helped.

`top_k` is a setting, not a metric

top_k is simply how many results the retriever returns.

top_k = 3

means “give me the three highest-ranked chunks.”

A larger K gives retrieval more chances to include the right evidence. It also brings more noise, more tokens, and more latency. More context does not always mean a better answer.

Hit@K

Hit@K asks one very small question:

Did at least one useful chunk appear in the first K results?

If refunds-2 is the only relevant chunk:

1. subscriptions-1
2. refunds-2
3. pricing-4

then:

Hit@1 = 0
Hit@3 = 1

Across the full eval set, it is the percentage of questions where the answer is yes.

Hit@K works well when one good chunk is enough.

Recall@K

Recall starts with everything the answer needed.

Suppose a question needs three chunks:

refunds-2
cancellations-1
data-retention-3

The retriever returns five results:

1. refunds-2          relevant
2. pricing-4          noise
3. cancellations-1   relevant
4. account-7          noise
5. security-2         noise

It found two of the three required chunks:

Recall@5 = relevant chunks found / all relevant chunks
         = 2 / 3
         = 0.67

In plain English:

How much of the required evidence did retrieval find?

The 3 is everything we wanted to find.

This matters for questions that need more than one source. The example passes Hit@5 because it found something useful. But recall is only 0.67 because data-retention-3 is still missing.

Precision@K

Precision looks at the exact same results, but asks a different question.

The retriever returned five chunks, but only two were relevant:

Precision@5 = relevant chunks found / chunks returned
            = 2 / 5
            = 0.40

The 5 is everything we actually returned.

So the same retrieval run has:

Metric	Score	What it says
Recall@5	`2 / 3 = 0.67`	Found two of the three required chunks
Precision@5	`2 / 5 = 0.40`	Only two of the five returned chunks were useful

The easiest way I remember the difference:

Recall: What did we miss?
Precision: How much noise did we include?

The usual tension looks like this:

MRR

Mean Reciprocal Rank cares about how early the first useful result appears.

rank 1: 1 / 1 = 1.00
rank 2: 1 / 2 = 0.50
rank 3: 1 / 3 = 0.33
no relevant result: 0

For three questions with reciprocal ranks 1, 0.5, and 0:

MRR = (1 + 0.5 + 0) / 3 = 0.50

MRR rewards getting the first useful result near the top. It works well when one strong result can answer the question. It does not tell us whether we found every piece of evidence.

No retrieval metric tells the whole story. Hit@K can hide missing evidence. Recall can go up just because we returned more stuff. Precision can look great with a tiny K that misses an important chunk. The useful metric depends on what the question needs.

Generation metrics

Finding the right chunks is only half the job. We still need to check what the model did with them.

Faithfulness

Faithfulness asks:

Can every claim in the answer be backed up by the context?

Claim 1: Refunds are available within 14 days.       supported
Claim 2: Access continues until the billing period. supported
Claim 3: Account data is deleted immediately.       unsupported

A claim-level score could be:

faithfulness = supported claims / total claims
             = 2 / 3

The third claim came from nowhere in the provided context.

One catch: a faithful answer can still be wrong. If the source is outdated, the model can faithfully repeat outdated information. Faithfulness checks whether the answer stayed close to the context. It does not prove the source itself is true.

Correctness

Correctness is the obvious one: is the answer actually right? We usually compare it with a reference answer or human judgment.

Reference:
No. Refunds are only available within 14 days.
 
Generated:
No. A refund must be requested during the first 14 days.

The wording is different, but the meaning is the same. Exact text matching is usually a bad fit here.

Completeness

Completeness checks whether the answer covered all the important parts.

Question:

What happens to my subscription and data after cancellation?

Required points:

- access continues until the billing period ends
- data is retained for 30 days

If the answer only mentions continued access, it is not wrong. It is just missing half the answer.

Answer relevance

Answer relevance asks whether the response actually answered what the user asked.

Question:
Can I get a refund after 30 days?
 
Answer:
We offer monthly and annual plans.
You can manage them from the billing dashboard.

Everything in that answer may be true. It is still not an answer to the question.

Citation correctness

A citation is only useful when it supports the claim next to it.

Refunds are available within 14 days. [refunds-2]

is useful only if refunds-2 contains that policy.

There are really two checks:

Does every citation point to a chunk that was actually retrieved?
Does that chunk support the cited claim?

The first can be checked with code. The second needs someone—or another model—to understand the meaning.

Refusal accuracy

Sometimes refusing is the right answer.

If the documents say nothing about office locations, a question about a Tokyo office should get an honest “I do not have enough information,” not a believable guess.

Refusal evaluation needs both sides:

refuse when the evidence is insufficient
answer when the evidence is sufficient

But pushing refusals too hard creates a bot that says “I do not know” to everything. It looks safe. It is also useless.

Three layers of evaluation

There is no single perfect evaluator. A practical setup uses three layers.

Deterministic checks

Anything exact should be checked with code:

retrieval returned at least one chunk
cited chunk IDs exist in the retrieved set
output matches the required schema
latency stayed below a threshold
the answer is not empty
model and index versions were recorded

These checks are cheap and consistent, so they can run all the time. They just cannot understand whether a paragraph is genuinely supported by a source.

LLM judges

An LLM judge gets the question, context, generated answer, reference answer, and a rubric. Then it scores things like faithfulness or completeness.

Faithfulness: 4/5
Correctness: 3/5
Completeness: 2/5

This scales better than reading every answer by hand. But the judge is still a model. It can favour confident writing, change its mind between runs, or struggle with a vague rubric.

Good judge setups are boring on purpose:

use a specific rubric
ask for evidence with the score
keep temperature low
pin the model and prompt version
calibrate against human-reviewed cases
measure agreement on important slices

An LLM judge is a measuring tool, not the final source of truth.

Human review

Human review is slower, but it catches the weird stuff: unclear rubrics, stale answers, and failure types nobody thought to automate.

A simple review record is enough:

{
  "retrieval_passed": true,
  "faithful": true,
  "correct": true,
  "complete": false,
  "citation_correct": true,
  "notes": "The answer omitted the data-retention period."
}

For a small project, reviewing 20 to 50 questions properly can teach us more than a huge auto-scored dataset nobody has looked at.

Offline evals and production signals

Offline evals run a fixed benchmark before deployment.

Version A
chunk size: 300
top_k: 3
Hit@3: 0.84
MRR: 0.72
faithfulness: 0.89
 
Version B
chunk size: 500
top_k: 5
Hit@5: 0.93
MRR: 0.74
faithfulness: 0.80

Version B finds relevant evidence more often, but the answers are less faithful. Maybe the larger chunks or extra results added conflicting context.

This is why chasing one metric is risky. Retrieval quality, answer quality, latency, and cost all pull on each other.

Useful changes to compare offline include:

parsing and chunking
embedding models
keyword, semantic, or hybrid retrieval
metadata filters
candidate count and final top_k
rerankers
context ordering
prompts
generation models

Then production brings all the questions the benchmark never saw coming.

Useful signals include:

thumbs up or down
corrections and reformulated questions
support escalations
empty-retrieval and no-answer rates
refusal rate
citation clicks
latency, tokens, and cost
errors by model, index, tenant, or document version

These are clues, not proof. A user can like an answer because it sounds good even when it is wrong. A repeated question might mean the first answer failed, or the user might simply need the information again.

The useful pattern is simple: use production signals to find suspicious cases, review them, then turn real failures into eval cases.

Traces make failures local

A trace is the full story of one request.

For RAG, useful trace fields include:

question
rewritten query
metadata filters
embedding model
retrieved chunks and scores
reranked chunks and scores
final context
prompt version
generation model
answer
citations
latency
token usage
errors
user feedback

Normal logs may show a 200 and call it a day. A trace can show that the request worked technically, but the correct chunk ranked sixth while only four chunks reached generation.

We want enough detail to replay the decision path, but that does not mean storing every prompt and document forever. Traces still need access controls, retention limits, and redaction.

Reading failures from a trace

Most failures start looking familiar once the trace is visible.

The correct chunk was never retrieved

retrieved:
1. pricing-1
2. subscriptions-2
3. accounts-4
 
expected:
refunds-2

This could be bad parsing, awkward chunking, weak embeddings, a query mismatch, the wrong metadata filter, a stale index, or simply a K that is too small.

The correct chunk ranked too low

1. pricing-1
2. subscriptions-2
3. accounts-4
...
7. refunds-2

This looks like a ranking problem, not a generation problem. Hybrid search, better metadata, a larger candidate pool, or a reranker might help.

The candidate pool and final context size should stay separate:

That gives the reranker more options without dumping all 20 chunks into the model context.

The model received the evidence and answered badly

Context:
Refunds are available only during the first 14 days.
 
Answer:
Refunds are available within 30 days.

Retrieval did its job. The problem is probably later in the pipeline: conflicting context, poor ordering, a weak prompt, the wrong model, or one important line buried under too much text.

The answer is right but the citation is wrong

This is usually a citation-mapping or output-format issue. The prose being right does not make the citation right too.

The trace looks right but the eval fails

Sometimes the pipeline is fine and the eval is wrong.

Reference answers become stale. Relevant chunk IDs change after re-indexing. A judge rubric may punish valid phrasing. Documents may change while the golden set does not.

The eval setup needs versioning and review too. Otherwise we end up debugging the system against a broken test.

The oracle-context test

The quickest way to separate retrieval from generation is to skip retrieval once.

Run the same question twice:

If the normal answer is wrong but the oracle answer is correct, retrieval is probably the issue.

If both answers are wrong, the problem is more likely in context construction, the prompt, the model, or the expected answer.

It is a small test that removes a lot of guessing.

A useful trace shape

There is no perfect trace schema. It just needs to keep the details that can change the answer.

{
  "trace_id": "trace-9281",
  "question": "Can I cancel my annual plan and receive a refund?",
  "query": "annual plan cancellation refund",
  "retrieval": {
    "strategy": "hybrid",
    "index_version": "policies-2026-07-18",
    "candidate_k": 20,
    "results": [
      {
        "chunk_id": "cancellations-1",
        "score": 0.91
      },
      {
        "chunk_id": "refunds-2",
        "score": 0.86
      }
    ]
  },
  "reranking": {
    "model": "reranker-v2",
    "final_k": 5
  },
  "generation": {
    "model": "generation-model-v3",
    "prompt_version": "rag-prompt-v4",
    "answer": "Your plan remains active until the billing period ends...",
    "citations": ["cancellations-1", "refunds-2"]
  },
  "metrics": {
    "retrieval_latency_ms": 42,
    "generation_latency_ms": 840,
    "total_tokens": 1240
  }
}

If it can change the result, record its version:

parser and chunking configuration
embedding model
index snapshot
retrieval strategy and filters
candidate count and final K
reranker
prompt
generation model
judge model and rubric

Without those versions, we can see that quality dropped but not what changed.

The production loop

The production loop should be more than “change the prompt and try again.”

Running the full benchmark matters. A fix for one bad answer can quietly break ten others.

The compact mental model:

eval      = did the system behave well?
trace     = what happened on this request?
metric    = how are we measuring one property?
golden set = which behaviours should stay fixed?
oracle test = was retrieval actually the problem?

Evals tell us something broke. Traces tell us where. Then it becomes normal engineering: fix the right stage, rerun the benchmark, and keep the failure as a test.