How to evaluate a RAG before trusting it

A convincing answer can be worse than no answer at all. In a business, the problem with a RAG system is not only that it can be wrong; it is that it can be wrong with confidence in front of the person who has to act today.

The important test is not whether it responds quickly, but whether it knows when to stay quiet.

That is the standard we should apply before trusting a business RAG: not how smart it sounds, but how it behaves when documentation is incomplete, when sources conflict, and when the question does not deserve an automatic answer. That changes operations far more than a polished demo does.

The common mistake: evaluating a RAG like a search box

The usual trap is to test the system with polite questions. “What is the returns policy?”, “How do I request leave?”, “Where is the expense policy?” That can make it look good, but it still does not tell you whether it can support real work.

RAG stands for Retrieval-Augmented Generation. In plain terms, it combines search and generation. That means its risk is not only in what it writes, but in what it retrieves, what it ignores, and what it does when retrieval is weak or contradictory. If you only inspect the final sentence, you are looking at the shop window, not the workshop.

The practical question is more uncomfortable: can the system show where each answer comes from, and what does it do when it does not know? If you cannot answer that, you do not have a trustworthy assistant; you have a machine that sounds certain.

Five questions that separate help from noise

Before you approve a RAG, ask five very concrete questions:

1. Which sources is it expected to use, and which are out of scope?
2. Does it cite the origin of the answer in a verifiable way?
3. Does it abstain when evidence is weak?
4. Does it ask for clarification when the question is ambiguous?
5. Does it escalate to a person when the risk is operational or legal?

You do not need a lab to see this. You need a small set of real cases: an old policy, a duplicate document, a procedure with exceptions, a poorly phrased question and a query whose correct answer is “I do not know”.

OpenAI’s evals guidance points in the same direction: define testing criteria, use well-defined data sources and run evals to check system behavior against test data rather than intuition alone. Its graders approach also reflects that idea by measuring outputs against specific criteria instead of relying on scattered impressions. (platform.openai.com)

Mini-case: when the problem was not AI, but evidence

Imagine a midsize company with internal support and a RAG built on operating procedures. At first it looks successful: the team asks common questions and gets fast answers. A few weeks later, the real pattern appears.

When the technical team updates a process but the old PDF is still indexed, the system blends both versions. When a question has an exception by customer or by country, it answers as if the rule were universal. And when it cannot find enough support, it fills the gap with a polished sentence.

In that scenario, the failure is not only the model. It is the mix of sources without hierarchy, the lack of visible validity dates, and the absence of a clear rule for when a human must review the answer. The lesson is uncomfortable but useful: a RAG does not clean up document chaos; it amplifies it.

The uncomfortable decision: less automation, more control

This is where the unpopular part begins. A good business RAG should not answer everything. In fact, the more important the decision, the more often it should abstain.

That feels less useful, but in real operations it is often more valuable for a system to say “needs review” than to provide a polished invented answer. In support, purchasing, HR or compliance, a wrong answer costs more time than it saves.

It helps to separate three levels:

• general reference questions, where the system can answer with some latitude;
• operational questions, where it must cite and verify;
• sensitive questions, where it should stop and escalate.

That design does not slow the project down; it prevents a first successful demo from becoming a trust problem later.

How to review it without building bureaucracy

Evaluation does not have to be heavy. It can start with a simple table: question, expected source, correct answer, confidence level, action if evidence is missing. What matters is repeating it with real cases, not invented ones.

Three signals matter more than a polished presentation: answers with clear citations, abstention when evidence is missing, and human reviews that actually improve the source base. If none of those show up, the system is still in demo mode, even if people are already using it.

OpenAI’s evals approach points in that direction too: define test data, measure specific criteria and review results systematically instead of trusting first impressions. (platform.openai.com)

What to ask this week

If you are evaluating a RAG, this week you can make progress with three practical questions:

• Which three wrong answers would actually damage trust?
• Which documents must outrank the rest, and how is that visible in practice?
• Who reviews doubtful answers, and what is the rule for correcting the knowledge base?

If a system is going to affect real decisions, the standard should not be that it looks ready. The standard should be that it knows when not to speak. At Codefuente, we usually start from that boundary: less promise, more testing, and more operational control.