Many companies start a RAG project with the wrong question: “Which model should we use?” or “Which tool should we build it in?” The better question is much less glamorous: what internal knowledge deserves to be retrievable, and at what level of risk?
If company knowledge is not well governed, RAG does not fix it; it just gives it a more convincing interface.
A useful RAG does not turn documentation into magic intelligence. It reduces the time between an operational question and an answer that can be used safely. That requires more than indexing documents. You have to decide what goes in, what stays out and what happens when the system is unsure.
The real situation: fast answers, slow decisions
Picture a mid-sized company with sales, operations and support spread across ERP, SharePoint, email and tickets. Each team stores information in its own format. When someone asks “How do we handle a late return?” or “What happens if a premium customer opens a critical ticket?”, the answer exists, but it is fragmented.
The first instinct is usually to ingest everything and launch a chatbot. The common result is worse than expected: the system answers with false confidence, mixes old and current versions, and is only useful in part of the cases. The problem is not the model. The problem is that the knowledge has no hierarchy.
A well-designed RAG should help in specific scenarios, for example:
- repetitive internal support questions,
- operational rules that change often,
- quick access to current policies and procedures,
- searching evidence inside tickets and technical documentation.
If the use case does not have repeat volume or a clear cost of error, the project usually becomes an expensive demo.
Which sources should be included, and which should not
Here is the first serious decision: not every source should be indexed. Most RAG failures come from trying to include everything.
Sources that usually work well:
- version-controlled internal manuals,
- approved procedures,
- support knowledge bases,
- controlled technical documentation,
- business FAQs validated by owners.
Sources that need caution:
- email threads,
- informal chats,
- personal notes,
- ownerless documents,
- outdated files without version control.
That does not mean they are useless. It means their value is contextual, not normative. A common mistake is letting the system cite an email as if it were policy. That creates an illusion of precision that later gets paid for in operations.
A practical criterion: if a source can change a customer decision, compliance outcome or financial impact, it needs an owner, an effective date and explicit review. If it does not, it should not answer on its own.
Mini-case: when the assistant knew too much
A services company with several business units wanted to reduce the time the operations team spent searching internal documentation. Manuals, historical tickets and project notes were loaded into a pilot RAG.
At first, the system looked useful. But when real cases were reviewed, three problems appeared:
- it returned procedures that had already been retired,
- it mixed criteria from two different teams,
- it kept answering even when the documentation was ambiguous.
The fix was not “better prompting.” The project had to be reorganized: the corpus was limited to current documents, sources were separated by domain, confidence labels were added, and the system was forced to answer “not found” when evidence was insufficient.
The real value came later: less time wasted searching, fewer unnecessary escalations and fewer invented answers. Not because the AI became smarter, but because it was allowed to be less reckless.
The uncomfortable mistake: wanting it to answer everything
Here is the part many teams do not want to hear: a good RAG should refuse to answer often.
That feels like a bad user experience. In reality, it is a sign of control. If the system answers everything, it is usually extrapolating. And in operations, extrapolation is just another form of invention.
The most useful decision is often to define three behaviors:
- answer with citation and source when evidence is strong,
- ask for clarification when context is missing,
- escalate to a person when risk is high or the answer is not unambiguous.
It is also worth accepting that not every problem needs generation. Sometimes a better filtered search is enough. RAG should not become a substitute for document governance.
Signs the project is on the right track
Before scaling, some signals matter more than the demo:
- answers are based on current, visible sources,
- the system shows when evidence is insufficient,
- users stop asking humans about repetitive topics,
- content owners review and correct the corpus,
- wrong answers can be traced back to a specific source.
If you cannot audit an answer, you do not have a knowledge assistant; you have a black box with a friendly interface.
Conclusion
An internal-knowledge RAG is won not by using the newest model, but by defining the scope, document quality and failure behavior properly. The difference between an interesting pilot and an operational tool lies in content discipline, not AI shine. At Codefuente, we usually start with that decision map before building anything, because in internal knowledge, order matters more than ambition.