What is RAG for business: the short answer
RAG, short for retrieval-augmented generation, is how an AI tool answers from your own documents. When someone asks a question, the tool first searches your files for the passages that match. The model then writes an answer from those passages and links to each one. Nothing is learned permanently: the model reads the relevant pages again for every question.
This guide is for the owner or operations lead of a business of 10 to 50 people who has been offered a tool that "chats with your documents", or has just switched on Microsoft 365 Copilot. The demo answers perfectly. What it skips: when a RAG system gets something wrong, it has usually read the wrong document correctly. The fixes sit in your files and your permissions, not in the model.
We build company knowledge bases that a team reads and a model retrieves from. Below: how RAG works, what it suits, the three places it breaks, and a test you can run on any tool before you trust it.
How RAG works, step by step
A RAG system does two jobs at different times. Before anyone asks anything, it prepares your documents. When someone asks, it searches, filters and answers.
| Step | When it runs | What happens |
|---|---|---|
| Split | When a document is added or changed | Each document is cut into passages of a few paragraphs |
| Index | Straight after | Each passage is stored with its words and a list of numbers that captures its meaning, called an embedding |
| Search | On every question | The tool finds the handful of passages whose words or meaning best match the question |
| Filter | On every question | Passages the person asking isn't allowed to see are removed |
| Answer | On every question | The model gets the question and the passages, with instructions to answer only from them and cite each one |
The index is the part most owners never see. It is a copy of your documents, cut into pieces and stored somewhere else. Most of what goes wrong follows from that. A copy can fall behind the original, and it doesn't carry the original's permissions unless someone builds that in.
OpenAI's guide to model accuracy compares the choice to passing an exam. You can attend six months of classes and learn the material, or you can bring the textbook. RAG is the textbook. The guide recommends it when the model lacks knowledge: facts missing from its training, facts that are out of date, or private information. Fine-tuning, which retrains the model on examples, is for problems of format, tone or reasoning. Your returns policy is a textbook problem.
The name comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, where the documents searched were Wikipedia. Lewis later told NVIDIA the team would have chosen a better name had they known how far it would spread.
Say you run a 30-person wholesaler, and a salesperson asks whether a new customer can have 60-day payment terms. The search returns three passages: the credit policy updated in March, a 2023 email to a customer saved as a PDF, and a finance manager's note about exceptions. The model answers from whichever passages rank highest. If the 2023 email ranks first, the salesperson gets a three-year-old answer with a citation attached.
What RAG is good for, and what it isn't
RAG suits questions whose answer is written down somewhere, spread across more documents than anyone could paste into one chat. It handles live numbers, totals across records and unwritten judgement badly.
| Question | Does RAG fit? | Better route |
|---|---|---|
| "What does our warranty cover for a trade customer?" | Yes | The warranty policy, retrieved and cited |
| "How do we onboard a new client?" | Yes | One process document per task |
| "What did we agree with this supplier about late deliveries?" | Yes, if the contract is indexed | The contract, with the clause cited |
| "How many units are in stock?" | No | The stock system, asked at the moment of the question |
| "Which customers ordered less this year than last?" | No | A report or a database query |
| "Should we take on this client?" | No | A person, because the judgement isn't written down |
| "Can it answer in our house tone?" | Not by itself | Instructions and examples, or fine-tuning |
Totals catch people out. The search returns a handful of passages, so a question that needs every record gets an answer built from the few records that matched. It reads like a total, and it isn't one.
Live figures need a different design. The index is a snapshot, so the tool should ask the system that owns the figure at the moment of the question. Once the model starts calling your systems instead of reading documents, you are building something closer to an AI agent, with its own questions about limits. Our comparison of AI agents vs workflows covers those.
Out-of-date documents: the most common wrong answer
Many wrong answers from a RAG system are accurate readings of an out-of-date document. The model does what it was built to do: it answers from what the search gave it.
Models defer to what they are given, even when they know better. In the ClashEval study, Stanford researchers gave six leading models documents with deliberately wrong facts, across 1,294 questions on subjects such as drug dosages and sports records. The models adopted the wrong retrieved content over their own correct knowledge more than 60 percent of the time. The less confident a model was in its own answer, the more readily it took the document's word.
Professional tools fail the same way. A 2024 Stanford study tested the AI research tools from LexisNexis and Thomson Reuters, which use RAG over legal databases and had been marketed with claims of "hallucination-free" citations. They produced false information between 17 and 33 percent of the time. In one answer, a tool stated a legal test as current law although the US Supreme Court had overturned it in 2022. The old ruling matched the question well. Nothing in the search knew it had been overturned.
So when a tool gives a wrong answer, read the source it cites before you blame the model. Usually the model read correctly and the document was wrong, or an older one ranked higher. The fixes:
- Keep one current version of each document where the tool searches. Move old versions to an archive the index doesn't read.
- Give every document a status and a date the tool can read, and filter out anything marked out of date before the search ranks it.
- Update the index when a document changes, not on a monthly schedule. Ask any vendor how long a change takes to reach the answers.
- Show the date of each cited source in the answer, so the reader can spot a 2023 email.
- Give every document an owner and a review date. Our company second brain guide covers how to keep that going.
Permissions: the index doesn't know who is asking
A RAG system is only as private as its filter. Your files sit in folders with permissions, and the index copies passages out of those folders. Unless the permissions travel with every passage and the search checks them on every question, anyone who can ask the tool can read what it indexed.
Assistants such as Microsoft 365 Copilot answer from the data each user already has permission to access. That moves the risk rather than removing it. Microsoft's own documentation gives the example of a budgeting site whose owner never set proper permissions. A marketing specialist asks Copilot a budget question and gets an answer from it. He would never have found the site by browsing; asking in plain words found it for him.
Microsoft's stopgap, Restricted SharePoint Search, limits Copilot to an allow list of up to 100 sites. Microsoft says it "isn't a security boundary", and from 31 July 2026 it can't be switched on for new customers. The lasting fix is the unglamorous one: check who can open what before you connect anything.
In a system built for you, the filter has to run inside the search, before any passage reaches the model. An instruction such as "don't reveal salary data" is not a lock, because by then the passage is already in front of the model. OWASP, a non-profit that ranks security risks in AI applications, lists unauthorised access and leaks between users among the main weaknesses of RAG systems, and recommends permission-aware stores. We treat "who may see this" as a property of each passage, checked where the data leaves the store.
Permissions also change after the index is built. Someone moves team, a folder gets locked down, a person leaves. Each change has to reach the index, and so does every deletion. The ICO's guidance on the right to erasure says erasure must reach backup systems as well as live ones. Treat the index as one more copy that a deletion has to reach.
Missed searches: when nothing matches, it answers anyway
The third failure is quieter. The search misses the passage that holds the answer, or no passage holds it, and the model writes a fluent answer anyway.
Researchers at Deakin University studied three RAG systems built for research, education and biomedical work, and listed seven points where such systems fail. The first two: the answer isn't in the documents at all, or it is there but ranks too low to be returned. In the first case, the paper notes, the system should say it doesn't know, but it can be fooled into giving a response.
A fluent answer hides a failed search. The reply reads the same whether the tool found the policy or stitched something together from a passage that was nearly relevant. That is why every answer should cite its sources, and why a tool that finds no source should say so and name who to ask.
Search by meaning also has a blind spot for exact strings. Microsoft's Azure AI Search documentation says product codes, specialised jargon, dates and people's names are found better by keyword search, because it matches exactly. A question about part AX-2210 can return a passage about AX-2201, since the two look almost the same to a search by meaning. Good systems run both kinds of search and merge the results, which is called hybrid search.
Two more causes sit in the documents themselves. A passage cut from a long file can lose the context that made it make sense, which is why short notes holding one idea each retrieve better. And a file someone else wrote can hide instructions aimed at the model. Our guide to AI document processing covers checking files before they go anywhere near an index.
How to test a RAG tool before you trust it
Test any RAG tool, bought or built, with your own questions, and score the sources as well as the answers. A demo uses questions the vendor chose.
- Collect 30 to 50 real questions from the last month, in the words your staff used. For each, write down the right answer and the document that holds it.
- Add trap questions from the table below.
- Score every answer three ways: is it right, does it cite the right document, and did it decline when it should have?
- Run the whole set again after every change to the documents, the settings or the model.
| Trap question | What a good tool does |
|---|---|
| One whose answer changed this year | Gives the new answer and cites the new document |
| One the asker isn't allowed to answer, asked from a junior account | Declines, without hinting at the content |
| One your documents don't answer | Says it doesn't know, and names who to ask |
| One built around a product code, a name or a date | Returns that exact item, not a near neighbour |
The Deakin researchers concluded that a RAG system can only really be validated once people use it, and that its robustness evolves rather than being designed in. So keep a log of every question the tool couldn't answer or got wrong. Each entry is a missing document, a stale one or a search to tune. We write the question set before we build anything, because it is the one definition of "working" that both sides can check.
Buy or build: you may already own a RAG tool
Many firms already pay for a tool that does RAG. Try it first, and build only for what it can't do.
| Option | Examples | Suits | Watch for |
|---|---|---|---|
| A chat project with uploaded files | ChatGPT projects, Claude projects | One team's reference set | Whether a change to the original reaches the tool, and how fast |
| The assistant in your office suite | Microsoft 365 Copilot | Firms whose documents already live in that suite | It follows your existing sharing, including every old mistake |
| A system built for you | Your documents and systems, your own cloud account | Answers inside your own software or a customer channel; sources with different permissions; exact codes; a record of which passages each answer used | Someone has to own it and keep running the tests |
Claude's projects show how blurred the line has become. They switch to retrieval automatically once the uploaded files approach the model's limit, so many owners already use RAG without knowing it.
When you don't need RAG
You don't need RAG, and you may not need us, in these cases:
- Your documents fit in one conversation. Attach them and ask; see the FAQ below on long inputs.
- The answers you need are live numbers in a system you already have. Connect to that system, or build a report.
- The documents don't exist yet, or three of them disagree. Write and settle them first; our guide to writing process documents your team will actually use is the place to start.
- Two people ask a few questions a month. A well-named folder and the search box you already have will do.
- You want the model to sound like you. That takes instructions and examples, not retrieval.
Where to start this week
Pick the 20 questions your team asked most this month. For each one, find the document that answers it and note three things: its date, whether another version exists, and who can open it.
If most answers sit in one current document that the right people can open, a tool you already own will probably do well. Switch it on and run the trap questions. If the answers live in people's heads, old emails and folders shared with everyone, your project is the documents, and that is where the first month should go.
Questions
Do we need a vector database?
Not for an off-the-shelf tool, which keeps its own index. For a custom build, an extension such as pgvector adds search by meaning to a Postgres database, next to its ordinary keyword search. The index can then sit in a database you already run, in your own account. A separate vector database is worth considering once the volume of documents, or the speed you need, outgrows that.
Models can now read whole books at once. Do we still need RAG?
For a small set of documents, often not: attach them and ask. For a large or changing set, retrieval still helps in three ways. Each question is faster and cheaper to answer when the model reads a handful of passages instead of every file. Permissions can be checked passage by passage. And models don't use long inputs evenly: a 2023 study found they did best when the relevant text sat at the start or end of a long input, and worse when it sat in the middle (Liu et al.). Newer models have improved, so test with your own documents.
Does our data leave the company?
With a hosted tool, your documents and the index sit with the provider, who processes them on your behalf. Under UK and EU GDPR that needs a processor contract with the terms in Article 28. Find out where the index is stored, how long it's kept, and whether your plan allows the provider to train on it. A custom build can keep the index in your own cloud account.
How we wrote this: drafted with AI, then checked and edited by the people who build our projects. Figures come from our own work or from the linked sources.