An assistant is only as useful as its sources

RAG in plain words

Retrieval-augmented generation, or RAG, is a way to make an AI assistant answer from your documents instead of from its general training. When someone asks a question, the system first searches your document set for the passages most likely to hold the answer. It then gives those passages to a language model, with an instruction to answer from them and to say so when they do not contain the answer.

So there are two steps, retrieval and generation, and each can fail. If retrieval brings back the wrong passages, the model writes a fluent answer from the wrong material. If the right passage does not exist, or exists in three conflicting versions, no model can repair that. Most disappointing assistants have a source problem, and most of the fix is document work.

Why source quality decides the result

An assistant repeats what it finds. An outdated policy, a draft that was never approved and the current version all look equally valid to the search step unless something marks them apart. The assistant cannot know that last year's procedure was replaced.

Document hygiene is therefore where most of the gain is. Scanned pages with no text layer cannot be searched until they are converted. Tables lose meaning when they are flattened into lines of text. Long documents without headings split into passages that carry no context. Duplicates crowd out better results.

• One current version of each document
• Clear titles, dates and owners
• Headings that describe the section beneath them
• Searchable text, including scans and tables
• Superseded files archived out of reach

Choose and prepare the source set

Start narrow. Pick one group of users and the questions they ask most, then select only the documents that answer those questions. A small, clean set that its owners vouch for beats everything on the shared drive.

For each source, name an owner who confirms that it is current and agrees to keep it so. Record metadata the system can use: document type, date, version, department, region. Split documents along their own structure, by section and heading, so that each passage makes sense when read alone.

Then set up the update path. When a document changes at the source, the assistant's copy must change too, and deleted documents must disappear from answers.

• A defined user group and question list
• An owner for every source
• Useful metadata on every document
• Passages split by section
• Automatic updates and removals

Respect access permissions

An assistant that searches everything will show people things they were never meant to see: salaries, contracts, board papers, another customer's file. Retrieval must apply the same permissions as the source system, for each user, at the time of the question. Filtering after the answer is written is too late, because the restricted content has already shaped it.

Check this before you add any sensitive source. Many organisations find during this step that their shared folders are open far wider than anyone intended, and the assistant simply makes that visible. Fix the permissions at the source, and handle personal data in line with your privacy and compliance requirements.

Evaluate with real questions

Collect real questions from the people who will use the assistant: support tickets, emails to the help desk, the things new joiners ask. Fifty to a hundred is enough to start. For each one, have a subject expert write the correct answer and note which document contains it.

Then test in two parts. For retrieval, check that the right passage came back near the top. For the answer, check that it is correct, complete and supported by that passage. Include questions the documents cannot answer, and confirm that the assistant says it does not know. Rerun the set whenever you change the documents, the search settings or the model.

• Real questions from real users
• Reference answers written by experts
• Retrieval checked separately from the answer
• Unanswerable questions included
• A rerun after every change

Show sources to users

Every answer should show where it came from: the document title, the section, and a way to open the original at that passage. A user can then verify a claim in seconds. It also teaches people to treat the assistant as a fast route to the source, which is the right level of trust.

When the assistant finds nothing relevant, it should say so and point to a person or team. Add a simple way to flag a wrong answer. Those flags usually lead back to a document that is missing, outdated or contradictory, which makes them the best input you have for improving the source set.

When a company brain is worth building

A company brain is a multi-purpose assistant trained on a company's own data. We build these, along with voice AI and NLP systems on custom RAG pipelines. Before you commission one, check the conditions.

It is worth building when many people ask repeated questions, the answers exist in documents, those documents have owners, and time is being lost finding them. It is a poor fit when the knowledge lives mostly in people's heads, when documents contradict each other and nobody will arbitrate, when few questions are asked, or when a wrong answer carries a high cost and no expert reviews it. In those cases, fix the documents or the process first.

• Many repeated questions
• Answers that exist in writing
• Documents with owners
• A sensible review step for high-stakes answers

What to do next

Pick one team and collect fifty questions they ask often. Find the documents that answer them, and check that each is current, owned and searchable. That exercise shows how ready your sources are before you spend on software. Our free AI readiness report is a useful wider starting point.