AI Document Search for Your Team: How RAG Works
How AI document search (RAG) works, step by step: accuracy, privacy, costs and when you don't need it, with a real example from our work.
Qazi Ehsan
Xenova Solutions

An AI document search system (usually built with RAG, retrieval-augmented generation) lets your team ask questions in plain English and get answers drawn from your own files, with a link back to the source. It works by finding the most relevant passages in your documents first, then asking a language model to answer using only those passages. For most small businesses, a focused proof of concept on one set of documents is the sensible first step before rolling it out company-wide.
Last updated: November 2026
We build these systems for UK, US and European businesses, and the same questions come up again and again from owners and operations managers: will it make things up, can it keep our data private, do we even need it, and what does it cost? This guide answers those questions in plain English.
How a RAG system works, step by step
The term comes from a 2020 research paper by Lewis and colleagues, which described combining a language model with a searchable index of documents. In a business setting, the pipeline looks like this:
- Collect the documents. PDFs, Word files, policies, SOPs, contracts, help-desk articles, wiki pages. Scanned documents need OCR first.
- Split them into chunks. Each document is cut into passages, usually a few hundred tokens (a couple of hundred words) or fewer, so the system can retrieve just the relevant part rather than a whole 60-page manual.
- Turn chunks into embeddings. Each chunk is converted into a list of numbers that captures its meaning, and stored in a vector database (for example, PostgreSQL with the pgvector extension).
- Search when someone asks a question. The question is converted the same way, and the system finds the closest-matching chunks. Good systems combine this “meaning” search with ordinary keyword search, so exact terms like product codes or clause numbers are not missed.
- Generate a grounded answer. The top passages are handed to a language model with instructions to answer only from them, cite where each point came from, and say “I don’t know” when the documents don’t cover it.
- Show the sources. The user sees the answer plus links to the exact documents and sections, so they can check it.
Retrieval quality is where most of the engineering effort goes. Anthropic’s published tests on contextual retrieval found that adding context to each chunk, combining embeddings with keyword (BM25) search and then reranking the results cut the rate of failed retrievals (in the top 20 results) by 67% in their benchmarks. Your numbers will differ, but the lesson holds: how you chunk and search matters as much as which AI model you pick.
Will it make things up?
This is the most common worry we hear, and it is fair. A plain chatbot fills gaps from its general training, which is where invented answers come from. A well-built RAG system reduces this a lot, but it does not remove it entirely. These are the controls we put in place:
- Answer-only-from-sources instructions, with a clear “I couldn’t find this in our documents” fallback and, where it matters, a hand-off to a named person.
- Citations on every answer, so staff can click through and check the original wording.
- A test set of real questions (typically 50–100) with known correct answers, run before launch and after every change.
- Feedback buttons so wrong or unhelpful answers are logged and fixed, usually by improving the source document rather than the AI.
One point people often miss: the system is only as good as your documents. If two policies contradict each other, or the current process lives only in someone’s head, the AI will reflect that. Tidying up outdated and duplicate files is often the most valuable part of the project.
Do you actually need RAG?
Not always. If your entire knowledge base is small, you may not need a retrieval system at all. Anthropic notes that a knowledge base under about 200,000 tokens (roughly 500 pages) can simply be included in the prompt in full. RAG earns its place when you have hundreds or thousands of documents, files that change often, different access levels for different staff, or a need for precise citations. If you are weighing RAG against training a custom model, our guide to RAG vs fine-tuning covers that decision.
| Option | Best when | Main limits | Typical cost |
|---|---|---|---|
| Paste documents into a general AI assistant | A handful of short documents, occasional one-off questions | No access control, no shared source of truth, staff may upload data you don’t want shared | Per-user subscription |
| Off-the-shelf “chat with your docs” tool | Standard file types, simple needs, quick start | Limited control over chunking, search and data location; harder to connect to your own systems | Monthly subscription, often per user or per document volume |
| Whole knowledge base in the prompt (long context) | Under roughly 500 pages that rarely change | Cost and speed grow with every question; no fine-grained permissions | Pay per use of the AI model |
| Custom RAG system | Hundreds or thousands of documents, permissions, citations, integration with your tools | Needs set-up, testing and ongoing upkeep | Xenova RAG Proof of Concept from £4,000 (about 3 weeks), then hosting and model usage |
Can it keep our documents private?
Yes, but it has to be designed in from the start. The choices that matter most are:
- Where the data lives. The document store and vector database can be hosted in the UK or EU, or on your own servers.
- Which model answers. You can use a commercial model API under business terms, or an open model (such as Llama or Qwen) hosted privately, so documents never leave infrastructure you control.
- Who can see what. Permissions should be checked at retrieval time, so a junior employee asking about salaries cannot be shown the HR folder.
- What gets indexed. Leave out personal data the system doesn’t need.
If your documents contain personal data, UK GDPR still applies. The ICO’s guidance on security and data minimisation in AI is clear that you should only process the personal data you need for your purpose, and that using AI changes your security risk profile. This is not legal advice, so check your own obligations with your data protection lead.
Can it answer questions from our database too?
RAG is designed for text: policies, manuals, contracts and emails. Questions like “how many orders shipped late last month?” live in a database, and answering them reliably is a different technique, where the AI writes a database query instead of searching documents. The two can sit behind the same chat window, but we treat the database side with extra caution: read-only access, a fixed list of allowed tables and a test set of known questions and correct results before anyone relies on it. If your use case needs the AI to take actions as well, such as updating records or sending emails, that moves into AI agent territory.
Our experience
Our experience: we built an AI document Q&A system for a client in Finland that lets their users ask questions about their own documents and get answers grounded in them. Across our RAG work, the pattern is consistent: the first version is rarely held back by the AI model; it is held back by retrieval and by the state of the documents. That is why we start every project with a set of real questions from the people who will use it. You can see more of our projects on our work page.
How to get started
- Pick one painful use case. For example, staff repeatedly asking the operations manager the same process questions, or support agents searching five systems for one answer.
- Gather the documents for that use case only. Remove outdated versions and duplicates.
- Write down 30–50 real questions people ask today, with the correct answers.
- Build a proof of concept and measure it against those questions before widening the scope.
- Decide on rollout based on results: more documents, more teams, integrations with Slack, Teams or your intranet.
Ongoing costs depend mainly on the number of questions, the size of your document set and the hosting choice. Our guide to how much an AI chatbot costs breaks down build and running costs in more detail.
Frequently asked questions
Can I put our SOPs into an AI assistant before we hire staff?
Yes, and it is a good moment to do it, because writing the SOPs down clearly is half the work. Keep each procedure short and step by step, make sure the system cites the SOP it used, and have it say so when a question isn’t covered rather than guessing.
Why do “chat with your documents” tools sometimes give wrong answers?
Usually because the right passage was never retrieved, the documents contradict each other, or the model was allowed to fill gaps from general knowledge. Better chunking, combined keyword and meaning search, strict answer-from-sources rules and a test set of real questions fix most of these issues.
Can a RAG system run on private or self-hosted models?
Yes. Open models such as Llama or Qwen can be hosted on your own or UK/EU cloud servers, and the vector database can run there too. The trade-off is higher set-up and hosting effort compared with a commercial API, so it is worth it mainly when data sensitivity requires it.
Want to see it working on your own documents?
Our RAG Proof of Concept starts from £4,000 and takes about three weeks, using a real slice of your documents and your team’s real questions. See what is included on our packages page, or book a free 30-minute scoping call and we will tell you honestly whether RAG is the right fit.