Guide

What Is RAG? Retrieval-Augmented Generation Explained

How an AI model answers from your company's documents, and where the method stops working.

2026-11-105 min read

Retrieval-augmented generation (RAG) is a method that lets an AI language model answer questions from your own documents. Before the model writes anything, a search step finds the passages that match the question. The model then answers from those passages and names them as sources.

The model itself is not changed. It learns nothing permanently; it reads the relevant text again for every question. That is why RAG is the usual way to make a general model, such as GPT, Claude or an open-weight model, useful for the knowledge of one company.

How RAG works, step by step

1. Documents are split and indexed. Handbooks, policies, contracts, wiki pages and PDFs are cut into short sections, often a few paragraphs each. For every section, an embedding model calculates a list of numbers (a vector) that represents its meaning. Sections and vectors are stored in an index. Many systems also keep a traditional keyword index, because exact terms such as article numbers are found more reliably that way.

2. A question retrieves the matching passages. When someone asks a question, it is converted into a vector in the same way. The index returns the sections whose meaning is closest to the question, usually a handful. The wording does not have to match: a question about "working from abroad" also finds the policy titled "Remote work outside Germany".

3. The model answers from the passages and cites them. The question and the retrieved sections go to the language model together, with the instruction to answer only from this text. The answer refers to the sections it used, so the reader can open the source and check it.

If no section fits, a well-built system says so and does not guess.

What RAG is good for in a company

RAG suits every case where the answer is written down somewhere and people spend time looking for it.

  • Internal questions. Leave policies, travel expenses, IT instructions, onboarding. Employees ask in ordinary language and get the answer together with the page it comes from.
  • Support and service. The team finds the right passage in manuals and earlier solutions without knowing the exact search term.
  • Sales and project work. Product data, pricing rules and past quotes can be searched by content.
  • Technical documentation. Maintenance instructions, standards and specifications across many files.

Aven, our AI agent for company knowledge, works on this principle: it answers from the knowledge a team has stored and links every answer to its source.

Where RAG fails

Bad source documents. RAG repeats what the documents say. If two versions of a policy contradict each other, or the current rule exists only in someone's head, the answer is wrong or missing. Scanned PDFs without text recognition and tables saved as images are not found at all.

Missing permissions. If the index ignores who may read what, the system shows content to the wrong people, or it has to be restricted so far that it is no longer useful. Access rights have to be carried over from the source systems and applied in the search step, before the model sees any text.

Questions that need calculation. "How many contracts expire next year?" or "What was the average order value in March?" cannot be answered from five retrieved passages. These questions need a count over all the data, which is a database query and not a text search. A good system recognizes them and passes them to a suitable tool, or says that it cannot answer them.

RAG versus fine-tuning

The two are often confused. Fine-tuning trains the model further on examples and changes how it behaves. RAG leaves the model as it is and gives it text to read.

RAG Fine-tuning
What changes The text the model reads for each question The model itself
Suitable for Knowledge: facts, rules, documents Behavior: tone, format, specialist vocabulary
Updating Index the new document Train again
Sources Every answer can cite them None
Access rights Can be applied per person Not possible; the knowledge is inside the model

For company knowledge, RAG is the right place to start. Fine-tuning is worth it when the model should write or format differently. It is not a reliable way to add knowledge. Our guide to on-premise AI covers this decision in more detail.

What a company needs to start

  • One clear use case. One team and the questions it asks every day, not all company knowledge at once.
  • The documents for it. Current, in a readable format, and stored in a known place.
  • A person responsible for the content. Someone who decides which version applies and closes gaps.
  • A decision on where the data may be processed. In the cloud under a data processing agreement with EU hosting, or on your own servers when documents must not leave the company, as described on our local LLM page.
  • Test questions with known answers. A list of real questions from the team shows whether the search finds the right passages, and whether a change makes the system better or worse.

We build RAG systems for businesses. A project starts with a two- to four-week proof of concept on real data.

Frequently asked questions

What does RAG stand for?

Retrieval-augmented generation. "Retrieval" is the search for matching passages. "Generation" is the model writing the answer. "Augmented" means that the retrieved text is added to what the model receives.

What is a RAG system?

A RAG system has three parts: the index of the documents, the search in that index and the language model that writes the answer. In a company, it also includes the connection to the source systems, the access rights and a log of the sources used.

Is RAG the same as ChatGPT with a file upload?

The principle is similar. The difference is scale and control: a RAG system for a company indexes thousands of documents, keeps them current, applies access rights and records which sources were used for an answer.

Does a RAG system still make things up?

Much less often, because it answers from the supplied text and not from memory. It can still misread a passage or combine two sections wrongly. That is why answers carry sources, so you can check important details there.

Can RAG run without a cloud provider?

Yes. The embedding model, the index and the language model can all run on your own hardware, so neither documents nor questions leave the company network.

Have a project in mind?

Tell our AI project wizard what you're building and get an honest read on scope, approach, budget, and timeline in minutes — free, no sales call.

Start the project wizard