RAG or fine-tuning? Start by asking where the knowledge lives

Sooner or later every AI project meets the same question: the model is clever, but it does not know anything about our business. How do we fix that? There are two broad answers, and they are often presented as rivals. They are closer to different tools for different jobs.

The two approaches in one paragraph each

Retrieval, usually called RAG, leaves the model as it is. When someone asks a question, the system first searches your documents for relevant passages, then hands those passages to the model and asks it to answer from them. The knowledge stays in your documents.

Fine-tuning changes the model itself by training it further on your examples. The knowledge, or more often the style and behaviour, ends up inside the model’s weights.

Ask where the knowledge lives

If what you need is in documents that change, such as contracts, policies, manuals, tickets and price lists, use retrieval. You can update a document and the system knows the new version a minute later. You can show the source of every answer. You can respect permissions, because the search step only returns what the user is allowed to see. None of that is practical with fine-tuning.

If what you need is a consistent way of doing a narrow task, such as classifying claims into your forty categories or extracting fields from your suppliers’ invoices, and you have thousands of examples, fine-tuning a small model can be faster, cheaper per item and easier to host privately.

What usually decides it

• Do you need to cite sources? Then retrieval.
• Does the information change weekly? Retrieval.
• Do different users have different access rights? Retrieval.
• Is it one narrow task at very high volume? Consider fine-tuning.
• Must it run on your own hardware with a small model? Fine-tuning often helps.

In the knowledge projects we have delivered, about nine in ten are retrieval systems. Where we fine-tune, it is usually a small model doing one step inside a larger pipeline.

The part that matters more than either

Whichever you choose, the result depends on evaluation. Have your own experts write a few hundred real questions with the answers they would accept, and score the system against them after every change. Most disappointing knowledge systems we are asked to rescue were never measured. Once they are, the fixes tend to be mundane: better document parsing, sensible chunking, removing the 2017 version of the policy that nobody deleted.