RAG vs fine-tuning: which is right for your AI feature?

When an off-the-shelf language model isn't enough, teams usually reach for one of two techniques: retrieval-augmented generation (RAG) or fine-tuning. They're often mentioned together and frequently confused, but they fix different problems.

A simple way to remember the difference:

RAG changes what the model knows at the moment it answers. Fine-tuning changes how the model behaves.

This guide explains how each works, when to choose which, what they cost, and how to combine them.

#The problem both are trying to solve

A general-purpose model knows a lot about the world, but nothing about your business: your products, policies, customers, documents or house style. It also has a training cut-off, so it doesn't know what changed last week.

There are three ways to close that gap:

1. Prompting: describe what you want, with examples, in the request.
2. Retrieval (RAG): look up relevant information and include it in the request.
3. Fine-tuning: train the model further on examples of what you want.

Most teams should try them in that order.

#What RAG does

RAG fetches relevant passages from your own data (documentation, tickets, policies, product data) and gives them to the model alongside the question. The model answers using that context, and can cite where each part of the answer came from.

#How a RAG pipeline works

1. Ingest: collect documents from their sources (help centre, drive, database, CMS).
2. Chunk: split them into passages small enough to be specific but large enough to make sense.
3. Embed: convert each passage into a vector that captures its meaning.
4. Store: save vectors and passages in a vector index, which can be a dedicated vector database or an extension like pgvector in PostgreSQL.
5. Retrieve: for each question, find the most relevant passages, often combining semantic search with keyword search.
6. Re-rank (optional): reorder candidates with a more precise model.
7. Generate: send the question plus the selected passages to the model, with instructions to answer only from them and cite sources.

#Choose RAG when

• knowledge changes often and must stay current
• answers need citations users can check
• data is spread across many documents
• different users may see different data, and permissions must be respected
• you need to remove information quickly (delete it from the index and it's gone)

#Where RAG goes wrong

In our experience, poor RAG answers are rarely the model's fault. The usual causes are:

• Bad chunking: passages that cut tables in half or separate a heading from its content.
• Weak retrieval: semantic search alone missing exact terms like product codes or names.
• Stale or duplicate content: three versions of the same policy, all slightly different.
• Too much context: dozens of loosely relevant passages drowning out the right one.
• Missing permissions: retrieving documents the user shouldn't see.

Improving retrieval is usually the highest-leverage work in a RAG project.

#What fine-tuning does

Fine-tuning continues training a model on examples of the inputs and outputs you want. It changes how the model responds (tone, structure, format, decision patterns) rather than giving it fresh knowledge.

#How fine-tuning works

1. Collect examples: pairs of input and ideal output, ideally from real usage and reviewed by experts.
2. Clean and balance: remove errors, contradictions and duplicates; cover edge cases.
3. Hold out an evaluation set: examples the model never trains on, used to measure results.
4. Train: run fine-tuning through a provider's API or on open-weight models in your own infrastructure.
5. Evaluate and compare against the base model with good prompting.
6. Deploy and monitor, and retrain as requirements change.

#Choose fine-tuning when

• you need a consistent tone, structure or output format
• the task is narrow and repeated at high volume
• prompts are getting very long just to explain the format
• you want a smaller, faster or cheaper model to match a larger one on a specific task
• you have enough high-quality examples to teach the behaviour

#Where fine-tuning goes wrong

• Trying to teach facts. Fine-tuning is a poor way to keep knowledge current; it can memorise outdated or incorrect details.
• Low-quality data. The model faithfully learns your inconsistencies.
• No baseline. Teams fine-tune before checking whether better prompting would have been enough.
• Forgetting maintenance. Every model or requirement change can mean retraining.

#Side by side

#Decision guide

Ask these questions in order:

1. Does the answer depend on information the model can't know (your documents, recent changes, customer data)? → You need RAG.
2. Is the problem mostly about format, tone or a narrow repeated task? → Try better prompting first; if that hits a limit, fine-tune.
3. Are costs or latency too high at scale? → Consider fine-tuning a smaller model.
4. Do you need both current knowledge and a consistent style? → Combine them.

#Combining both

Many production systems use both techniques:

• RAG supplies the facts: relevant, current, permission-checked passages.
• A fine-tuned model shapes the answer: consistent format, tone and decisions.

For example, a support assistant might retrieve the latest policy articles and order details, then use a model fine-tuned on thousands of approved replies to write in the company's voice and structure.

#Measuring quality

Whichever approach you choose, you need evaluation before launch and on every change:

• Build an evaluation set of real questions with expected answers or acceptance criteria.
• For RAG, measure retrieval separately: did the right passages come back? Then measure answer quality.
• Track groundedness: is each claim supported by the retrieved sources?
• Track refusals: does the system correctly say "I don't know" when the answer isn't in the data?
• Watch cost and latency per request, not just accuracy.

Without measurement, it's impossible to know whether a change made things better or just different.

#Cost and latency

• RAG adds retrieval time and longer prompts, but caching, smaller embedding models and tight retrieval keep it fast.
• Fine-tuning adds training cost and maintenance, but can cut per-request cost by shortening prompts or allowing a smaller model.
• Streaming responses improves perceived speed for both.

#Privacy and security

• Respect document permissions at retrieval time, so users only get answers from data they're allowed to see.
• Use providers and settings that exclude your data from training public models, or run open-weight models in your own cloud.
• Avoid putting sensitive personal data into fine-tuning sets unless you have a clear legal basis and need.
• Log and review outputs, with personal data handled carefully.

#Our recommendation

Start with RAG and good prompts, measure quality on real examples, and only fine-tune when you hit a clear limit in style, format or cost. Invest early in retrieval quality and evaluation. They pay off whichever way the project goes.

Read more in how to add AI to your product, see our AI development services, or tell us about your use case.