For most businesses, the answer to RAG vs fine-tuning is RAG first. If your problem is that the model doesn’t know your facts (your policies, product specs, contracts, tickets), retrieval augmented generation hands it those facts at query time, with sources you can check and update tomorrow. Fine-tuning changes how a model behaves (format, tone, a narrow classification task, or a cheaper model that matches a bigger one), not what it knows to be true. A surprising number of use cases need neither.
That’s the short version. The rest of this piece covers what each technique actually changes, how to tell which problem you have, where each one fails in production, and the order we would build things in.
What each technique actually changes
Retrieval augmented generation
RAG leaves the model untouched. When a question comes in, your system searches a store of your own content, picks the most relevant passages and puts them into the prompt alongside the question. The model answers from what it has been handed. The idea was formalised in a 2020 paper by Lewis et al., but the production version is mostly plumbing. You ingest documents, split them into chunks, embed them and run a search, which is usually hybrid: vector similarity plus keyword matching. You can optionally re-rank the results, and then you assemble the prompt.
A typical RAG architecture therefore has more moving parts in the data pipeline than in the model call, and that is its strength. If a policy changes on Tuesday, you re-index the document and the assistant is correct by Tuesday afternoon. You can show the user which passage an answer came from. You can also filter retrieval by the user’s permissions, so the finance team’s documents never reach a warehouse operative’s prompt.
Fine-tuning
Fine-tuning changes the model itself. You supply hundreds to thousands of input–output examples, and training nudges the model’s weights so it reproduces that pattern. Hosted providers offer this for some of their models, and OpenAI’s fine-tuning guide is a reasonable reference for how the workflow looks. Open-weight models can be fine-tuned on your own hardware, typically with parameter-efficient methods such as LoRA, which train a small adapter instead of the whole network.
Fine-tuning is good at shaping behaviour. It can teach a model to always return a given JSON schema, to write in a particular register, to map messy inputs to your 40 internal categories, or to follow the house style for a specific document type. It is also the main route to distillation: you take a task a large, expensive model does well and train a small, fast model to do the same narrow job. What it does badly is store facts. A model fine-tuned on your product catalogue will absorb some of it, unevenly. It can’t tell you which document a claim came from, and there’s no clean way to update it when prices change next month.
The deciding question: knowledge problem or behaviour problem?
Before you compare vendors or frameworks, write down the failure you’re trying to fix. Suppose the model gives wrong or vague answers because it has never seen your information. That’s a knowledge problem, and retrieval is the fix. Now suppose the model has the information, because you’ve put it in the prompt, but it formats the output inconsistently, misses the house tone, or gets a narrow judgement call wrong in a repeatable way. That’s a behaviour problem. Better prompting comes first, and fine-tuning comes in only if prompting plateaus.
Concrete cases make the line clearer. Take an internal assistant that answers staff questions about HR policy, IT procedures and the employee handbook. That is a knowledge problem, so the answer is RAG with access control.
Now take a system that reads inbound supplier emails and routes them into one of fifteen queues using your own, slightly idiosyncratic categories. That is a behaviour problem. Start with a well-structured prompt and a handful of examples. Consider fine-tuning a small model once you have a few thousand consistently labelled emails and the volume makes per-call cost matter.
A tool that drafts quotes from a price list is a knowledge problem with a twist. The prices belong in a database, queried through a tool call. They don’t belong in a vector index, and certainly not in model weights.
That last case is worth dwelling on. Structured data such as stock levels, order status, prices and account balances should be fetched with a proper query, not retrieved as text chunks by similarity search. Plenty of disappointing RAG projects are really database lookups forced through an embedding model. Our LLM integration guide for production systems covers the tool-calling pattern in more detail. If the data lives in an older platform, integrating LLMs into legacy enterprise systems covers the practical routes in.
RAG vs fine-tuning side by side
| Dimension | RAG | Fine-tuning |
|---|---|---|
| What it changes | The prompt, at query time | The model’s weights |
| Best for | Answering from your documents; facts that change | Consistent format and tone; narrow classification; distilling to a smaller model |
| Data you need | Your existing documents, cleaned and deduplicated | Hundreds to thousands of consistent, labelled examples |
| Keeping it current | Re-index the changed documents | Retrain, re-evaluate, redeploy |
| Traceability | Can cite the source passage | None; answers come from the weights |
| Access control | Filter retrieval per user | Anything in training data is potentially reachable by every user |
| Where the cost sits | Ingestion pipeline, search infrastructure, longer prompts per call | Building the dataset; retraining when the base model changes |
| Typical failure | Retrieves the wrong chunk and answers confidently from it | States outdated or blended facts; overfits to quirks in the examples |
| Time to a first useful version | Short for a well-scoped corpus | Longer, dominated by dataset preparation |
Where RAG goes wrong in production
Retrieval is the whole game
When a RAG system gives a bad answer, the model is usually doing exactly what it was told: answering from the passages it was given. It’s the passages that were wrong. A chunk split a table in half, or separated a rule from the exception two paragraphs later. The search matched on vocabulary instead of meaning. Or the right document was a scanned PDF nobody ran OCR on.
The fix is almost never a bigger model. It’s better ingestion, chunking that respects document structure, hybrid search and a re-ranker. Add metadata filters (document type, date, department) so the search space is already small before similarity is computed.
Stale and contradictory sources
RAG is only as current as its index, and only as consistent as the content you feed it. If three versions of the expenses policy live in SharePoint, retrieval will happily surface the 2021 one. Ingestion needs an owner, a schedule and a rule for which version wins. We treat documentation freshness as infrastructure in our own lab: our home lab documentation keeps itself current with Gitea, GitHub and a nightly snapshot. The same principle applies to any corpus an assistant answers from.
Permissions and leakage
A retrieval layer that ignores who is asking is a data breach waiting for a curious employee. Permission filtering has to happen at retrieval time, using the same identities as the source systems, and it has to be tested with deliberately awkward questions. Our guide to data security and privacy in AI projects goes through the controls worth having from day one.
Where fine-tuning goes wrong
Fine-tuning projects tend to fail before training starts. The dataset is the product. If your examples are inconsistent (two people labelling the same email differently), the model will learn the inconsistency faithfully. Expect most of the effort to go into collecting, cleaning and agreeing on examples. On hosted platforms the training run itself is often the cheapest step.
The second failure is maintenance. A fine-tuned model is tied to its base model. When the provider retires that base or ships a better one, you retrain, re-evaluate and redeploy. With RAG, swapping the underlying model is closer to a configuration change plus an evaluation run, which matters when the model market moves every few months.
The third failure is regulatory, and it matters for Irish, UK and EU businesses. Personal data used in training ends up diffused through the weights. Under GDPR, honouring a right-to-erasure request against a fine-tuned model is awkward at best, and in practice it means retraining without that record. With RAG, you delete the document and re-index.
If your training data includes customer records, settle the GDPR question before you build, not after. If the system makes or supports decisions about people, check where it sits under the AI Act as well. Our EU AI Act compliance checklist for SMEs is a sensible starting point.
When you need neither, and when you need both
Context windows on current frontier models are large enough to hold a small, stable corpus in full. A 40-page product manual, a single contract or a set of standard operating procedures can simply go into the prompt, often with prompt caching to keep repeated calls affordable. There’s no index, no chunking and no retrieval errors. Retrieval earns its complexity once the corpus is too large for that, changes often, or needs per-user permissions.
Using both together makes sense when you have a genuine knowledge problem and a genuine behaviour problem at volume. One example is a claims-handling assistant that must answer from policy documents (RAG) and produce output in a rigid regulatory format that prompting alone doesn’t hold reliably (fine-tuning). Another is a high-volume pipeline where a fine-tuned small model with retrieval replaces a large general model to cut cost and latency. Either way, that is a second-phase optimisation and rarely the place to start.
If you want a quick way to sort your own use case, run it against this checklist:
- Answers depend on documents that change more than a few times a year: RAG.
- Users need to see where an answer came from: RAG.
- Different users are allowed to see different information: RAG with permission filtering at retrieval time.
- The data is structured (prices, stock, orders, balances): tool calls into the system of record, not RAG.
- The corpus fits comfortably in the context window and rarely changes: long context, no retrieval layer.
- Output format or classification stays inconsistent despite good prompts and examples: consider fine-tuning.
- You have thousands of consistent, labelled examples and high call volume: fine-tuning or distillation is worth pricing.
- Training data contains personal data: resolve the GDPR position before any fine-tuning.
If this were our business, here’s where we’d start
We wouldn’t start by choosing between RAG and fine-tuning. The first step is to collect 50 to 100 real questions or inputs the system will face, with the answers a competent member of staff would give, and turn them into an evaluation set. Then we’d build the simplest possible baseline: a good prompt, the relevant documents pasted in or a basic retrieval layer, and a script that scores the output against the evaluation set.
That baseline tells you which problem you actually have. If answers are wrong because the right passage wasn’t found, invest in retrieval. If the right passage was found but the output is still wrong in a consistent way, tighten the prompt, then look at fine-tuning. If the baseline is already good enough, ship it and spend the budget elsewhere.
It’s also worth checking early whether your documents and systems are in a state that can be retrieved from at all. An AI readiness audit of your software architecture answers that quickly. Our breakdown of what it costs to build an AI agent shows where the money tends to go once you commit.
D3X Solutions LABS is an AI engineering studio in Ireland that designs and builds this kind of system. That covers retrieval pipelines, evaluation harnesses, tool integrations into existing systems and, where the evidence supports it, fine-tuned models. We start with the evaluation set and the baseline, so the choice between RAG and fine-tuning is made on your data, not on preference. We also build so the underlying model can be swapped later without a rebuild. If you’re weighing up which approach your use case needs, book a free 30-minute call and we’ll work through it with you.
Frequently Asked Questions
Is RAG better than fine-tuning?
Neither is better in general; they solve different problems. RAG fixes a knowledge gap by giving the model your documents at query time, while fine-tuning changes the model's behaviour, such as output format, tone or a narrow classification task. For most business assistants that answer from company information, RAG is the right starting point.
Can fine-tuning teach an LLM new knowledge?
Only partially and unreliably. A fine-tuned model absorbs some facts from its training data, but unevenly, without being able to cite a source, and with no simple way to update those facts when they change. If accuracy on current facts matters, use retrieval or tool calls instead.
How much data do you need to fine-tune an LLM for business use?
Typically hundreds to thousands of high-quality, consistently labelled input–output examples, depending on the task and base model. Consistency matters more than raw volume: if two people label the same input differently, the model learns that inconsistency. Most of the effort goes into building and cleaning the dataset, not the training run.
Do I still need RAG now that models have large context windows?
Not always. If your corpus is small, stable and fits comfortably in the prompt, pasting it in full (often with prompt caching) can be simpler and more accurate than retrieval. RAG becomes worthwhile when the content is too large, changes frequently, or must be filtered by user permissions.
Can you combine RAG and fine-tuning?
Yes. A common pattern is a fine-tuned model that reliably produces a strict output format, fed with retrieved documents for the facts. It's usually a second-phase optimisation, justified once you have evaluation results showing prompting alone can't hold the required behaviour or the call volume makes a smaller model worth training.
Is fine-tuning on customer data a GDPR risk?
It can be. Personal data used in training is diffused through the model's weights, so honouring an erasure request generally means retraining without that data. With RAG you can delete the source document and re-index, which is far easier to manage under GDPR.
Want this working in your business?
D3X Solutions LABS builds AI agents and automation that pay back fast. Not sure where to start? A free 30-minute call is enough to find the first thing worth building.
Get in touch
