RAG vs fine-tuning: which should you use?

Key facts

• Retrieval-augmented generation was introduced by Lewis et al. (NeurIPS 2020), combining a pretrained generator with a dense vector index of Wikipedia to address provenance and the updating of world knowledge. (Source: Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401))
• Ovadia et al. (2023) found that RAG consistently outperformed unsupervised fine-tuning at injecting knowledge into language models, both for facts seen in training and for entirely new ones. (Source: Ovadia et al., Fine-Tuning or Retrieval? (arXiv 2312.05934))
• Gekhman et al. (EMNLP 2024) found that fine-tuning examples introducing new facts are learned much more slowly than familiar ones, and that as they are learned the model's tendency to hallucinate increases. (Source: Gekhman et al., Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (arXiv 2405.05904))
• LoRA (Hu et al., 2021) freezes a model's pretrained weights and trains small low-rank matrices, reducing trainable parameters about 10,000 times and GPU memory about 3 times compared with full fine-tuning of GPT-3 175B. (Source: Hu et al., LoRA (arXiv 2106.09685))
• QLoRA (Dettmers et al., 2023) fine-tuned a 65-billion-parameter model on a single 48GB GPU while preserving the task performance of full 16-bit fine-tuning. (Source: Dettmers et al., QLoRA (arXiv 2305.14314))
• As of November 2026, OpenAI's self-serve fine-tuning is closed to organisations that have never used it, and existing active customers can create new fine-tuning jobs only until 6 January 2027. (Source: OpenAI API deprecations)
• RAFT (Zhang et al., 2024) combines the two methods by fine-tuning a model to answer from retrieved documents, ignore distractor documents and cite the passage that supports its answer. (Source: Zhang et al., RAFT (arXiv 2403.10131))
• Dr. Dulan Dias built Legify.ai, an AI legal-contract platform, at EchonLabs around a jurisdiction-based retrieval-augmented generation engine. (Source: Dr. Dulan Dias: Journey)

Definitions

Retrieval-augmented generation (RAG)

A pattern in which the application searches its own data for passages relevant to a request and adds them to the prompt, so the model answers from supplied facts rather than from memory.

Fine-tuning

Further training of a pretrained model on task-specific examples. It changes the model's weights, and so its default behaviour on every request.

LoRA

Low-Rank Adaptation, a parameter-efficient fine-tuning method that freezes the original weights and trains small added matrices, which makes fine-tuning far cheaper in memory and storage.

Embedding

A vector of numbers that represents the meaning of a piece of text. In RAG, embeddings let the system find passages close in meaning to a question, not only those sharing its words.

Knowledge or behaviour: the question that decides it

Ask what is wrong with the model's answers today. If it lacks facts, such as your policies, your products or anything after its training cut-off, that is a knowledge problem and retrieval fixes it. If it has the facts but answers in the wrong shape, tone or level of detail, or is inconsistent at a narrow task, that is a behaviour problem and fine-tuning is the stronger tool.

The research supports the split. Ovadia et al. found RAG consistently beat unsupervised fine-tuning at injecting knowledge, and Gekhman et al. found that fine-tuning on facts a model did not already know made it more prone to hallucinate. Their conclusion: models acquire knowledge mostly in pretraining, and fine-tuning mostly teaches them to use it.

Chapter 5 of From Model to Product, Model Architecture Patterns, sets out prompting, fine-tuning and retrieval as the three design patterns around a language model: retrieval for domain-specific, changing or proprietary information, and fine-tuning when outputs must follow strict formats or the domain's language defeats the base model.

The decision table

Read each row against your product. Most teams find the knowledge rows point to RAG and only one or two behaviour rows point to fine-tuning, which is the case for adding fine-tuning later rather than first.

Try them in this order: prompt, retrieve, fine-tune

Each step is cheaper to try and to undo than the next, so move on only when your measurements say the current one has run out.

1. Build a small evaluation set first: real inputs, each with what a good answer contains. Without it you cannot tell whether any later step helped; how to evaluate an LLM application covers how.
2. Prompt well. Clear instructions, a response format and a few examples solve more than teams expect. Stop here if the evaluation passes.
3. Add retrieval when answers fail for lack of facts. Retrieve only what each question needs, and check that the right passages come back before you judge the answers.
4. Fine-tune when the evaluation still fails on behaviour that prompting cannot fix reliably, or when a prompt full of examples costs too much at volume.
5. Revisit the choice as volume grows: a fine-tuned small model can take over a task that a large prompted model has proved worth doing.

When fine-tuning is the right call

Fine-tuning earns its cost when the problem is consistent behaviour at volume, and the evidence is failed evaluations rather than a hunch.

Parameter-efficient methods made it far cheaper. LoRA trains small added matrices instead of the whole model, and QLoRA fine-tuned a 65-billion-parameter model on a single 48GB GPU. Access is the other constraint: as of November 2026, OpenAI is winding down its self-serve fine-tuning, and a fine-tuned model lasts only as long as its base model. Tuning an open-weight model keeps that decision in your hands; open-weight vs proprietary models covers the trade.

Good reasons to fine-tune:

• A strict output format or house style that instructions and schemas do not hold reliably across thousands of requests.
• A narrow, high-volume task such as classification or extraction, where a small tuned model can replace a large prompted one at lower latency and cost.
• Prompts that carry dozens of examples on every call, where training the examples in shortens every request.
• Domain language, such as clinical or legal phrasing, that the base model handles poorly.

When RAG is the right call, and what it costs

RAG is the default whenever the answer depends on facts the model cannot know: your documents, your customers' records, today's prices, this week's policy. It is also the only one of the two that can show a source for each claim and respect per-user permissions at request time.

From practice: Legify.ai, the AI legal-contract platform Dr. Dulan Dias built at EchonLabs, uses a jurisdiction-based RAG engine. Legal content is a clear case for retrieval, because a clause that is right in one jurisdiction can be wrong in another, and the law changes, so the applicable text has to be selected at request time rather than trained into weights.

The costs are real. Every retrieved passage is input tokens on every call, so RAG raises cost per request; see LLM cost per user. Retrieval quality caps answer quality, and long contexts are no shortcut: Liu et al. found models use information in the middle of a long input markedly worse than at its start or end.

Storage is often simpler than vendors suggest. Chapter 11 of From Model to Product builds retrieval from ordinary database queries with no vector database at all, and pgvector or a vector database covers when semantic search earns a store of its own.

Using RAG and fine-tuning together

The two are not rivals: retrieval supplies the facts and fine-tuning shapes how the model uses them. Each common combination fixes a specific weakness.

• Fine-tune the generator to use retrieved context. RAFT trains a model to answer from the relevant documents, ignore distractors and cite the supporting passage, which suits a domain with a settled body of documents.
• Fine-tune the retrieval side. When search misses your domain's vocabulary, tuning the embedding model or adding a reranker can lift answer quality without touching the generator.
• Distil a prompted system into a smaller model. Run a large model with RAG, review its outputs, and, where the large model's terms allow it, train a smaller model on the good ones; it keeps the retrieval step but costs less per call.
• Fine-tune for format, retrieve for facts. A model tuned to your schema or house style still gets its facts from retrieval, so they stay current without retraining.

Mistakes that waste the effort

Most disappointing projects in this area make one of five mistakes.

• Fine-tuning to teach facts. It is slow, the facts go stale, and on the evidence above it can raise hallucination rates.
• Judging RAG by its answers alone. Measure retrieval separately, since no generator can fix a passage that never arrived.
• Fine-tuning without a held-out evaluation set, so nobody can tell whether the tuned model is better or just different.
• Stuffing whole documents into the prompt because the context window allows it, which raises cost and buries the passage that matters.
• Forgetting the lifecycle. A fine-tuned model is tied to its base model, and an index is tied to its embedding model, so plan for retraining and re-embedding when either is replaced.

Frequently asked questions

Sources

1. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401)
2. Ovadia et al., Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (arXiv 2312.05934)
3. Gekhman et al., Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (arXiv 2405.05904)
4. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (arXiv 2106.09685)
5. Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (arXiv 2305.14314)
6. Zhang et al., RAFT: Adapting Language Model to Domain Specific RAG (arXiv 2403.10131)
7. Liu et al., Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172)
8. OpenAI API deprecations