RAG vs fine-tuning for company data, and where long context fits

A company wants an assistant that answers from its own material: the policy handbook, the product sheets, the contract templates, three years of support tickets. In the first meeting someone proposes fine-tuning a model on the data. Someone else says RAG. A third person points out that current models read hundreds of thousands of tokens, so why not paste everything in. All three are answering different questions, and the project goes better once that is clear.

What each approach actually changes

There are four tools here, and they act in two different places.

Fine-tuning adjusts the weights by training on examples of inputs and the outputs you want. What it reliably teaches is behaviour: a format, a tone, a classification scheme, a way of handling a task. What it does badly is store facts.

That split, knowledge in the request versus behaviour in the weights, settles most arguments before they start. If the problem is that the model does not know your refund policy, the fix is to show it the refund policy. If the problem is that it knows the policy and still answers in three paragraphs when your support team writes two lines, that is behaviour, and prompting comes first, with fine-tuning behind it.

Approach	What it changes	Freshness	Permissions	Citations
Prompting	Instructions, worked examples and output format for one request	Edit the prompt	Not applicable	Not applicable
Long context	Puts the documents themselves into the request	Next request sees the new documents	A different context per user, filtered by whole file	Possible if each document is labelled in the request
RAG (retrieval-augmented generation)	A search step puts only the relevant passages into the request	Re-indexed when the sync job sees the change	Each passage carries its access list, filtered before the model	Per passage, checked mechanically
Fine-tuning	The weights	Old facts stay until you retrain	None: anything trained in can come out for any user	None: the weights do not record sources

Why fine-tuning is a poor way to teach facts

The intuition that training on your documents makes the model know them is natural and mostly wrong. Ovadia and colleagues compared the two directly and found that RAG consistently beat unsupervised fine-tuning at answering questions about the material, both for facts the model had partly seen before and for entirely new ones. A later study by Gekhman and colleagues, presented at EMNLP 2024, found that fine-tuning examples containing new facts are learned more slowly than examples consistent with what the model already knows, and that once they are learned, the model’s tendency to hallucinate rises with them.

Where fine-tuning earns its cost is in narrow, repeated behaviour. OpenAI’s own model optimisation guide lists classification, specific output formats and correcting instruction-following failures as the jobs for supervised fine-tuning, and notes that a tuned model can run with shorter prompts, which saves tokens at volume.

There is also a platform question, new in 2026. The same guide now says OpenAI is winding down its self-serve fine-tuning platform. Its deprecations page says that, from 6 January 2027, existing customers will no longer be able to create new fine-tuning jobs, and that tuned models remain available for inference only until their base models are deprecated. On a hosted platform, a tuned model lives only as long as the model it was tuned from, and then you train again. The alternative is to tune an open-weight model yourself, usually with an adapter method such as LoRA, which keeps the training cheap but makes you responsible for hosting, scaling and upgrading the model.

RAG vs long context for a small corpus

If the material is small, retrieval may be unnecessary. Anthropic’s write-up on contextual retrieval suggests that a knowledge base under about 200,000 tokens, roughly 500 pages, can simply go into the prompt. A staff handbook, a price list and a set of product sheets often fit.

The cost objection is weaker than it used to be because of prompt caching. When the same long prefix is sent repeatedly, the provider stores it, and later requests read it at a fraction of the normal input price. At the time of writing (October 2026), Anthropic’s prompt caching documentation prices cache reads at a tenth of the base input price on most models, and lower on some, with writes at 1.25 times the base price for the default five-minute cache. As an example with hypothetical numbers: say the handbook is 80,000 tokens and the assistant handles a steady stream of questions. Nearly every request after the first pays the cached rate on those 80,000 tokens, while the question and the answer are billed normally.

Long context has three limits. The first is attention. Liu and colleagues showed that models use information at the start and end of a long input better than information in the middle, and a question whose answer sits on page 240 of 500 is the case to test on whichever model you use. The second is that every request carries the whole corpus, so the cost and latency grow with the material rather than with the question. The third is permissions, which deserves its own section.

We met the attention limit in conversation form when we built the MVP for EroPlay.ai as technology partner: a long roleplay outgrows what a model can read at once, and the fix was summarising the conversation so a character stayed consistent. The same reasoning applies to documents. Once the material no longer fits comfortably, something has to choose what the model sees. That choice is retrieval, whatever it is called. Conversation memory has its own trade-offs, covered in memory for long AI conversations.

Freshness and permissions decide most projects

Company data changes and company data has owners. These two facts push most real projects towards retrieval more than any accuracy benchmark does.

Freshness first. In a RAG system a changed document is re-indexed as soon as the sync job sees it, and the next answer uses the new version. A deleted document disappears from the index and from every answer after that. In a fine-tuned model the old price list is in the weights until you retrain, and there is no practical way to remove one document from a trained model. You retrain without it and evaluate again.

Permissions are the stronger argument. In a retrieval system each passage carries the access list of the document it came from, and the search query filters by the identity of the person asking before any text reaches the model. The salesperson’s question never retrieves the board minutes, so the model cannot quote them. Weights have no access lists. Anything in the training data can come out for any user who asks the right way, and no prompt instruction makes that safe. Long context sits in between: you can build a different context per user, but at that point you are filtering documents by permission, which is retrieval at the level of whole files.

The part teams underestimate is that permissions change as often as content. Someone moves department, a shared folder is locked down, a contractor leaves. The sync job has to copy the access lists along with the text and on the same schedule, and a stale access list is a data leak that no evaluation will catch unless you test for it.

Citations come with retrieval

Because retrieved passages arrive with identifiers, the model can be asked to cite the passage behind each claim, and the application can check the citation mechanically: does the quoted text exist in the cited passage? An answer whose citation does not hold up is shown as unsupported or not shown. People trust an assistant they can check in one click, and they stop checking one they cannot.

This is the same rule we use for documents in bilz.ai, where we worked as technology partner: every extracted number links back to the region of the invoice it came from, and a value without a valid citation is treated as missing. The mechanics are in reading supplier invoices with an LLM. A fine-tuned model cannot cite in this way, because the weights do not record where a fact came from. It can produce something that looks like a citation, which is worse than none.

What each one costs to build and to keep current

RAG costs weeks, and little of that goes on the model. It goes on connectors to the places the documents live, on turning PDFs with tables and scanned pages into clean text, on splitting documents into passages that still make sense alone, on metadata and access lists, and on search quality. Plain embedding search misses exact terms such as product codes and clause numbers, so it is usually paired with keyword search and a reranking step. In Anthropic’s tests in the same contextual retrieval article, adding context to each chunk, combining embeddings with BM25 keyword search and reranking the results cut the rate at which relevant passages were missing from the top 20 results by 67 per cent.

Approach	To build	To keep current
Prompting	Hours	Little beyond editing the prompt
Long context	Days: gather the documents, clean them into text, cache the prefix, write the evaluation set	Little beyond updating the documents
RAG	Weeks, mostly connectors, parsing, chunking, access lists and search quality	Run the sync jobs, re-embed changed documents, re-embed everything when the embedding model changes
Fine-tuning	Curate the training set	The most: maintain the training set, then retrain and fully evaluate for every changed fact and every base model deprecation

Evaluating RAG and fine-tuning separately

A RAG system fails in two places, and they need separate numbers. Retrieval fails when the right passage is not among the ones the model sees: measure that as recall at the number of passages you pass, against questions where you know which document holds the answer. Generation fails when the right passage was there and the answer still went wrong, ignored it, or added something it did not say. Score faithfulness to the retrieved text and citation validity separately from correctness. Include questions the corpus cannot answer, because the right behaviour is to say so, and a system that never says so is guessing. How we build these sets from real past cases is in building an evaluation set from past cases.

For fine-tuning, the baseline is the important part. A tuned model has to beat the base model with the best prompt you can write, including a dozen good examples, not a bare prompt. A tuning result measured against a weak baseline is not a result.

When to combine RAG and fine-tuning

The common combination is retrieval for knowledge and something else for behaviour. A support assistant retrieves the relevant policy passages and drafts a reply in the team’s style. Usually the style comes from instructions and examples in the prompt, and the format from structured output, as described in reliable structured output from LLMs. Fine-tuning joins only when the behaviour stays wrong after that, at a volume where the shorter prompt pays for the training.

Fine-tuning also fits around retrieval rather than inside it. A small tuned model can classify incoming questions, rewrite them into search queries or rerank passages, which are narrow, repeated jobs where tuning works well and a cheap model is good enough. How to split that work between small and large models is in routing between cheap and frontier models.

Long context and retrieval combine too. As models read more, retrieval can return whole documents rather than small chunks, so the search only has to find the right contract and the model reads all of it.

A decision procedure

This is the order we follow, and each step has to fail on measured results before the next one starts.

1. Collect 30 to 50 real questions with correct answers and the documents that contain them.
2. Paste the right documents into the prompt by hand, question by question, and check the answers. If the model gets them wrong with the right material in front of it, no retrieval system will fix that. The problem is the documents, the instructions or the behaviour.
3. If the whole corpus fits within a couple of hundred thousand tokens, everyone may see all of it and it changes rarely, use long context with caching.
4. If the corpus is larger, changes weekly or has per-user permissions, build retrieval with access-list filtering, hybrid search and checked citations.
5. Fine-tune only for a behaviour that stays wrong after good prompting, when you have hundreds of examples of the right output and accept retraining whenever the base model is retired.
6. Run every step against the same question set, so each decision is a number.

Step 2 is the cheapest experiment on the list and it decides most projects in a day. If the answers are good with the right documents pasted in, the rest of the work is getting the right documents in front of the model, and that is a search and permissions problem rather than a model problem. The AI consulting page describes how we run that assessment, and custom AI apps covers building what comes after it. You can write to us with a sample of the questions your people ask.