Prompting vs RAG vs fine-tuning
Pick the cheapest approach that works.
Create a free account Try part of it here
Why it matters
"Should we fine-tune?" is one of the most common questions teams ask about language models. The answer is usually "not yet". Fine-tuning means training a model further on your own examples. It works well for some jobs. But it's slow to change, it ties you to one model version, and teams often use it for jobs it does badly, like teaching facts. Knowing what to try first, and what to try next, saves months of work and a lot of money.
What you'll be able to do
- Match the problem to the tool: prompting for instructions, retrieval for knowledge, fine-tuning for behavior and cost
- Estimate monthly cost for prompting, prompting with caching, and a fine-tuned small model
- Work out when a fine-tune pays for itself
- Plan an evaluation-first approach before choosing
What it covers
- Three tools for three problems
- How you pay for a model
- Worked example: one request, one payback
- Tagging reviews at three volumes
- A decision you can explain
- If you do fine-tune
Three tools for three problems
Picture a new hire at your company. Three different things can go wrong, and each has its own fix.
- They don't know what you want. You tell them clearly, maybe with a few samples of good work.
- They don't know the facts, like this season's prices or the new refund rules. You hand them the handbook and ask them to look things up.
- They need a knack that's hard to put into words, like the exact feel of your brand voice. They learn it by practice on many examples.
A language model has the same three fixes. Telling it clearly is prompting: instructions and examples inside the request, as in Phase 1. Handing it the handbook is retrieval, or RAG (retrieval-augmented generation). Your app finds the right documents and pastes them into the request, also as in Phase 1. Practice is fine-tuning: more training on your own examples. It changes the model's weights, the millions or billions of numbers it learned in training.
There's a fourth case. Prompting works, but at your volume it costs too much or runs too slowly. Then you can fine-tune a smaller, cheaper model to do that one job. It's like training a junior to handle a routine task, instead of paying a senior expert to do it.
| The problem | Usually the right tool |
|---|---|
| The model doesn't know what you want it to do | Prompting: clear instructions, a few examples, structured output (Phase 1) |
| The model doesn't know the facts (your products, policies, last week's numbers) | Retrieval (RAG): put the relevant documents in the context (Phase 1) |
| You can't describe the behavior well enough, or it must be extremely consistent | Fine-tuning on examples of the behavior |
| Prompting works but costs too much or is too slow at your volume | Fine-tuning a smaller model (or distilling a big one, later this phase) |
Show the answer
Put them in the context, for example with RAG
Facts change, so they belong where you can update them for free. Weights are for behavior.
Two rules sit behind the table.
Facts go in the context. Behavior can go in the weights. The context is everything the model reads in one request. The SFT lesson showed that fine-tuning teaches format and style reliably. Facts are different. A fine-tuned model learns them unreliably. You can't update them without training again. And the model can't point to where a fact came from. Retrieval handles all three. The facts sit in the request, you update them by editing a document, and the model can quote its source.
Start cheap and measure. Changing a prompt takes minutes. A fine-tune takes days of work on the data. You also have to redo it when a better base model comes out. So climb one step at a time. Take the next step only when a test shows the current one isn't good enough.
That test is an evaluation set: real example inputs with the right answers. Think of it as a practice exam with an answer key. You score every option on the same set, so you compare like with like.
You can mix the tools. Many real systems retrieve documents and use a fine-tuned model. That model has learned how to use the documents in the house style.
Show the answer
Retrieve the current policy text into the prompt (RAG) and tell the model to answer from it
Missing or changing facts are a knowledge problem, so put the facts in the context. When the policy changes again, you edit one document.
How you pay for a model
Cost is often the real reason to fine-tune. To see why, you need to know how the bill works.
Providers charge by the token, a chunk of text. A token is roughly three-quarters of an English word. Prices are quoted in dollars per million tokens. You pay for input tokens, which is everything you send. You also pay for output tokens, which is everything the model writes back. Output usually costs more per token.
There's one discount to know about. With prompt caching (from Phase 1), the provider reuses its work on the fixed opening part of your prompt. Those cached tokens cost a fraction of the normal input price. In this lesson's prices, they cost a tenth.
Here's the cost of one request:
In plain words: multiply the number of tokens by the price per million, then divide by a million. Do it once for input and once for output, and add the two. Then multiply by the number of requests to get the month's bill.
A fine-tune also has a one-time setup cost. You pay to collect and label examples, plus a training bill. After that, it may save money every month. The payback period is how long those savings take to cover the setup cost.
In plain words: divide what you spend up front by what you save each month. If you save nothing, or the new option costs more, it never pays back.
Worked example: one request, one payback
Say a request sends 1,000 input tokens, and input costs $2 per million tokens. Leave output out for now. One request costs 1,000 × 2 ÷ 1,000,000 = $0.002, a fifth of a cent. Over 10,000 requests a month, that's $20.
Now a fine-tune. It costs $600 to set up. It brings a monthly bill down from $250 to $100, so it saves $150 a month. The payback is 600 ÷ 150 = 4 months.
Predict what the code prints, then run it.
price_per_million = 2.00 # dollars per million input tokens tokens = 1_000 # input tokens in one request per_request = tokens * price_per_million / 1_000_000 print(f"one request: ${per_request}") print(f"10,000 requests: ${per_request * 10_000:.2f}") setup = 600 # one-time cost of the fine-tune before, after = 250, 100 # monthly bill without and with it print(f"payback: {setup / (before - after)} months")
One request costs very little, but volume multiplies it. After four months the fine-tune has repaid its $600 setup. From then on, the $150 a month is pure saving. A real comparison also counts output tokens, and costs that never show up on a price list, like engineering time.
print(900 / (400 - 100), "months")
Show the answer
3.0 months
The saving is $400 − $100 = $300 a month, and 900 ÷ 300 = 3. Python prints 3.0 because / always gives a decimal number.
Tagging reviews at three volumes
Now a real job. A marketing team wants every product review tagged with one of 12 themes, such as delivery, price or sizing. There are two ways to do it.
- A large model, prompted. It needs long instructions and about 30 worked examples in every request. That's 4,000 tokens before the review even starts.
- A small model, fine-tuned. It already learned the task in training. It needs only the review and a short instruction of about 50 tokens.
Each review is about 150 tokens, and each answer (the theme) is about 5. The fine-tune has a one-time setup cost of $3,050. That covers labeling 1,500 example reviews, plus a small training bill.
The code compares three options at three volumes. The middle option is the large model with its 4,000 tokens of instructions cached. Prices vary by provider and change often, so these are illustrative:
# illustrative prices in dollars per million tokens; check your provider's current price list LARGE = {"input": 5.00, "cached_input": 0.50, "output": 25.00} SMALL_TUNED = {"input": 0.80, "output": 4.00} def monthly_cost(requests, input_tokens, output_tokens, price, cached_tokens=0): fresh = input_tokens - cached_tokens per_request = (fresh * price["input"] + cached_tokens * price.get("cached_input", price["input"]) + output_tokens * price["output"]) / 1e6 return requests * per_request instructions = 4_000 # instructions plus 30 worked examples, in every prompted request review, label = 150, 5 # tokens per review, and per answer setup_cost = 3_050 # one-time: labeling 1,500 examples, plus a small training bill print(f"{'reviews/month':>13} {'large, prompted':>16} {'+ caching':>10} {'small, tuned':>13} fine-tune pays off after") for volume in [20_000, 200_000, 2_000_000]: prompted = monthly_cost(volume, instructions + review, label, LARGE) cached = monthly_cost(volume, instructions + review, label, LARGE, cached_tokens=instructions) tuned = monthly_cost(volume, review + 50, label, SMALL_TUNED) months = setup_cost / (cached - tuned) print(f"{volume:>13,} {'$' + format(prompted, ',.0f'):>16} {'$' + format(cached, ',.0f'):>10} {'$' + format(tuned, ',.0f'):>13} {months:.1f} months")
Read the first row. At 20,000 reviews a month, the prompted large model costs $418. Caching brings that down to $58. The fine-tuned small model costs about $4. The last column is the payback formula. It treats the cached option as "before", because that's the cheapest way to run the large model. The table rounds to whole dollars, but the code uses the exact bills, $57.50 and $3.60. So it's $3,050 ÷ ($57.50 − $3.60) = $3,050 ÷ $53.90, which is about 56.6 months.
The numbers teach three things.
- Caching changes the picture. It cuts the prompted cost about seven times, because the long instructions are the same in every request. Always compare against the cached price.
- At low volume, prompting wins. At 20,000 reviews a month, the fine-tune takes nearly five years to pay for its setup.
- At high volume, fine-tuning wins fast. At 2 million reviews a month, it pays back in under a month. But that holds only if the small model is as accurate on your evaluation set. Check that before you switch.
The real comparison has more costs too. Count the engineering time, the retraining for each new model generation, and the monitoring.
Go deeper How the cost function adds things up
monthly_cost prices one request, then multiplies by the number of requests.
freshis the input that isn't cached: all the input tokens minus the cached ones.- Fresh tokens are charged at the
"input"price. - Cached tokens are charged at the
"cached_input"price.price.get("cached_input", price["input"])means "use the cached price if there is one, otherwise the normal input price". The small model's price list has no cached price, so it falls back. - Output tokens are charged at the
"output"price. 1e6is Python's short way to write 1,000,000. Dividing by it turns "per million" prices into dollars.
Check the first row by hand. One prompted request sends 4,150 input tokens: 4,150 × $5 ÷ 1,000,000 is $0.02075. Its 5 output tokens add $0.000125. That's about $0.0209 a request, and 20,000 requests come to $418.
At low volume, the setup cost never pays back. At high volume it pays back in days. The prices here are examples; plug in your own.
Show the answer
Caching is the cheapest way to run the large model, so it's the fair thing to beat
Comparing against the uncached price would make the fine-tune look far better than it is. Anyone running the large model at volume would turn caching on.
A decision you can explain
The order of the questions matters more than any single rule. The function below asks them in order:
- For every task: build an evaluation set, and try a strong model with a clear prompt.
- Does the task need facts the model can't know? Add retrieval.
- Is the behavior hard to describe, and do you have at least 200 good examples? Fine-tune, if prompting can't reach the quality you need.
- Is the volume huge (500,000 requests a month or more), or does speed matter most? Consider a fine-tuned small model.
The cut-offs of 200 examples and 500,000 requests are rough rules of thumb for this lesson. They aren't fixed laws. Here it is as code, applied to four typical requests:
def recommend(task): steps = ["Build an evaluation set, then try a strong model with a clear prompt and a few examples"] if task["needs_facts_it_cannot_know"]: steps.append("Add retrieval (RAG): facts belong in the context, not the weights") if not task["behavior_easy_to_describe"] and task["good_examples"] >= 200: steps.append("Fine-tune on the examples, if prompting can't reach the quality bar") if task["requests_per_month"] >= 500_000 or task["latency_critical"]: steps.append("Consider a fine-tuned small model for cost and speed, if it matches on the evals") return steps tasks = { "Answer return-policy questions": dict(needs_facts_it_cannot_know=True, behavior_easy_to_describe=True, good_examples=0, requests_per_month=3_000, latency_critical=False), "Tag 2M reviews a month by theme": dict(needs_facts_it_cannot_know=False, behavior_easy_to_describe=True, good_examples=1_500, requests_per_month=2_000_000, latency_critical=False), "Write in our very particular voice": dict(needs_facts_it_cannot_know=False, behavior_easy_to_describe=False, good_examples=800, requests_per_month=10_000, latency_critical=False), "Weekly competitor summary": dict(needs_facts_it_cannot_know=True, behavior_easy_to_describe=True, good_examples=5, requests_per_month=4, latency_critical=False), } for name, task in tasks.items(): print(name) for i, step in enumerate(recommend(task), 1): print(f" {i}. {step}")
Every path starts the same way: an evaluation set and a good prompt. Only some tasks ever need more.
- The return-policy bot and the competitor summary need facts, so they get retrieval.
- The review tagger runs 2 million times a month, so a small fine-tuned model is worth testing.
- Only the brand voice points to fine-tuning for quality. It's hard to describe, and there are 800 good examples to learn from.
Show the answer
Only step 1: build an evaluation set and try a strong model with a clear prompt
50 is below the 200 examples the fine-tune step asks for. The task needs no outside facts, and 10,000 requests a month is far below 500,000. So only the first step is left: work on the prompt, and collect more examples if it falls short.
If you do fine-tune
- Data. Collect a few hundred to a few thousand clean, consistent examples of exactly the behavior you want. Include the hard cases. Set a test set aside before training, and never train on it.
- Baselines. A baseline is the option you have to beat. Score three models on the same evaluation set: the fine-tuned model, the prompted large model and the prompted small model. The most common win is a small model fine-tuned to match a large one on one narrow task. Beating a large model at its general skills is rare.
- Method. For open models, the usual default is parameter-efficient fine-tuning. It trains a small set of extra numbers instead of all the weights. LoRA, the next lesson, is the main example. Hosted providers also let you fine-tune some of their models through an API.
- Upkeep. Budget for training again when you change base models. Keep monitoring too, because the messages your users send change over time.
For marketers: before anyone proposes fine-tuning, ask for three numbers. First, the quality the current prompt reaches on a labeled test set. Second, the monthly volume. Third, the cost per request. Those three usually settle the question.
Practice what you just read
- 3 code examples you can edit and run
- a practice exercise that checks your code
- a 5-question quiz
- 5 review cards for spaced repetition
Try a quiz question
Which problem is fine-tuning best suited to?