Key takeaways
- If the problem is 'the model doesn't know our facts', you need retrieval, not fine-tuning.
- If the problem is 'the model won't follow our format', fine-tuning may help.
- Build an evaluation set before either. Without it you can't tell whether anything improved.
- Most projects that fine-tuned first would have got further with better retrieval.
01The distinction that matters
Retrieval-augmented generation gives the model the right information at the moment it answers. Fine-tuning changes how the model behaves. Those solve genuinely different problems, and conflating them is the most common expensive mistake in applied AI right now.
If your chatbot doesn't know your refund policy, that's a knowledge problem — retrieval. If your chatbot knows the policy but explains it in a tone that doesn't sound like you, that's a behaviour problem — potentially fine-tuning.
02Why retrieval usually wins
It's cheaper to build, far cheaper to update, and it can cite its sources. When your policy changes, you update a document rather than retrain a model. When someone asks where an answer came from, you can show them.
Fine-tuning bakes knowledge into weights, where it can't be inspected, cited or easily corrected. For anything that changes — pricing, policy, product detail — that's the wrong shape.
03When fine-tuning earns its cost
Three cases genuinely justify it. First, consistent structured output where prompting keeps drifting. Second, a specialised domain vocabulary the base model handles badly. Third, cost — fine-tuning a small model to match a large model's performance on one narrow task can cut inference spend substantially at volume.
Notice that none of those is 'the model doesn't know about our company'. That's retrieval.
04The step almost everyone skips
Build an evaluation set first. Thirty to a hundred real questions with answers you'd accept, scored automatically. Without it, you cannot tell whether a prompt change, a retrieval change or a fine-tune actually helped — you're relying on the impression of whoever tried it last.
This is genuinely the highest-value thing to build in any LLM project, and it's the one that gets cut for time. Teams then spend months tuning blind.
05Why RAG systems usually fail
Almost always retrieval, not the model. If the right passage isn't retrieved, a capable model answers confidently from memory and sounds authoritative doing it. That reads as hallucination but it's a search problem.
Measure retrieval accuracy separately from answer quality. Chunking strategy affects results more than model choice, and hybrid search — semantic plus keyword — beats pure vector search, because pure vector search misses exact terms like part numbers and error codes.
Written by Sedwis Engineering · Engineering team · Published 14 July 2026