This is one of the first questions when deploying AI in a company: how to make the model respond based on your knowledge, not general information. There are two paths - and most often, they’re confused or the more expensive one is chosen unnecessarily.
RAG: search first, then answer#
RAG (retrieval-augmented generation) first retrieves relevant snippets from your database, then instructs the model to answer only based on them, with citations. Knowledge lives outside the model - in a vector database - so:
- you update data without retraining the model,
- answers include citable sources, which limits hallucinations when retrieval is on-target and a confidence threshold is set,
- if the match is weak, the system escalates to a human instead of fabricating.
Embeddings are computed locally with the BGE-M3 model, so the content to be embedded never leaves your infrastructure.
Fine-tuning: change the model’s behavior#
Fine-tuning further trains the model on your examples and alters its weights: it cements style, tone, and output format. This is powerful when you need a consistent "voice" or a highly specific format that prompting alone can’t enforce. But it’s costly and not suited for fresh facts: new knowledge would require another training cycle.
When to use which#
| Criterion | RAG | Fine-tuning |
|---|---|---|
| Fresh/up-to-date data | yes | no |
| Deployment cost | low | high |
| Update without retraining | yes | no |
| Style/behavior control | partial | full |
| Hallucination risk | low (with a confidence threshold) | medium |
| Citable sources | yes | no |
| Time to first results | weeks | months |
| Required amount of data | little (documents) | a lot (training pairs) |
Rule of thumb: if the problem is access to knowledge (customers can’t find answers) - use RAG. If the problem is consistent style/format: use fine-tuning. Often, the optimal solution is a hybrid: RAG brings in facts, light fine-tuning cements the voice. Walk through the specifics in the decision tree.
If you’re leaning toward fine-tuning, see when fine-tuning really makes sense - and when it’s a costly mistake.
The most common mistake: fine-tuning on documents#
The most common mistake sounds like this: "we want the model to know our documents." That is not a job for fine-tuning - fine-tuning changes style and behavior, it is not factual memory, so the model can still hallucinate facts, just in your style. Knowledge from documents is a RAG task with source citations.
Second - orders of magnitude: you launch a RAG pilot in weeks, while fine-tuning is months of work plus GPUs, training data (at least several hundred good input-output pairs), and maintaining successive versions. We break down the full list of cases where fine-tuning is justified or a mistake in the article When fine-tuning makes sense.
What we build with RAG#
RAG is the foundation of Concierge RAG - an assistant on your knowledge with citations, multilingual support, and escalation to humans. The same pattern powers multilingual help desks and document intelligence.
Try it live#
Paste your own text and ask a question - you’ll see RAG with live citations (same sandbox as in the playground: zero retention, PII masked).
FAQ#
RAG or fine-tuning: what to choose at the start?#
Most often, RAG. It’s cheaper, updatable without retraining, and provides citable sources. Fine-tuning makes sense when you need a permanent change in style or format, not fresh facts. Many deployments start with RAG and add light fine-tuning only when a consistent "voice" is critical.
Does RAG require sending data to the cloud?#
Not necessarily. We keep embeddings and search local (BGE-M3 + Qdrant), and only a masked prompt - without PII - goes to the cloud. Sensitive data and entire on-prem deployments never leave your infrastructure.
Does fine-tuning reduce hallucinations?#
Not like RAG. Fine-tuning cements style, but the model can still "fabricate" when it lacks facts. RAG with citations and a confidence threshold (escalation to a human when the match is weak) is the primary defense against hallucination.