This is one of the first questions when deploying AI in a company: how to make the model respond based on your knowledge, not general information. There are two paths—and most often, they’re confused or the more expensive one is chosen unnecessarily.
RAG: search first, then answer#
RAG (retrieval-augmented generation) first retrieves relevant snippets from your database, then instructs the model to answer only based on them, with citations. Knowledge lives outside the model—in a vector database—so:
- you update data without retraining the model,
- answers include citable sources, which limits hallucinations when retrieval is on-target and a confidence threshold is set,
- if the match is weak, the system escalates to a human instead of fabricating.
Embeddings are computed locally with the BGE-M3 model, so the content to be embedded never leaves your infrastructure.
Fine-tuning: change the model’s behavior#
Fine-tuning further trains the model on your examples and alters its weights—it cements style, tone, and output format. This is powerful when you need a consistent "voice" or a highly specific format that prompting alone can’t enforce. But it’s costly and not suited for fresh facts: new knowledge would require another training cycle.
When to use which#
| Criterion | RAG | Fine-tuning |
|---|---|---|
| Fresh/up-to-date data | yes | no |
| Deployment cost | low | high |
| Update without retraining | yes | no |
| Style/behavior control | partial | full |
| Hallucination risk | low (with a confidence threshold) | medium |
| Citable sources | yes | no |
| Time to first results | weeks | months |
| Required amount of data | little (documents) | a lot (training pairs) |
Rule of thumb: if the problem is access to knowledge (customers can’t find answers)—use RAG. If the problem is consistent style/format—use fine-tuning. Often, the optimal solution is a hybrid: RAG brings in facts, light fine-tuning cements the voice. Walk through the specifics in the decision tree.
If you’re leaning toward fine-tuning, see when fine-tuning really makes sense—and when it’s a costly mistake.
The most common mistake: fine-tuning on documents#
The most common mistake sounds like this: "we want the model to know our documents." That is not a job for fine-tuning—fine-tuning changes style and behavior, it is not factual memory, so the model can still hallucinate facts, just in your style. Knowledge from documents is a RAG task with source citations.
Second—orders of magnitude: you launch a RAG pilot in weeks, while fine-tuning is months of work plus GPUs, training data (at least several hundred good input-output pairs), and maintaining successive versions. We break down the full list of cases where fine-tuning is justified or a mistake in the article When fine-tuning makes sense.
What we build with RAG#
RAG is the foundation of Concierge RAG—an assistant on your knowledge with citations, multilingual support, and escalation to humans. The same pattern powers multilingual help desks and document intelligence.
Try it live#
Paste your own text and ask a question—you’ll see RAG with live citations (same sandbox as in the playground: zero retention, PII masked).
FAQ#
RAG or fine-tuning—what to choose at the start?#
Most often, RAG. It’s cheaper, updatable without retraining, and provides citable sources. Fine-tuning makes sense when you need a permanent change in style or format, not fresh facts. Many deployments start with RAG and add light fine-tuning only when a consistent "voice" is critical.
Does RAG require sending data to the cloud?#
Not necessarily. We keep embeddings and search local (BGE-M3 + Qdrant), and only a masked prompt—without PII—goes to the cloud. Sensitive data and entire on-prem deployments never leave your infrastructure.
Does fine-tuning reduce hallucinations?#
Not like RAG. Fine-tuning cements style, but the model can still "fabricate" when it lacks facts. RAG with citations and a confidence threshold (escalation to a human when the match is weak) is the primary defense against hallucination.