Dependency on a single API provider is a silent risk: costs grow with traffic, data leaves your environment, and switching providers means rewriting integrations.
What constitutes sovereignty#
- Serving LLM locally (vLLM, Ollama) — predictable latency and cost.
- Embedding server (BGE-M3) as the foundation for semantic search.
- RAG on company knowledge — answers from your documents, with citations.
- Router / gateway unifying input and controlling cost.
Design for exit, not lock-in#
Key principle: the system must allow switching providers — never the other way around. A router lets you mix local models (for sensitive paths) with cloud models (where power is needed), without rewriting code.
What about GDPR#
Compliance is designed from the start (compliance-by-design): in the on-prem variant, the generation layer never leaves the company, and whenever the router sends selected, safe paths to the cloud, PII is masked before anything goes out. Security and GDPR take priority over individual features.
You don’t need a GPU cluster right away — we tailor the variant to real workload and budget. Predictable cost matters, not maxing out hardware.
How much hardware you really need#
The hardware variant depends primarily on model size and quantization — that is, how much VRAM is needed to load the entire model into the card’s memory. In practice, the rough requirements look like this:
- A 7B model in Q4 quantization — about 4–5 GB of VRAM. It fits on a single consumer card.
- A 13B model — about 8–10 GB of VRAM. A single 24 GB card is enough, with headroom for context.
- A 70B model — at least 40–48 GB of VRAM, meaning two 24 GB cards joined by NVLink or one professional 48 GB+ card.
That is why sovereignty does not mean a cluster from day one. A realistic pilot usually starts with a small Q4 model — first on CPU for a prototype, and only once volume and latency requirements justify it does it move to GPU. The full breakdown of cards, throughput, and hardware cost is covered in our guide to local LLMs and choosing a GPU.
When self-hosting beats an API#
This is a financial decision, not just a technical one. With steady, high query volume, the one-time hardware cost (CAPEX) amortizes, and you mainly keep paying for power — then your own infrastructure works out cheaper than a per-token bill. With low or irregular traffic, the cloud usually wins: you don’t lock up capital in hardware you won’t use. The full break-even calculation — with concrete ranges and a break-even scenario — is in the comparison of local LLM cost vs API cost.
What sovereignty asks for in return#
Sovereignty isn’t only upside — it also brings operational obligations you have to factor into the decision honestly:
- Maintenance and on-call — your own infrastructure needs monitoring and incident response (on-call), not just a one-off deployment.
- Model and driver updates — new model versions, serving libraries, and GPU drivers have to be updated and tested.
- Hardware redundancy — a single server is a single point of failure; high availability means spare hardware or failover.
- Scaling at peaks — the cloud scales elastically with traffic; locally you handle peaks within the limits of the hardware you own (a hybrid with a router helps here).
- Access to the newest models — the latest models are sometimes available in an API earlier than locally.
That is why we tailor the variant to real workload, keep high-volume paths local, and add the cloud where elasticity is needed. The full operational bill (monitoring, updates, oversight) is laid out in our guide to AI agent maintenance costs.
Self-hosted vs cloud API#
| Self-hosted | Cloud API | |
|---|---|---|
| Cost | Predictable (CAPEX + power) | Variable, grows with traffic |
| Data privacy | Data stays with you | Data leaves to provider |
| Control | Full (model, version, fine-tuning) | Limited to API |
| Provider dependency | None (you can switch) | Lock-in on pricing and features |
| Entry threshold | Higher (hardware, deployment) | Low (API key) |
| Maintenance and scaling | On your side (on-call, updates, redundancy) | On the provider’s side (elastic scaling) |
Related paths#
For a deeper implementation path, read the guide to self-hosted LLMs and GDPR, the comparison of local LLM cost vs API cost, and the product page for BGE-M3 semantic search. You can compute your own break-even point in the inference cost calculator.
FAQ#
What is sovereign AI infrastructure?#
Models on your hardware, with ownership of code and data — self-hosting instead of dependency on a single provider. We design it so you can switch providers, never the other way around.
Do I need my own servers or GPUs?#
Not necessarily. We tailor the variant to real workload and budget — from small models to a cluster. Predictable cost matters, not maxing out hardware.
How does sovereign infrastructure affect cost?#
Predictable cost instead of a surprise bill: instead of paying per token in the cloud, you control performance and cost on your own hardware.