Skip to content
TRUELAB.AI
[Lab note]2026-05-122 min read#local-llm#ollama#sovereignty

Why we run LLMs locally

Cloud APIs are the easy default. Here is what changed our math: latency you control, prices that don't move, and data that never crosses the Atlantic.

Every AI project we scope starts with the same question from the client: "Which API should we use?" Increasingly, our answer is: none.

The obvious objections

Local models used to mean toy models. That stopped being true somewhere between Llama 3 and the current generation of open-weight releases. For the retrieval, extraction, classification and summarization work that makes up most production AI, a well-chosen 8B–70B model on your own hardware is not a compromise — it is the boring, correct choice.

The objections we hear are always the same three:

  1. "Local models are worse." For open-ended creative reasoning, often yes. For your actual workload — structured extraction over invoices, grounded answers over your wiki — the gap disappears once you measure it. We benchmark per use case, and the local option wins more often than the API marketing suggests.
  2. "GPUs are expensive." A used RTX 4090 costs about two months of a moderately busy GPT-4-class API bill. It then runs for years. The accounting department can do the rest.
  3. "It's harder to operate." True. That is an engineering problem, and engineering problems have solutions you own. Vendor deprecation notices do not.

What a minimal deployment looks like

Our smallest production pattern is one box, one binary, one systemd unit:

# Ollama as a service, bound to the internal WireGuard interface only
curl -fsSL https://ollama.com/install.sh | sh
OLLAMA_HOST=10.10.0.3:11434 ollama serve &
ollama pull llama3.3:70b-instruct-q4_K_M

In front of it: NGINX for TLS, WireGuard for transport, and a thin Node.js service that owns prompting, retries and logging. No request ever leaves the network. The GDPR data-flow diagram for this setup fits on a napkin — try that with a US API provider and a transfer impact assessment.

When we still reach for an API

Honesty clause: frontier reasoning tasks, one-off prototypes, and workloads too small to justify hardware still go to hosted models — through an EU endpoint where possible. Sovereignty is a dial, not a religion. But the default has flipped: we now need a reason to send your data out, not a reason to keep it home.

That inversion is the whole point. Once the weights sit on your hardware, model upgrades become a download, not a contract negotiation.