To run a self-hosted LLM properly you need three things. The first is a GPU with enough VRAM for the model weights plus the KV cache your real context lengths and concurrency demand. The second is an inference server that batches requests efficiently. The third is the operational layer around both: authentication, monitoring, updates and a fallback for when the box goes down. A single 24 GB card will serve a capable 8–14B model to a small team today. For 70B-class models you need 48 GB or more, multiple cards or a unified-memory machine, and each of those trades speed against capacity in a different way.
This piece assumes you already have a reason to keep inference in-house on hardware you own and want to know what you are signing up for. If you are still deciding whether to self-host at all, start with our comparison of self-hosted LLM vs API on cost, latency and GDPR.
Start with the workload, not the GPU
The usual sizing mistake is to buy the card first and then discover what it can run. Work the other way round. Before you look at hardware, pin down four numbers:
- The model size you need: 8B, 30B or 70B parameters.
- Typical and maximum context length: a chat assistant answering short questions is very different from a pipeline stuffing 30-page contracts into the prompt.
- Concurrency: how many requests run at the same moment, not how many users exist.
- Acceptable latency: time to first token and tokens per second.
Those four numbers decide almost everything else.
VRAM: the arithmetic that matters
Model weights take roughly parameters × bytes per parameter. An 8B model at 16-bit precision needs about 16 GB. Quantised to 4-bit it needs around 5 GB. A 70B model at 16-bit needs about 140 GB, which no single consumer card holds, and at 4-bit it drops to roughly 40 GB. Quantisation is how most local LLM hardware punches above its weight. It costs some quality, and the cost varies by model and task.
The part people forget is the KV cache, the attention state held for every token in every active request. For a Llama 3 70B-style architecture (80 layers, 8 key-value heads, 128-dimension heads, 16-bit cache), that works out to roughly 0.3 MB per token. One request at 32k tokens of context therefore needs about 10 GB on top of the weights. Four of those in parallel need about 40 GB. The smaller 8B variant needs around 0.13 MB per token, so an 8k-token request costs roughly 1 GB.
This is why a model that loads fine on your test box falls over with an out-of-memory error the first time three colleagues paste in long documents at once. Size for the context and concurrency you will actually see. Then leave a few gigabytes of headroom for the CUDA runtime and activation memory.
Memory bandwidth decides speed
Generating each new token means reading essentially all of the model’s weights from memory. Single-stream speed is therefore bounded by memory bandwidth divided by model size.
A high-end consumer card has around 1 TB/s of bandwidth. Running a 40 GB quantised model, that gives a theoretical ceiling of about 25 tokens per second per stream before any other overhead. The same card running a 5 GB model is far quicker.
Unified-memory machines turn this trade-off around. They can hold much larger models than a single discrete card, but their bandwidth is typically several times lower, so large models run noticeably slower. They are excellent for development and single-user work, and less suited to serving a busy team.
Batching changes the picture for throughput. A good server processes many requests in one pass over the weights, so total tokens per second across users can be far higher than the single-stream figure.
Hardware options compared
Here is how the realistic options line up for a business deploying in-house rather than building a data centre.
| Setup | Usable memory | Comfortable model size | Good for | Watch out for |
|---|---|---|---|---|
| Single 24 GB consumer card (RTX 3090/4090 class) | 24 GB | 7–14B at 8-bit or 4-bit; ~30B at 4-bit with short context | Small team assistant, classification, extraction, RAG over internal docs | Consumer drivers and warranty terms, 450 W-class power draw, no headroom for long contexts on larger models |
| 48 GB workstation card | 48 GB | 70B at 4-bit with modest context; 30B with generous context | Better quality tier without multi-GPU complexity | Higher cost per GB; KV cache still tight for many long concurrent requests |
| Two or more GPUs with tensor parallelism | 48–192 GB+ | 70B at 8-bit, or higher concurrency on smaller models | Serving a department with real concurrency | Motherboard lanes, PSU sizing, cooling, more complex configuration |
| Unified-memory machine (Mac Studio, DGX Spark class) | 64–128 GB+ shared | Large models fit, including 70B+ quantised | Development, evaluation, single-user or low-concurrency workloads | Lower bandwidth means slower generation; less mature serving stack on some platforms |
| Rented GPU in an EU cloud region (baseline) | Whatever you rent | Any | Testing before buying, burst capacity, fallback | Ongoing cost; data leaves your premises, though it can stay in the EU |
The rest of the box matters more than people expect. You need a power supply with real headroom, because GPUs spike above their rated draw. Airflow must keep the card from thermal throttling. Fast NVMe storage makes loading a 40 GB model a matter of seconds rather than minutes. Enough system RAM matters if you ever offload layers to the CPU.
A single-GPU workstation under sustained load can sit somewhere around 600–800 W. Run 24/7, that is in the region of 15–20 kWh a day. Price it at your actual tariff before you call self-hosting free. These machines are also loud and hot, so they belong in a comms cupboard or rack, not under someone’s desk in an open-plan office. If you want a sense of how this looks as a small permanent setup, we document ours in Inside the D3X Home Lab: Proxmox, DGX Spark and AI Agents.
The software stack
Inference server
There are two broad families. llama.cpp and tools built on it, such as Ollama, are the easiest way to run an LLM locally. They use the GGUF quantised format, run on almost anything including CPUs and Apple silicon, and get you from zero to a working endpoint in minutes. They suit development, single users and light team use.
vLLM and similar servers are built for throughput. They provide continuous batching, paged KV-cache management, tensor parallelism across GPUs and support for formats like AWQ, GPTQ and FP8. If more than a handful of people or an automated pipeline will hit the box concurrently, this is the class of server you want. It is more opinionated about hardware and takes more effort to configure.
Gateway and API shape
Expose the model through an OpenAI-compatible API, which both families support. Your application code then stays portable, and switching between local and hosted models becomes a configuration change rather than a rewrite.
Put a gateway in front of the inference server to handle API keys, per-team rate limits, request logging and routing. Routing matters most: when the local box is down or saturated, the gateway can send traffic to an approved API fallback. LiteLLM is a common choice. Like any component on the request path, it needs patching, and our note on LiteLLM 1.107.0 and what to patch now shows why.
The broader patterns for wiring models into real applications, including retries, timeouts, structured outputs and evaluation, are covered in our LLM integration guide for production systems.
Observability
A GPU box with no metrics is a box you will only hear about when it breaks. At minimum, track time to first token, tokens per second, request queue depth, GPU memory and utilisation, temperature and error rate. vLLM exposes Prometheus metrics, and NVIDIA’s DCGM exporter covers the hardware side. A simple Grafana dashboard plus an alert on queue depth and GPU temperature will catch most problems before users do.
Picking the model and proving it is good enough
Check the licence first. Open-weight models range from permissive Apache 2.0 terms to custom community licences with usage conditions. Read the actual licence file on the model card, not a summary.
Then test quality on your own work, not leaderboards. Build an evaluation set of real examples from the task: 50 to 200 inputs with known-good outputs. Run it against your candidate local models and against a hosted frontier model as a baseline. Test the exact quantised build you intend to deploy, because a 4-bit version can behave differently from the full-precision one. For many extraction, classification and summarisation tasks a well-chosen mid-size model is entirely adequate. For open-ended reasoning, the gap to frontier models can be real.
If the local model lacks knowledge of your domain, retrieval usually fixes that more cheaply than training. Our piece on RAG vs fine-tuning walks through how to choose.
Operations: where self-hosting actually costs you
The purchase is a one-off. The ongoing work is where the real cost of a self-hosted LLM lives.
Driver and CUDA updates are the most common source of self-inflicted outages. A routine OS update pulls in a new kernel, the NVIDIA driver no longer matches, and the inference container fails to start. Pin versions, run the stack in containers with a known-good CUDA base image, and update deliberately.
Model changes need the same discipline. Pin the exact model revision, re-run your evaluation set before swapping, and keep the previous version on disk so rollback is instant.
A single box is a single point of failure. Decide in advance what happens when it dies. The options are an API fallback through the gateway, a degraded mode, or a second machine.
Security is mostly network hygiene. Bind the inference server to an internal interface only. Never expose its port to the internet; Ollama, for instance, has no authentication of its own. Force all access through the authenticated gateway.
Remember that prompt and response logs from an internal assistant will contain personal and commercially sensitive data. Self-hosting keeps that data on your premises, but GDPR obligations on retention, access control and purpose limitation still apply to everything you log. Our AI security and compliance guide for EU firms covers the wider picture.
Finally, someone has to own the box: patching, watching the dashboard and answering when it misbehaves. If nobody’s job description includes that, the deployment will quietly rot.
Before you buy anything, work through this list:
- Write down target model size, typical and maximum context, peak concurrency and latency targets.
- Calculate VRAM for weights plus KV cache at peak concurrency, with headroom.
- Rent a comparable GPU in an EU cloud region for a week and run your real workload on it.
- Build an evaluation set and confirm the quantised local model meets your quality bar against an API baseline.
- Confirm the model licence permits your use.
- Plan the physical location: power, cooling, noise and network.
- Choose the inference server and gateway, and define the fallback path.
- Set up metrics and alerts before go-live, not after the first outage.
- Name an owner and write a runbook for updates, model swaps and recovery.
- Price electricity and staff time alongside hardware when you compare against API costs.
When it is worth it
A private LLM deployment earns its keep when data classification rules out third-party processing, when volume is steady and high enough that owned hardware beats per-token pricing, or when inference has to keep working offline or right next to local systems. It also makes sense as a sandbox where teams can experiment without worrying about what leaves the building.
For low or spiky volume, or tasks that genuinely need frontier-level reasoning, hosted APIs usually win. A hybrid setup is often the honest answer: local models for sensitive or high-volume routine work, and an API for the hard cases. Run the numbers properly, including staff time, using the same approach as our guide to calculating automation ROI.
How D3X Solutions LABS helps
We start with the workload rather than the hardware: agreeing the tasks, measuring token volumes and context lengths, and classifying the data involved.
We then build an evaluation set from your real examples and test candidate open-weight models against a hosted baseline on rented GPUs, so you see any quality gap before spending money on equipment. Those results drive the sizing: VRAM for weights and KV cache at your real concurrency, power and cooling, and whether one box, two cards or a hybrid with an API makes more sense.
When it is time to build, we set up the inference server, an OpenAI-compatible gateway with authentication and fallback routing, metrics and alerting, and a pinned, containerised stack that survives OS updates. We finish with a written runbook covering patching, model upgrades with re-evaluation, and recovery, so your team can own it day to day.
If you are weighing up a self-hosted LLM and want a second opinion on sizing or whether it makes sense at all, book a free 30-minute call using the form below.
Frequently Asked Questions
What hardware do I need to run an LLM locally?
For a capable 7–14B model serving a small team, a single GPU with 24 GB of VRAM is enough, especially with 4-bit or 8-bit quantisation. For 70B-class models you need roughly 48 GB or more, either a workstation card, multiple GPUs or a unified-memory machine. Always size for the KV cache at your real context length and concurrency, not just the model weights.
Can a self-hosted LLM match GPT or Claude quality?
For focused tasks like extraction, classification, summarisation and RAG over internal documents, a well-chosen open-weight model is often good enough. For complex open-ended reasoning and agentic work, frontier hosted models usually still lead. The only reliable way to know is to test the exact quantised model you plan to deploy on an evaluation set built from your own data.
Is a self-hosted LLM automatically GDPR compliant?
No. Keeping inference on your own hardware removes third-party processing, which simplifies things, but GDPR still applies to the data you process and log. Prompt and response logs need retention limits, access controls and a lawful purpose like any other personal data store.
Should I use Ollama or vLLM?
Ollama, built on llama.cpp, is the fastest way to get a model running and suits development, single users and light team use. vLLM is designed for throughput, with continuous batching, efficient KV-cache management and multi-GPU support, so it is the better choice when many users or automated pipelines hit the server concurrently. Both can expose an OpenAI-compatible API, so switching later is manageable.
How much power does a local LLM server use?
A single high-end consumer GPU can draw around 450 W under load, and a complete single-GPU workstation often sits somewhere around 600–800 W when busy. Running continuously, that is roughly 15–20 kWh per day. Multiply by your actual electricity tariff and include it in any cost comparison against API pricing.
Can I self-host an LLM on a Mac?
Yes. Apple silicon Macs with large unified memory can load surprisingly big quantised models using llama.cpp-based tools. The trade-off is lower memory bandwidth than high-end discrete GPUs, so generation is slower and concurrency is limited. They work well for development, evaluation and single-user assistants rather than serving a busy team.
Want this working in your business?
D3X Solutions LABS builds AI agents and automation that pay back fast. Book a free 30-minute call and we'll map where this fits in your business.
Get in touch
