Hosted AI models are the quickest way to get started, but they aren’t always the right home for your data. Client files, health records, financial data and source code often can’t be sent to a third party. Running an open-weight model on your own hardware solves that, and at steady volume it can also cost less than paying per request.
This guide covers what you need to know before buying a GPU or renting a server.
Start with the task, not the model
A local model is a tool for a specific job. Before comparing models, write down:
- The task. Answer questions from internal documents, classify incoming emails, extract fields from invoices, draft replies.
- What good looks like. Twenty to fifty real examples with the answers you’d expect. This becomes your test set.
- Volume and speed. How many requests per hour, and how long a user will wait.
- Constraints. Must it run offline? Which data can it see? Who may use it?
Narrow, well-defined tasks are where smaller local models do best. “Answer anything” is where large hosted models still lead.
How much GPU memory you need
The model’s weights have to fit in GPU memory (VRAM). A useful rule of thumb is the number of parameters multiplied by the bytes stored per parameter:
| Model size | 16-bit | 8-bit | 4-bit |
|---|---|---|---|
| 8B parameters | ~16 GB | ~8 GB | ~5 GB |
| 14B parameters | ~28 GB | ~14 GB | ~8–9 GB |
| 32B parameters | ~64 GB | ~32 GB | ~18–20 GB |
| 70B parameters | ~140 GB | ~70 GB | ~40 GB |
These figures are for the weights alone. The model also needs memory for its working state (the KV cache), which grows with context length and with the number of people using it at once. Leave 20–30% headroom, more if you plan long documents or many concurrent users.
In practice, an 8B model at 4-bit runs comfortably on a single workstation GPU. A 70B model needs data-centre GPUs or several cards working together.
Quantization: a little quality for a lot of memory
Quantization stores weights with fewer bits: 8-bit, 5-bit or 4-bit instead of 16. Memory use drops sharply, and for many business tasks the quality loss is small. It isn’t zero, though, and it varies by model and task, which is why you test on your own examples rather than trusting a leaderboard.
Serving the model
You need software that loads the model and exposes it to your applications. Three common choices:
- Ollama: the easiest way to run a model on one machine, with a simple HTTP API. Good for prototypes and small teams.
- llama.cpp: efficient inference on modest hardware, including CPU-only machines, using quantized GGUF model files.
- vLLM: high-throughput GPU serving with request batching and an OpenAI-compatible API. The usual choice when many users share one model.
A first local test can take minutes:
# Download and chat with a model locally (replace <model> with your choice)
ollama pull <model>
ollama run <model> "Summarize our returns policy in three bullet points."For a shared internal service, vLLM serves an OpenAI-compatible API that most existing tools can talk to:
vllm serve <hugging-face-model-id> --max-model-len 8192
# API available at http://localhost:8000/v1Fine-tuning with LoRA
Often a good base model plus your documents (retrieval) is enough. When it isn’t, because you need a particular tone, format or domain vocabulary, fine-tuning adapts the model itself.
LoRA trains small adapter matrices instead of every weight in the model, so it needs far less memory and time than full fine-tuning. QLoRA goes further by loading the base model in 4-bit while training the adapters, which makes fine-tuning larger models possible on a single GPU.
What matters most is the data:
- A few hundred to a few thousand high-quality examples of the input and the output you want.
- Consistent formatting, with no contradictions.
- A held-out test set the model never sees during training.
Evaluate before you trust it
Run the same test set through the base model, your fine-tuned model and, if allowed, a hosted model. Compare accuracy, response time and cost per request. Repeat the evaluation whenever you change the model, the data or the prompt. Without numbers, every change is a guess.
Treat the model server like any other internal service
- Don’t expose it to the internet. Put it behind authentication and a reverse proxy.
- Log who called it and when.
- Patch the host operating system, GPU drivers and serving software.
- Check the model’s licence terms for commercial use.
A sensible first project
- Pick one narrow task with clear right and wrong answers.
- Collect 50 real examples and expected outputs.
- Test two or three open-weight models with Ollama or llama.cpp.
- If the best one is close but not good enough, add retrieval or fine-tune with LoRA.
- Deploy behind authentication, measure, and expand from there.
If you want help sizing hardware or running that first evaluation, get in touch. It’s the kind of project we enjoy.