Ollama Hardware: VRAM & Quantization

Part of a series on running AI workflows across model providers — this one turns model choice into a hardware question you can answer.

The previous article treated model size and quantization as the two levers you pull when choosing what to run. This one goes one level deeper into the hardware side of that equation, because "will this fit" is a more specific, more answerable question than it first appears.

VRAM is the number that matters most

For GPU-accelerated inference — the way Ollama runs fastest — the deciding constraint is usually GPU memory (VRAM), not general system RAM. A model has to fit, largely or entirely, in VRAM to run at a usable speed; when it doesn't, Ollama can fall back to system RAM or a mix of both, but at a real cost to speed. Roughly speaking, more VRAM opens up larger models or less aggressive quantization at usable speeds; less VRAM pushes you toward smaller models, heavier quantization, or both.

The honest starting question for any local setup isn't "which model" — it's "how much VRAM do I actually have."

This is a genuinely different planning exercise than anything the cloud-provider articles in this series required. Azure's deployment types, Bedrock's provisioned throughput, and every other cloud pricing model let you pay your way to more capacity. Ollama's hardware ceiling doesn't move until you change the hardware.

What quantization actually costs you

The previous article introduced quantization as a size-and-speed lever. The honest version of that tradeoff is a curve, not a cliff: going from full precision down to a moderate quantization level typically costs very little measurable quality for a real gain in size and speed, while pushing quantization very aggressively starts to cost more quality for each further step down. Where exactly that curve bends depends on the specific model and task — which is why testing against your actual use case, not a generic benchmark, is the only reliable way to know where your acceptable tradeoff point is.

Moderate quantization is usually close to free. Aggressive quantization is a real, task-dependent bet.

CPU-only is a real option, with real limits

Not every deployment has a GPU available — a box provisioned purely for CPU inference is a legitimate setup, particularly for lower-traffic internal tools where latency tolerance is higher. CPU inference is meaningfully slower than GPU-accelerated inference for the same model, which makes model size and quantization choices even more consequential in a CPU-only setup: a model that runs comfortably fast on a modest GPU can be genuinely unusable at CPU speeds if it's too large.

A use case that shows the planning payoff

A team standing up an internal Ollama-backed tool for after-hours log summarization initially provisioned a GPU instance sized for their largest candidate model at full precision, based on a rough estimate rather than actual testing. Running the same task through DNotifier's Prompt Testing Studio against a smaller model at moderate quantization on a considerably cheaper GPU tier showed indistinguishable output quality for their specific log format — the over-provisioned instance was solving a problem that testing showed didn't actually exist. Downsizing to the smaller instance freed budget for a second box running a different model for an unrelated pipeline stage.

The practical takeaway

Hardware planning for Ollama comes down to three honest questions in order: what does the task actually need in terms of model capability, what's the smallest and most quantized version of a model that clears that bar, and what's the cheapest hardware tier that runs that specific combination at a speed the task tolerates. Skipping straight to "buy the biggest GPU available" tends to mean paying for headroom the task never uses.

Frequently asked questions

Does more system RAM help if I don't have enough VRAM?

Ollama can offload to system RAM when a model doesn't fully fit in VRAM, but this comes with a real speed penalty — it's a fallback, not a substitute for adequate VRAM for latency-sensitive use.

Is there a fixed VRAM number I need for a "good" local setup?

No single number applies universally — it depends entirely on which model size and quantization level you're targeting, which is why testing against your actual candidate models matters more than following a generic hardware rule of thumb.

Should I always buy more hardware than I currently need?

Not by default — the same over-provisioning risk this series has flagged for cloud deployments (Azure's Provisioned Throughput, Bedrock's reserved capacity) applies here too. Test against real, representative tasks before sizing hardware up.

Does Apple Silicon (M-series) hardware work well for local models?

Ollama runs on macOS including Apple Silicon, and unified memory architectures can be a genuinely efficient fit for certain model sizes — specifics depend on your exact hardware generation and the model in question.

The Bottom Line

Next in this series: when running locally is actually the better call, and when it honestly isn't.

Read part five: When Local Beats Cloud (And When It Doesn't). Explore dnotifier.com.