Choosing a Model to Run Locally

Part of a series on running AI workflows across model providers — this one is about hardware-bounded model choice, not catalog browsing.

Every cloud provider in this series hands you a menu of models and lets their infrastructure worry about what it takes to run them. Ollama flips that entirely: the model you can use is bounded by the hardware sitting in front of you, which makes model selection a genuinely different exercise than it's been for any other provider in this batch.

Parameter count sets the floor

Models are commonly described by parameter count — roughly, how many adjustable values the model has learned, which correlates loosely with capability and much more directly with how much memory it needs to run. Smaller models (in the few-billion-parameter range) run comfortably on modest hardware, including many laptops, and handle straightforward tasks well: classification, simple extraction, drafting short responses. Larger models generally reason better and handle more nuanced tasks, at the direct cost of needing more capable — and more expensive — hardware to run at a usable speed.

The question isn't "which model is best" — it's "which model actually fits what I have."

Quantization: trading precision for size

This is where local model selection gets a second, less obvious axis. Models are typically distributed in a compressed format called GGUF, at different quantization levels — commonly labeled things like Q8, Q4, and lower. Full precision keeps every bit of the model's learned weights; quantization reduces that precision to shrink the model's memory footprint and speed up inference, at some cost to output quality. A Q4 quantization of a given model is dramatically smaller and faster than its full-precision version, and for a meaningful share of everyday tasks, close enough in quality that the difference isn't the deciding factor.

Quantization is why a model that "shouldn't" fit on your hardware often does — at a real, but often acceptable, quality cost.

This means the actual decision isn't just "which model family" the way it was for Azure's catalog in the earlier batch — it's "which model family, at which quantization level, does my specific hardware handle at a speed I can live with." A larger model at aggressive quantization and a smaller model at full precision can end up genuinely competitive for a given task, and the only reliable way to know which wins for your use case is testing both.

Matching model to task, not task to model

The practical approach mirrors the deployment-selection advice from the Azure batch: start from what the task actually needs rather than defaulting to the biggest model that fits. A classification or short-extraction task rarely benefits from a large model's additional reasoning depth — a smaller, faster, more heavily quantized model often does the job at a fraction of the latency, leaving hardware headroom for tasks that genuinely need it.

A use case that shows this in practice

A team building a local document-tagging tool initially defaulted to the largest model their workstation's GPU could technically fit, reasoning that bigger meant better tagging accuracy. Testing a meaningfully smaller model at a moderate quantization level against the same tagging task showed no measurable accuracy difference for their specific document types — the classification task simply didn't need the larger model's additional capacity. Switching freed up enough memory headroom to run two models concurrently for different pipeline stages, which the larger model's footprint hadn't allowed.

Testing before committing, same as every other provider

DNotifier's Prompt Testing Studio works the same way here as it does for every cloud provider in this series: point it at different Ollama-hosted models — different sizes, different quantization levels — with representative prompts from the actual task, and compare real output quality against real latency before committing.

Frequently asked questions

Is a smaller quantized model always worse than a larger one at full precision?

Not always in practice — for many everyday tasks, the difference is small enough that the smaller, faster option is the better tradeoff. It depends heavily on the specific task's complexity.

What quantization level should I start with?

There's no universal answer verifiable from documentation alone — a moderate quantization level (commonly Q4-class) is a reasonable starting point to test against, then adjust based on real results.

Does model choice affect the DNotifier setup at all?

No — switching which model you run through Ollama is a matter of pulling a different model and updating the model name in your sendAI() call; the DNotifier-side configuration doesn't change.

Can I run multiple models on the same Ollama instance?

Yes, hardware permitting — ollama list shows everything currently pulled, and your code selects which one to call per request.

The Bottom Line

Next in this series: a closer look at the hardware side of this equation — what VRAM tiers actually mean for what you can run.

Read part four: Ollama Hardware: VRAM & Quantization. Explore dnotifier.com.