Run AI on your own box: a practical map of local models

There are two ways to use AI tools today. The default one is somebody else's: you open a chat window, your words leave your building, and the bill arrives as a subscription or a per-token meter you don't control. The other way is older and less fashionable: run the model on hardware you own. I run a mix of both, and this is the practical map I wish someone had handed me — what you need, what it costs, where local genuinely wins, and where you'd be punishing yourself by refusing the cloud.

Why run AI locally at all

Three honest reasons, in the order I weigh them.

Privacy and control. Anything you type into a hosted model crosses a boundary you don't administer. For personal notes, client documents, financials, anything covered by an NDA — that boundary matters. A local model never sees your words leave the room, and you are the access policy.

Cost shape. Hosted AI is a meter. Local AI is a machine you already own plus electricity. If you use AI constantly — and if you're building with agents, you do — the meter never sleeps. The box doesn't care how many tokens you push through it at 2 a.m.

It teaches you the thing. This is the underrated one. Running your own models forces you to understand what a "7B model" actually is, what context windows cost, why responses slow down — and that understanding transfers to every hosted tool you'll ever evaluate. You stop being someone AI happens to.

What you actually need

The honest rule of thumb: it's about memory, not magic. Model sizes are measured in parameters (7B = 7 billion), and a compressed ("quantized") model needs roughly its size in gigabytes of RAM to run comfortably. So:

• 8 GB machine — small models (3–8B, 4-bit quantized) run fine; 7–8B is the ceiling. Great for drafting, summarizing, classifying. Slow on old hardware but usable.
• 16 GB machine — comfortable 7–14B territory. This is the sweet spot for most people: modern small models at this size are genuinely useful for everyday work.
• A GPU with 8–24 GB VRAM, or an Apple Silicon Mac — this is where local AI stops feeling like a compromise. An M-series MacBook with 16 GB+ of unified memory is arguably the best consumer machine for local AI.

You don't need a server rack. You need one decent machine and an evening.

The on-ramp: Ollama

The tool that made this all accessible is Ollama — one installer, then ollama run llama3 and you're talking to a model on your own machine. It serves an OpenAI-compatible API on your localhost, which means everything you already know about wiring up AI tools — chat frontends, automation, agents — points at your own box by changing one URL.

That last sentence is the whole trick. The tools ecosystem (chat UIs, RAG systems, agent builders) speaks a standard protocol. The cloud speaks it. Your machine speaks it. Which tool you should pick for each layer, I cover in the head-to-heads guide — the local-vs-hosted choice is just one more job to decide first.

Where local wins, and where it doesn't

Local wins when: the data is sensitive; the task is repetitive and well-covered by small models (summarize, extract, rewrite, classify); you're calling it from automation all day; or you're offline — on a plane, in a factory, in a power cut.

The cloud still wins when: you need frontier-level reasoning; the task is rare and one-off (installing gigabytes to summarize one PDF is the "dashboard you open twice a year" trap); or latency and polish are the product. I pay for hosted models. I just won't let them be the only door.

The point was never "local always." The point is that the default should be a decision, not a sign-up you sleepwalked into.

The map, compressed

1. Install Ollama on your best machine. Run a 7–8B model. Talk to it about your actual work. 2. Point one automation at it — if you run n8n or an agent pipeline, swap one hosted call for localhost. Watch the meter flatline. 3. Notice what breaks. Small models are worse at long reasoning and sprawling context — that's the boundary of the map, and knowing it makes you better at choosing hosted tools when they're worth it. 4. Only then buy hardware, if the meter says you should.

If this starts with the question of whether to own the stack at all, and ends with the tools that run the work — and if the models start talking to each other, that's teams of software agents.