What local models actually cost you, measured on two consumer GPUs
Every "run AI locally" article quotes someone else's benchmark, usually on a datacentre card you will never own. I wanted numbers from hardware a person can actually buy, on the prompts a person actually types. So I built a bench harness, pulled six open models from 0.5B to 7.6B parameters, and measured them on the machine under my desk.
The results are on the benchmarks page, regenerated from the JSON the harness writes. This is what the numbers mean and what surprised me.
The machine, so you can discount it
An RTX 3070 (8GB) and a GTX 1660 SUPER (6GB), 8 CPU cores, 31GB of RAM, running Ollama 0.40.6. Two GPUs of different generations is not a clean setup — Ollama picks one, and it is not always the one you would pick — but it is a realistic one. Plenty of people have an old card and a newer one.
Everything was measured with every other model unloaded first. My first run was not, and it produced a result that would have been actively misleading: mistral:7b measured 3.98 tokens per second because a previous model's process was still holding the CPU. Re-measured on a clean GPU, it hit 64. The harness now unloads the runtime and refuses to start above a load average of 3, because a benchmark that silently measures contention is worse than no benchmark.
Throughput: size dominates, as expected, until it does not
Decode speed in tokens per second, median of three runs:
| Model | Params | VRAM | Short answer | Write code | Reasoning |
|---|---|---|---|---|---|
| qwen2.5:0.5b | 494M | 579MB | 261 | 261 | 250 |
| llama3.2:1b | 1.2B | 1.7GB | 146 | 140 | 211 |
| qwen2.5:1.5b | 1.5B | 1.3GB | 154 | 151 | 216 |
| llama3.2:3b | 3.2B | 3.0GB | 101 | 100 | 201 |
| qwen2.5:7b | 7.6B | 5.7GB | 61 | 61 | 69 |
| mistral:7b | 7.2B | 6.1GB | 64 | 64 | 58 |
Roughly a 4× spread from the smallest to the largest. The scaling is not linear in parameters — a 7B model is about 15× the size of a 0.5B model but runs about 4× slower — because decode is dominated by memory bandwidth, and these are all small enough to fit in GPU memory.
Two things worth noting. The "Reasoning" column is faster than the other two for every small model, which is not a real effect: those models answered the arithmetic question in two or three tokens, so there is less to decode. It measures verbosity, not thinking speed. And 61 versus 64 tokens per second between two 7B models is inside the noise of a single seating — do not read a winner from it.
Memory is where the real decision lives
| Model | VRAM held |
|---|---|
| qwen2.5:0.5b | 579MB |
| qwen2.5:1.5b | 1.3GB |
| llama3.2:1b | 1.7GB |
| llama3.2:3b | 3.0GB |
| qwen2.5:7b | 5.7GB |
| mistral:7b | 6.1GB |
An 8GB card runs a 7B model with room for a browser. A 6GB card runs a 3B comfortably and a 7B by spilling. Below that, you are choosing between a small model and no model — and for a lot of jobs, a 1–3B model that fits is worth more than a 7B that thrashes.
Note that VRAM does not track parameter count neatly: llama3.2:1b at 1.7GB uses more than qwen2.5:1.5b at 1.3GB, because of quantization and context defaults. Pull the tag, measure the footprint, do not assume from the name.
The number that matters most is the one nobody publishes
Cold start — the wait before the first token when the model is not already in memory:
| Model | Cold start |
|---|---|
| llama3.2:1b | 42s |
| qwen2.5:1.5b | 52s |
| llama3.2:3b | 55s |
| qwen2.5:0.5b | 63s |
| mistral:7b | 75s |
| qwen2.5:7b | 104s |
A minute of waiting before the first word. That is the actual first-run experience of local AI, and it is why "it works on my machine" demos often do not survive contact with a fresh session.
It is also a one-time cost, and that distinction is everything. A warm follow-up request on the same model answered in 56 milliseconds end to end. So the design question is not "how fast is the model" but "how do I keep it warm" — a long keep_alive, a process that holds the model, or an app that loads once and stays open. My figures are pessimistic here too: the disk was 95% full during the run, which slows loading. On a machine with free space, expect meaningfully less.
Speed means nothing if the answer is wrong
I asked each model one arithmetic question with a checkable answer: three items cost 45 rupees, what do eight cost?
The correct answer is 120. Four of the six got it wrong. qwen2.5:0.5b said 360. llama3.2:3b said 135. The two 7B models both got it right.
All six correctly wrote a working dedupe function in JavaScript. So on this evidence: these models are good at pattern-shaped code generation and unreliable at even trivial arithmetic. That is the exact failure mode that makes people distrust local AI, and it is why the bench records correctness next to speed. A model that produces 260 tokens per second of wrong answer is not fast, it is confidently wrong quickly.
What I would actually run
- 0.5B–1.5B on a small box: for classification, extraction, routing, and summarising short text. Fast enough to feel instant, small enough to stay resident. Do not ask it to reason.
- 3B on a 6–8GB card: the sweet spot for a general assistant you keep loaded. Good enough at code, still wrong at arithmetic.
- 7B on an 8GB card: the smallest size that passed the reasoning check. If correctness matters more than latency, this is the floor — and it is the point where a hosted model starts to look genuinely competitive on cost.
The honest summary: local AI on consumer hardware is real, fast once warm, and less capable than the marketing suggests at exactly the tasks people most want to delegate. Measure your own box before you commit an evening to it — the harness is in the repo, and the benchmarks page prints the method, the prompts, and every limitation I could think of, because a number without those is a claim, not a measurement.