Local model benchmarks
How fast each model actually runs on one specific machine, with the prompts, the method, and the spread printed alongside the numbers. These are my measurements, not a leaderboard copied from somewhere else.
Throughput, median of 3 runs
| Model | Short answer tok/s | Write code tok/s | Reasoning tok/s | Cold start seconds | VRAM |
|---|---|---|---|---|---|
| mistral:7b7.2B · Q4_K_M | 64.244 tok | 63.5ok | 57.9ok | 74.9s | 6,146MB |
| qwen2.5:7b7.6B · Q4_K_M | 60.923 tok | 60.9ok | 68.9ok | 104.2s | 5,667MB |
| llama3.2:3b3.2B · Q4_K_M | 101.138 tok | 100.2ok | 201.5no | 54.8s | 3,046MB |
| qwen2.5:1.5b1.5B · Q4_K_M | 154.215 tok | 151.2ok | 215.8no | 52.1s | 1,334MB |
| llama3.2:1b1.2B · Q8_0 | 146.136 tok | 140.0ok | 211.2no | 42.2s | 1,684MB |
| qwen2.5:0.5b494.03M · Q4_K_M | 261.136 tok | 261.4ok | 249.9no | 62.9s | 579MB |
Higher tok/s is faster. ok means the answer matched what the prompt required; no means it did not. Speed without a correct answer is not a result.
What was measured
Definitions
ttft_ms— request to first token, as load_duration + prompt_eval_duration. The model is warmed once first, so this is a warm start unless noted in cold_start_ms.tok_s— decode throughput: eval_count / eval_duration, reported by the runtime. Median of the runs.vram_mb— total GPU memory in use while the model is resident, from nvidia-smi. Includes other processes, so treat it as an upper bound.cold_start_ms— the first request after the model is not in memory: model load plus first-token latency. This is paid once per session, not per request — a warm follow-up on the same model answers in tens of milliseconds.
Prompts used
Exact text, unedited
- Short answer — “In one sentence, what does a web server do?”
- Write code — “Write a JavaScript function called dedupe that removes duplicate values from an array while preserving order. Return only the code, no explanation.”
- Reasoning — “A shop sells 3 items for 45 rupees. At that rate, how much do 8 items cost? Answer with the number only.”
What this does not tell you
Limits
- Measurements were taken with nothing else running on the machine; the load average at the start of the run is recorded in load_average_at_start. A busier box will be slower.
- One machine, one GPU generation, one seating. These are not general rankings — a different box will produce different numbers.
- Each prompt runs three times and the median is kept, but the sample is small. Read tok_s_min and tok_s_max for the spread.
- The reasoning prompt is one arithmetic question. It is a smoke test for whether the model can follow instructions, not a benchmark of reasoning.
- Quantization is whatever the pulled tag ships with. A different quant of the same model will differ.
- Cold-start figures include model loading from disk and were taken while the machine's disk was about 95% full, which makes them pessimistic. Warm throughput is unaffected by disk state.
Reproduce it: node bench/llm-bench.mjs --label "your machine" writes the same JSON from your own box.