It's tempting to compare local language models using one number: tokens per second.

But a model that generates text quickly isn't necessarily useful for coding, automation, retrieval, or tool calling.

Our local LLM measurements illustrate why evaluating the full workload matters as much as measuring speed.

The fastest model didn't pass the most checks

Agent Infra Guide measured Qwen2.5-Coder models using Ollama on an Apple M4 Pro with 24 GiB of unified memory.

The October 19, 2026 coding comparison used Q4_K_M quantization, a configured context of 4,096 tokens, and one concurrent client.

The smallest model generated tokens much faster, but it passed fewer checks in this particular test.

These counts represent repeated executions of a small deterministic coding task set, not independent assessments of general coding performance.

The takeaway isn't that one model is universally better. It's that generation speed and task success measure different things.

Throughput is only one dimension

Decode throughput measures how quickly output tokens are generated under a particular configuration.

It does not, on its own, tell you:

• Did the model satisfy the task's success criteria?
• Did it follow the required tool-call format?
• Did the task finish within its time or output-token limits?
• How long did the first response take?
• How much memory did the process consume?
• Were outcomes consistent across repetitions?

For an AI agent, these measurements help determine whether a model is useful for the intended application.

A fast response that fails the task is not necessarily a useful response.

Keep the experiment conditions visible

The Agent Infra Guide benchmark collection contains 15 measured configurations and 75 measured repetitions on a shared Apple M4 Pro desktop.

The machine has a 12-core CPU, 16-core GPU, and 24 GiB of unified memory, running macOS 15.5.

Individual measurements used different Ollama versions, including 0.35.0 and 0.40.3. The exact version is recorded with each run.

This is not a dedicated benchmarking server.

Results reflect the particular machine, workload, model artifact, runtime version, and experimental conditions.

They should not be interpreted as a universal ranking across hardware.

Memory measurements require context

The benchmark reports sampled serving-process resident set size (RSS).

RSS is not the same as dedicated GPU VRAM. Apple silicon uses unified memory, shared between CPU and GPU workloads.

Process RSS also does not independently identify the memory occupied by model weights, Metal allocations, KV cache, or compute buffers.

The measurements sample RSS every 250 milliseconds. A brief memory peak could occur between samples.

The report includes a separate physical-footprint measurement, but that metric also has limitations.

These figures are useful observations, not definitive minimum-memory requirements.

Before selecting hardware, combine a transparent memory estimate with measurements from your actual workload.

Why we retain failures and retract unsupported numbers

A trustworthy benchmark should show unsuccessful results, not just the most impressive measurements.

Our public benchmark report also documents a withdrawn historical claim.

An earlier Mac mini M4 result stated approximately 620 tokens per second for a Qwen2.5 7B configuration.

That claim lacked supporting raw measurement evidence and conflicted with the report's approximate memory-bandwidth sanity check.

Rather than replacing it with an invented number, the claim was retracted on September 17, 2026.

The correct historical result is unknown.

This is an important distinction between presenting impressive numbers and maintaining an auditable measurement record.

A better evaluation checklist

When comparing local LLM deployments, record:

1. Hardware, operating system, and background workloads
2. Exact model artifact and quantization
3. Runtime implementation and version
4. Configured context length and concurrency
5. Decode speed and the measurement window
6. Time to first token and prompt-processing performance, where measured
7. Memory metric, sampling method, and known limitations
8. Task success criteria, failures, and repeated-run variation

Not every benchmark measures every item. Record those omissions rather than treating unavailable measurements as known quantities.

In particular, short-prompt time-to-first-token measurements cannot establish performance on long agent prompts or large tool outputs.

Inspect the original measurements

The Agent Infra Guide benchmark page includes measurement records, workload outcomes, methodology notes, and documented limitations.

Explore the measured local LLM benchmarks and methodology

The goal isn't to declare one model the winner.

It's to help you decide which hardware and configuration can handle your actual workload.