Method
How it works
Every number is an estimate for llama.cpp, not a benchmark. Open a section to see the picture, then the short explanation.
Memory: weights, conversation, and a little spare room
Weights
The GGUF file. A smaller quant is a smaller file, and a slightly less faithful model.
KV cache
The conversation’s memory. A model that fits at 8K can miss at 128K.
Overhead
llama.cpp needs a little extra. A card driving a monitor cannot give you every byte.
Total memory is weights + KV cache + engine overhead. Real GGUF file sizes win. A file smaller than about 40% of parameters × bits-per-weight / 8 is treated as a stray shard and ignored. Q4_K_M is about 4.85 bits per weight.
Ordinary KV cache, in FP16, is 2 × layers × KV heads × head dim × 2 × context. Sliding-window layers only keep the window. DeepSeek-style MLA stores a short latent. Linear and Mamba layers keep a fixed state.
llama.cpp overhead is 0.3 GB plus 1% of the weights. A discrete GPU that drives a monitor loses 0.5 GB, then 5% more. Apple unified memory, and parts like Ryzen AI Max, budget 75% of RAM for the GPU and always leave about 5 GB for the OS. CPU-only runs keep 4 GB plus 10%.
The verdict
We pick the best quant that still leaves room for a normal chat. Runs great is at or under 60% of the usable pool, Runs is 85%, and Tight fit is 98%. If the whole file will not fit, some layers can sit in system RAM. That is slower, and the page says so.
The home page leads with models that fit fully at about 10 tokens per second or faster, largest first, with a discount when the only quant that fits is well below Q4. The three cards are the best general-chat, coding, and reasoning matches.
The speed
Each token rereads the weights. A wider memory bus finishes that read sooner. The range is 55–75% of the card’s peak, not a benchmark. Treat “20–30 tok/s” as a ceiling, not a promise. Prefill, speculative decoding, KV-cache quantization, and multi-GPU are not modeled.
Where the numbers come from
Model sizes and configs come from the public Hugging Face API. Hardware is a hand-checked table, one source link per row, including NVIDIA’s compare page and Apple’s specs. Ollama commands use the public library name when it matches. A job refreshes the catalog every six hours and drops fine-tunes and duplicate variants.
Nothing here is fitted to llama-bench yet. A speed you type on the home page stays in this browser.