bartowski
Qwen2.5 72B Instruct
Qwen2.5 72B Instruct is a 72.7B dense qwen2 model from bartowski. Configured context length is 32,768 tokens. License: other. Quant sizes are the GGUF file bytes published on Hugging Face when a file exists.
Parameters72.7B
Context32K
Attentiongqa
Licenseother
80 layers · 8 KV heads · head dim 128 · GGUF repo · config
Quant file sizes
| Quant | File size | Source |
|---|---|---|
| Q8_0 | 72.0 GB | Hugging Face file bytes |
| Q6_K | 59.9 GB | Hugging Face file bytes |
| Q5_K_M | 50.7 GB | Hugging Face file bytes |
| Q4_K_M | 44.2 GB | Hugging Face file bytes |
| IQ4_XS | 37.0 GB | Hugging Face file bytes |
| Q4_0 | 38.5 GB | Hugging Face file bytes |
| Q3_K_L | 36.8 GB | Hugging Face file bytes |
| Q3_K_M | 35.1 GB | Hugging Face file bytes |
| IQ3_M | 33.1 GB | Hugging Face file bytes |
| Q3_K_S | 32.1 GB | Hugging Face file bytes |
| IQ3_XXS | 29.7 GB | Hugging Face file bytes |
| Q2_K | 56.7 GB | Hugging Face file bytes |
| IQ2_XXS | 23.7 GB | Hugging Face file bytes |
| IQ1_M | 22.1 GB | Hugging Face file bytes |
On common machines
| Device | Verdict | Quant | Speed |
|---|---|---|---|
| GeForce RTX 4060 Ti 8GB | Partial offload | IQ3_XXS | 1.7–2.4 |
| GeForce RTX 3060 12GB | Partial offload | IQ3_M | 1.7–2.4 |
| GeForce RTX 4060 Ti 16GB | Partial offload | IQ4_XS | 1.6–2.2 |
| GeForce RTX 4090 | Partial offload | Q4_K_M | 1.8–2.4 |
| GeForce RTX 5090 | Tight fit | IQ2_XXS | 35–48 |
| Apple M4 Pro | Won't fit | IQ1_M | — |
| Apple M5 | Won't fit | IQ1_M | — |
| CPU only · DDR5-5600 | Runs on CPU | IQ1_M | 2.0–2.7 |
Smallest devices that fit it fully
- GeForce RTX 5090 — Tight fit, IQ2_XXS, 35–48 tok/s
- RTX 5000 Ada — Tight fit, IQ2_XXS, 11–15 tok/s
- Tesla V100 32GB — Tight fit, IQ2_XXS, 18–24 tok/s
- Apple M1 Max — Tight fit, IQ1_M, 9.0–12 tok/s
- Apple M2 Max — Tight fit, IQ1_M, 9.0–12 tok/s
- Apple M3 Max (30-core GPU) — Tight fit, IQ2_XXS, 6.1–8.4 tok/s
Run it
Commands for the GeForce RTX 4060 Ti 16GB estimate (IQ4_XS).
llama.cpp
llama-server -hf bartowski/Qwen2.5-72B-Instruct-GGUF:IQ4_XS -c 4096 -ngl 28
Ollama
ollama run qwen2.5
LM Studio
lms get bartowski/Qwen2.5-72B-Instruct-GGUF # Then load quant IQ4_XS