unsloth
Llama 3.3 70B Instruct
Llama 3.3 70B Instruct is a 70.6B dense llama model from unsloth. Configured context length is 131,072 tokens. License: llama3.3. Quant sizes are the GGUF file bytes published on Hugging Face when a file exists.
Parameters70.6B
Context128K
Attentiongqa
Licensellama3.3
80 layers · 8 KV heads · head dim 128 · GGUF repo · config
Quant file sizes
| Quant | File size | Source |
|---|---|---|
| BF16 | 131.4 GB | Hugging Face file bytes |
| F16 | 131.4 GB | Hugging Face file bytes |
| UD-Q8_K_XL | 75.7 GB | Hugging Face file bytes |
| Q8_0 | 69.8 GB | Hugging Face file bytes |
| UD-Q6_K_XL | 57.0 GB | Hugging Face file bytes |
| Q6_K | 107.8 GB | Hugging Face file bytes |
| UD-Q5_K_XL | 46.4 GB | Hugging Face file bytes |
| Q5_K_M | 46.5 GB | Hugging Face file bytes |
| Q5_K_S | 45.3 GB | Hugging Face file bytes |
| UD-Q4_K_XL | 39.7 GB | Hugging Face file bytes |
| Q4_K_M | 39.6 GB | Hugging Face file bytes |
| Q4_K_S | 37.6 GB | Hugging Face file bytes |
| IQ4_NL | 37.3 GB | Hugging Face file bytes |
| IQ4_XS | 35.3 GB | Hugging Face file bytes |
| Q4_1 | 41.3 GB | Hugging Face file bytes |
| Q4_0 | 37.4 GB | Hugging Face file bytes |
| UD-Q3_K_XL | 32.4 GB | Hugging Face file bytes |
| Q3_K_M | 31.9 GB | Hugging Face file bytes |
| Q3_K_S | 28.8 GB | Hugging Face file bytes |
| UD-IQ3_XXS | 25.8 GB | Hugging Face file bytes |
| UD-Q2_K_XL | 25.1 GB | Hugging Face file bytes |
| Q2_K | 49.4 GB | Hugging Face file bytes |
| UD-IQ2_XXS | 18.1 GB | Hugging Face file bytes |
| UD-IQ1_M | 16.0 GB | Hugging Face file bytes |
| UD-IQ1_S | 14.8 GB | Hugging Face file bytes |
On common machines
| Device | Verdict | Quant | Speed |
|---|---|---|---|
| GeForce RTX 4060 Ti 8GB | Partial offload | Q3_K_S | 1.8–2.5 |
| GeForce RTX 3060 12GB | Partial offload | UD-Q3_K_XL | 1.8–2.4 |
| GeForce RTX 4060 Ti 16GB | Partial offload | Q4_K_S | 1.6–2.1 |
| GeForce RTX 4090 | Tight fit | UD-IQ2_XXS | 25–34 |
| GeForce RTX 5090 | Tight fit | UD-IQ3_XXS | 32–44 |
| Apple M4 Pro | Tight fit | UD-IQ1_S | 8.7–12 |
| Apple M5 | Won't fit | UD-IQ1_S | — |
| CPU only · DDR5-5600 | Runs on CPU | UD-IQ2_XXS | 2.2–3.0 |
Smallest devices that fit it fully
- RTX 4000 Ada — Tight fit, UD-IQ1_M, 11–15 tok/s
- Radeon RX 7900 XT — Tight fit, UD-IQ1_M, 24–32 tok/s
- GeForce RTX 4090 — Tight fit, UD-IQ2_XXS, 25–34 tok/s
- GeForce RTX 3090 Ti — Tight fit, UD-IQ2_XXS, 25–34 tok/s
- GeForce RTX 3090 — Tight fit, UD-IQ2_XXS, 23–32 tok/s
- NVIDIA L4 — Tight fit, UD-IQ2_XXS, 7.5–10 tok/s
Run it
Commands for the GeForce RTX 4060 Ti 16GB estimate (Q4_K_S).
llama.cpp
llama-server -hf unsloth/Llama-3.3-70B-Instruct-GGUF:Q4_K_S -c 4096 -ngl 27
No matching name in the Ollama library snapshot.
LM Studio
lms get unsloth/Llama-3.3-70B-Instruct-GGUF # Then load quant Q4_K_S