Hardware

NVIDIA A40

NVIDIA A40 has 48 GB GDDR6 and 696 GB/s. See which open models fit, at which quant, and roughly how fast. Figures below use the default memory (48 GB), 32 GB system RAM, and DDR5-5600 for anything that spills. Updated 2026-10-17.

Memory48 GB GDDR6

Bandwidth696 GB/s

Kindgpu

Year2020

48GB GDDR6, 696 GB/s. Source.

RSS for new models on this device

Models that fit

Model	Verdict	Quant	Context	tok/s
Qwen3 Next 80B A3B Instruct 79.7B	Tight fit	UD-Q4_K_XL	8K	151–206
Qwen3 Next 80B A3B Thinking 79.7B	Tight fit	UD-Q4_K_XL	8K	151–206
Qwen3 Coder Next 79.7B	Tight fit	UD-Q4_K_S	8K	44–61
Mistral Small 4 119B 2603 119B	Tight fit	UD-IQ3_S	8K	46–63
Mistral Medium 3.5 128B 125B	Runs	UD-IQ2_XXS	8K	10–14
Devstral 2 123B Instruct 2512 125B	Runs	UD-IQ2_XXS	8K	10–14
Llama 4 Scout 17B 16E Instruct 108B	Tight fit	UD-IQ3_XXS	4K	36–50
Qwen3.5 122B A10B MTP 125B	Tight fit	UD-Q2_K_XL	8K	105–143
Qwen2.5 32B Instruct 32.8B	Runs	Q8_0	8K	10–14
Qwen2.5 Coder 32B Instruct 32.8B	Runs	Q8_0	8K	10–14
DeepSeek R1 Distill Qwen 32B 32.8B	Runs	Q8_0	8K	10–14
GLM 4 32B 0414 32.6B	Runs	UD-Q8_K_XL	8K	9.6–13
GLM 4.6V 107B	Tight fit	UD-Q2_K_XL	4K	36–50
Qwen3.8 27B 27.3B	Runs great	UD-Q8_K_L	8K	13–18
gemma 3 27b it 27.0B	Runs	UD-Q8_K_XL	8K	11–15
medgemma 27b text it 27.0B	Runs	UD-Q8_K_XL	8K	11–15
Qwen3.6 27B 26.9B	Runs	UD-Q8_K_XL	8K	11–15
Qwen3.5 27B 26.9B	Runs	UD-Q8_K_XL	8K	11–14
Kwaipilot KAT Coder V2.5 Dev 34.7B	Runs	Q8_0	8K	56–76
Qwen3.8 35B A3B Distill 35.5B	Runs	Q8_0	8K	114–155
GLM 4.5 Air 110B	Tight fit	UD-IQ2_XXS	8K	36–49
Qwen3.5 35B A3B 34.7B	Runs	Q8_0	8K	114–155
Qwen3.6 35B A3B 34.7B	Runs	UD-Q8_K_XL	8K	110–149
NVIDIA Nemotron 3.5 Lightning 30B A3B 32.9B	Runs	UD-Q8_K_XL	8K	97–132
Devstral Small 2 24B Instruct 2512 23.6B	Runs	UD-Q8_K_XL	8K	13–17
Mistral Small 3.2 24B Instruct 2506 23.6B	Runs	UD-Q8_K_XL	8K	13–17
Magistral Small 2509 23.6B	Runs	UD-Q8_K_XL	8K	13–17
NVIDIA Nemotron 3 Nano Omni 30B A3B Reasoning 31.6B	Runs	UD-Q8_K_XL	8K	101–137
Nemotron 3 Nano 30B A3B 31.6B	Runs	UD-Q8_K_XL	8K	89–122
GLM 4.7 Flash 29.9B	Runs	UD-Q8_K_XL	8K	48–65
Qwen3 Coder 30B A3B Instruct 30.5B	Runs	UD-Q8_K_XL	8K	88–120
Qwen3 30B A3B Instruct 2507 30.5B	Runs	UD-Q8_K_XL	8K	88–120
Qwen3 VL 30B A3B Instruct 30.5B	Runs	UD-Q8_K_XL	8K	88–120
Qwen3 30B A3B Thinking 2507 30.5B	Runs	UD-Q8_K_XL	8K	88–120
gemma 4 26B A4B it 25.2B	Runs great	UD-Q8_K_XL	8K	60–82
LFM2.5 24B A2B 24.0B	Runs great	Q8_0	8K	137–187
GLM 4.7 Flash REAP 23B A3B 23.0B	Tight fit	BF16	8K	59–81
gpt oss 20b 20.9B	Runs great	UD-Q8_K_XL	8K	95–129
Qwen2.5 14B Instruct 14.8B	Runs	F16	8K	12–17
Qwen3 14B 14.8B	Runs	BF16	8K	12–17

90 more models in the catalog do not fit this default profile, or ranked below the first 40 that do.

Example: Tight fit fully on your NVIDIA A40 at UD-Q4_K_XL, 8K context, about 151–206 tok/s. Weights 42.9 GB plus 0.75 GB of KV cache — 44.4 GB of 45.6 GB usable. Weights shown in the verdict use GiB (the same 1024³ counting as GPU memory).