Hardware

NVIDIA A100 40GB

NVIDIA A100 40GB has 40 GB HBM2 and 1555 GB/s. See which open models fit, at which quant, and roughly how fast. Figures below use the default memory (40 GB), 32 GB system RAM, and DDR5-5600 for anything that spills. Updated 2026-10-17.

Memory40 GB HBM2

Bandwidth1555 GB/s

Kindgpu

Year2020

40GB HBM2, 1555 GB/s. Source.

RSS for new models on this device

Models that fit

Model	Verdict	Quant	Context	tok/s
Llama 3.3 70B Instruct 70.6B	Tight fit	IQ4_XS	4K	22–30
DeepSeek R1 Distill Llama 70B 70.6B	Tight fit	IQ4_XS	4K	22–30
Hermes 4 70B 70.6B	Tight fit	IQ4_XS	4K	22–30
Qwen3 Coder Next 79.7B	Tight fit	UD-IQ4_XS	8K	117–159
Qwen2.5 72B Instruct 72.7B	Tight fit	Q3_K_M	4K	22–30
Mistral Medium 3.5 128B 125B	Tight fit	UD-IQ2_XXS	8K	23–31
Devstral 2 123B Instruct 2512 125B	Tight fit	UD-IQ2_XXS	8K	23–31
Qwen3 Next 80B A3B Thinking 79.7B	Tight fit	UD-Q3_K_XL	8K	399–545
Qwen3 Next 80B A3B Instruct 79.7B	Tight fit	UD-Q3_K_XL	8K	398–543
Qwen2.5 32B Instruct 32.8B	Tight fit	Q8_0	8K	23–32
Qwen3 32B 32.8B	Tight fit	Q8_0	8K	23–32
Qwen2.5 Coder 32B Instruct 32.8B	Tight fit	Q8_0	8K	23–32
DeepSeek R1 Distill Qwen 32B 32.8B	Tight fit	Q8_0	8K	23–32
Qwen3 VL 32B Instruct 32.8B	Tight fit	Q8_0	8K	23–32
GLM 4 32B 0414 32.6B	Tight fit	Q8_0	8K	24–33
gemma 4 31B it 30.7B	Tight fit	UD-Q8_K_XL	8K	20–27
Qwen3.5 122B A10B MTP 125B	Tight fit	UD-IQ1_M	8K	259–353
Mistral Small 4 119B 2603 119B	Tight fit	UD-IQ2_XXS	8K	131–179
Qwen3.8 27B 27.3B	Runs	UD-Q8_K_L	8K	30–41
gemma 3 27b it 27.0B	Runs	UD-Q8_K_XL	8K	24–32
medgemma 27b text it 27.0B	Runs	UD-Q8_K_XL	8K	24–32
Qwen3.6 27B 26.9B	Tight fit	UD-Q8_K_XL	8K	24–33
Qwen3.5 27B 26.9B	Tight fit	UD-Q8_K_XL	8K	24–32
Kwaipilot KAT Coder V2.5 Dev 34.7B	Tight fit	Q8_0	8K	125–170
Qwen3.8 35B A3B Distill 35.5B	Tight fit	Q8_0	8K	254–347
GLM 4.5 Air 110B	Tight fit	UD-IQ1_S	4K	96–130
Qwen3.5 35B A3B 34.7B	Tight fit	Q8_0	8K	254–347
Qwen3.6 35B A3B 34.7B	Tight fit	UD-Q8_K_XL	8K	245–334
Llama 4 Scout 17B 16E Instruct 108B	Tight fit	UD-IQ2_XXS	8K	89–122
GLM 4.6V 107B	Tight fit	UD-IQ1_S	8K	91–124
NVIDIA Nemotron 3.5 Lightning 30B A3B 32.9B	Tight fit	UD-Q8_K_XL	8K	216–295
Devstral Small 2 24B Instruct 2512 23.6B	Runs	UD-Q8_K_XL	8K	28–38
Mistral Small 3.2 24B Instruct 2506 23.6B	Runs	UD-Q8_K_XL	8K	28–38
Magistral Small 2509 23.6B	Runs	UD-Q8_K_XL	8K	28–38
Nemotron 3 Nano 30B A3B 31.6B	Runs	Q8_0	8K	236–322
NVIDIA Nemotron 3 Nano Omni 30B A3B Reasoning 31.6B	Tight fit	UD-Q8_K_XL	8K	225–307
GLM 4.7 Flash 29.9B	Tight fit	UD-Q8_K_XL	8K	107–146
Qwen3 Coder 30B A3B Instruct 30.5B	Tight fit	UD-Q8_K_XL	8K	197–269
Qwen3 30B A3B Instruct 2507 30.5B	Tight fit	UD-Q8_K_XL	8K	197–269
Qwen3 VL 30B A3B Instruct 30.5B	Tight fit	UD-Q8_K_XL	8K	197–269

90 more models in the catalog do not fit this default profile, or ranked below the first 40 that do.

Example: Tight fit fully on your NVIDIA A100 40GB at IQ4_XS, 4K context, about 22–30 tok/s. Weights 35.3 GB plus 1.25 GB of KV cache — 37.2 GB of 38.0 GB usable. Weights shown in the verdict use GiB (the same 1024³ counting as GPU memory).