Hardware

NVIDIA L4

NVIDIA L4 has 24 GB GDDR6 and 300 GB/s. See which open models fit, at which quant, and roughly how fast. Figures below use the default memory (24 GB), 32 GB system RAM, and DDR5-5600 for anything that spills. Updated 2026-10-17.

Memory24 GB GDDR6

Bandwidth300 GB/s

Kindgpu

Year2023

24GB GDDR6, 300 GB/s. Source.

RSS for new models on this device

Models that fit

Model	Verdict	Quant	Context	tok/s
Qwen3.5 35B A3B 34.7B	Tight fit	UD-Q4_K_XL	8K	79–108
Qwen3.6 35B A3B 34.7B	Tight fit	UD-Q4_K_XL	8K	78–107
GLM 4.7 Flash 29.9B	Tight fit	UD-Q5_K_XL	8K	33–45
Kwaipilot KAT Coder V2.5 Dev 34.7B	Tight fit	Q4_K_M	8K	41–56
Qwen3.8 35B A3B Distill 35.5B	Tight fit	Q4_K_M	8K	82–112
Qwen3 Coder 30B A3B Instruct 30.5B	Tight fit	UD-Q5_K_XL	8K	56–76
Qwen3 30B A3B Instruct 2507 30.5B	Tight fit	UD-Q5_K_XL	8K	56–76
Qwen3 VL 30B A3B Instruct 30.5B	Tight fit	UD-Q5_K_XL	8K	56–76
Qwen3 30B A3B Thinking 2507 30.5B	Tight fit	UD-Q5_K_XL	8K	56–76
Nemotron 3 Nano 30B A3B 31.6B	Tight fit	UD-Q4_K_XL	8K	63–86
NVIDIA Nemotron 3 Nano Omni 30B A3B Reasoning 31.6B	Tight fit	UD-Q4_K_S	4K	69–93
gemma 4 26B A4B it 25.2B	Tight fit	UD-Q5_K_XL	8K	31–42
NVIDIA Nemotron 3.5 Lightning 30B A3B 32.9B	Tight fit	IQ4_NL	8K	70–95
Qwen3 Coder Next 79.7B	Tight fit	UD-IQ1_M	8K	37–50
LFM2.5 24B A2B 24.0B	Tight fit	UD-Q6_K_XL	8K	66–90
Qwen3 Next 80B A3B Thinking 79.7B	Tight fit	UD-IQ1_S	4K	131–179
Qwen3 Next 80B A3B Instruct 79.7B	Tight fit	UD-IQ1_S	4K	130–178
GLM 4.7 Flash REAP 23B A3B 23.0B	Tight fit	UD-Q6_K_XL	8K	54–73
gpt oss 20b 20.9B	Runs great	UD-Q8_K_XL	8K	41–56
Qwen2.5 14B Instruct 14.8B	Runs	Q8_0	8K	9.5–13
NousResearch Hermes 4 14B 14.8B	Runs	Q8_0	8K	9.7–13
DeepSeek R1 Distill Qwen 14B 14.8B	Runs	Q8_0	8K	9.5–13
Qwen2.5 Coder 14B Instruct 128K 14.8B	Runs	Q8_0	8K	9.5–13
phi 4 14.7B	Runs	Q8_0	8K	9.6–13
gemma 4 12b it 11.9B	Runs	UD-Q8_K_XL	8K	9.8–13
DeepSeek R1 Distill Llama 8B 8.0B	Runs	BF16	8K	9.6–13
Llama 3.1 8B Instruct 8.0B	Runs	BF16	8K	9.6–13
Apertus 8B Instruct 2509 8.1B	Runs	BF16	8K	9.6–13
Qwen2.5 7B Instruct 7.6B	Runs	F16	8K	11–14
Qwen2.5 VL 7B Instruct 7.6B	Runs	BF16	8K	11–14
DeepSeek R1 Distill Qwen 7B 7.6B	Runs	F16	8K	11–14
Qwen2.5 Coder 7B Instruct 128K 7.6B	Runs	F16	8K	11–14
gemma 4 E4B it 7.5B	Runs	BF16	8K	10–14
gemma 4 E4B it qat 7.5B	Runs great	UD-Q4_K_XL	8K	34–46
LFM2.5 8B A1B 8.5B	Runs	BF16	8K	69–94
gemma 4 E2B it 4.6B	Runs great	BF16	8K	17–23
Qwen3.5 4B 4.2B	Runs great	BF16	8K	19–26
Qwen3 4B 4.0B	Runs great	BF16	8K	18–24
Qwen3 VL 4B Instruct 4.0B	Runs great	BF16	8K	18–24
Qwen3 4B Thinking 2507 4.0B	Runs great	F16	8K	18–24

90 more models in the catalog do not fit this default profile, or ranked below the first 40 that do.

Example: Tight fit fully on your NVIDIA L4 at UD-Q4_K_XL, 8K context, about 79–108 tok/s. Weights 20.7 GB plus 0.19 GB of KV cache — 21.4 GB of 22.8 GB usable. Up to 32K context still fits. Weights shown in the verdict use GiB (the same 1024³ counting as GPU memory).