Which local LLMs fit on an RTX 5090 (32 GB GDDR7)?

The RTX 5090 has 32 GB GDDR7 of discrete VRAM. After a 2 GiB reserve for the OS and display, about 30 GiB is usable for a model. At an 8K context the largest model in our list that fits is Qwen2.5-32B-Instruct at Q5_K_M (about 24.7 GiB). Mixtral-8x7B-Instruct-v0.1 at Q4_K_M is tight (about 27.6 GiB–28.6 GiB).

These are estimates from the model's config and file sizes, not measurements. How we calculate

Which models fit at 8K

Each model uses the highest-precision file that fits, one chat at a time.

Fits (58)

Model	Quant	Total
Qwen3.8-27B	GGUF Q6_K (mradermacher)	21.7 GiB–25.9 GiB
Mistral-Small-24B-Instruct-2501	GGUF Q8_0 (Unsloth)	24.7 GiB–25.6 GiB
Qwen3-VL-30B-A3B-Instruct	GGUF Q6_K (Unsloth)	24.2 GiB–25.1 GiB
Qwen3-30B-A3B-Instruct-2507	GGUF Q6_K (Unsloth)	24.2 GiB–25.1 GiB
Qwen3-30B-A3B	GGUF Q6_K (Qwen)	24.2 GiB–25.1 GiB
Qwen2.5-32B-Instruct	GGUF Q5_K_M (Qwen)	23.8 GiB–24.7 GiB
Qwen2.5-Coder-32B-Instruct	GGUF Q5_K_M (Qwen)	23.8 GiB–24.7 GiB
QwQ-32B	GGUF Q5_K_M (Qwen)	23.8 GiB–24.7 GiB
DeepSeek-R1-Distill-Qwen-32B	GGUF Q5_K_M (Unsloth)	23.8 GiB–24.7 GiB
Qwen2.5-VL-32B-Instruct	GGUF Q5_K_M (Unsloth)	23.8 GiB–24.7 GiB
OpenMath-Nemotron-32B	GGUF Q5_K_M (mradermacher)	23.8 GiB–24.7 GiB
Qwen3-32B	GGUF Q5_K_M (Qwen)	23.7 GiB–24.7 GiB
DeepSeek-V2-Lite-Chat	GGUF Q8_0 (mradermacher)	15.9 GiB–19.2 GiB
Gemma 3 27B IT	GGUF Q4_0 (Google QAT)	17.2 GiB–18.0 GiB
Qwen2.5-14B-Instruct	GGUF Q8_0	16.2 GiB–17.0 GiB
Qwen2.5-Coder-14B-Instruct	GGUF Q8_0 (Qwen)	16.2 GiB–17.0 GiB
OpenMath-Nemotron-14B	GGUF Q8_0 (mradermacher)	16.2 GiB–17.0 GiB
Phi-4-reasoning	GGUF Q8_0 (Unsloth)	16.2 GiB–17.0 GiB
phi-4	GGUF Q8_0 (Microsoft)	16.2 GiB–17.0 GiB
Llama 3.1 8B Instruct	Official BF16 safetensors	16.1 GiB–16.8 GiB
Qwen3-14B	GGUF Q8_0 (Qwen)	16.0 GiB–16.8 GiB
Mistral-Nemo-Instruct-2407	GGUF Q8_0 (Unsloth)	13.5 GiB–14.2 GiB
gpt-oss-20b	MXFP4 safetensors	13.1 GiB–13.8 GiB
GLM-4-9B-0414	GGUF Q8_0 (Unsloth)	9.8 GiB–10.3 GiB
granite-3.3-8b-instruct	GGUF Q8_0 (IBM)	9.5 GiB–10.1 GiB
Qwen3-VL-8B-Instruct	GGUF Q8_0 (Qwen)	9.4 GiB–10.0 GiB
Qwen3-8B-Base	GGUF Q8_0 (mradermacher)	9.4 GiB–10.0 GiB
Qwen3-8B	GGUF Q8_0	9.4 GiB–10.0 GiB
Qwen2.5-Math-7B-Instruct	GGUF Q8_0 (bartowski)	8.1 GiB–10.0 GiB
DeepSeek-R1-Distill-Llama-8B	GGUF Q8_0 (Unsloth)	9.1 GiB–9.7 GiB
Gemma 3 12B IT	GGUF Q4_0 (Google QAT)	8.5 GiB–9.0 GiB
Mistral-7B-Instruct-v0.3	GGUF Q8_0 (bartowski)	8.3 GiB–8.9 GiB
Qwen2.5-Coder-7B	GGUF Q8_0 (mradermacher)	8.1 GiB–8.7 GiB
Qwen2.5-Coder-7B-Instruct	GGUF Q8_0 (Qwen)	8.1 GiB–8.7 GiB
Qwen2.5-7B-Instruct	GGUF Q8_0	8.1 GiB–8.7 GiB
DeepSeek-R1-Distill-Qwen-7B	GGUF Q8_0 (Unsloth)	8.1 GiB–8.7 GiB
OpenMath-Nemotron-7B	GGUF Q8_0 (mradermacher)	8.1 GiB–8.7 GiB
Qwen2.5-VL-7B-Instruct	GGUF Q8_0 (Unsloth)	8.1 GiB–8.7 GiB
Qwen2-7B-Instruct	GGUF Q8_0 (Qwen)	8.1 GiB–8.7 GiB
Phi-3.5-mini-instruct	GGUF Q8_0 (bartowski)	6.9 GiB–7.6 GiB
Llama 3.2 3B Instruct	Official BF16 safetensors	7.0 GiB–7.5 GiB
Qwen3-VL-4B-Instruct	GGUF Q8_0 (Qwen)	5.2 GiB–5.8 GiB
Qwen3-4B-Instruct-2507	GGUF Q8_0 (Unsloth)	5.2 GiB–5.8 GiB
Qwen3-4B	GGUF Q8_0	5.2 GiB–5.8 GiB
Phi-4-mini-instruct	GGUF Q8_0 (Unsloth)	4.9 GiB–5.5 GiB
Qwen2.5-Coder-3B-Instruct	GGUF Q8_0 (Qwen)	3.8 GiB–4.2 GiB
Qwen2.5-3B-Instruct	GGUF Q8_0	3.8 GiB–4.2 GiB
Qwen2.5-VL-3B-Instruct	GGUF Q8_0 (Unsloth)	3.5 GiB–3.9 GiB
SmolLM2-1.7B-Instruct	GGUF Q8_0 (Unsloth)	3.3 GiB–3.8 GiB
Gemma 3 4B IT	GGUF Q4_0 (Google QAT)	3.3 GiB–3.8 GiB
Qwen3-1.7B-Base	GGUF Q8_0 (mradermacher)	2.7 GiB–3.2 GiB
Qwen3-1.7B	GGUF Q8_0	2.7 GiB–3.2 GiB
Llama 3.2 1B Instruct	Official BF16 safetensors	2.7 GiB–3.1 GiB
Qwen2.5-Coder-1.5B-Instruct	GGUF Q8_0 (Qwen)	2.1 GiB–2.5 GiB
Qwen2.5-1.5B-Instruct	GGUF Q8_0 (Qwen)	2.1 GiB–2.5 GiB
Qwen3-0.6B	GGUF Q8_0 (Qwen)	1.6 GiB–2.0 GiB
Gemma 3 1B IT	GGUF Q4_0 (Google QAT)	1.1 GiB–1.5 GiB
Qwen2.5-0.5B-Instruct	GGUF Q8_0 (Qwen)	0.8 GiB–1.2 GiB

Tight fit (2)

Model	Quant	Total
Mixtral-8x7B-Instruct-v0.1	GGUF Q4_K_M (mradermacher)	27.6 GiB–28.6 GiB
Drama 3	Unofficial int4 W4A16 (GPTQ-Marlin, with codec)	22.7 GiB–27.0 GiB

Won't fit (26)

Llama-3.1-Nemotron-70B-Instruct-HF · Q4_K_M · 43.6 GiB
Qwen2.5-72B-Instruct · Q4_K_M · 45 GiB
Qwen3-Next-80B-A3B-Instruct · Q4_K_M · 47.3 GiB
gpt-oss-120b · MXFP4 · 62.8 GiB
GLM-4.5-Air · Q4_K_M · 71.4 GiB
Qwen3.8-Flash-Next · Q4_K_M · 131.5 GiB
MiniMax-M2 · Q4_K_M · 134 GiB
Qwen3-235B-A22B · Q4_K_M · 137.1 GiB
Qwen3-235B-A22B-Instruct-2507 · Q4_K_M · 137.1 GiB
Llama 3.3 70B Instruct · BF16 · 137.2 GiB
DeepSeek-V4-Flash-0731 · FP8 · 183.7 GiB
GLM-4.6 · Q4_K_M · 208.4 GiB
GLM-4.5 · Q4_K_M · 209.2 GiB
MiniMax-M3 · Q4_K_M · 249.7 GiB
GLM-5.3-Flash · Q8_0 · 373.6 GiB
DeepSeek-V3 · Q4_K_M · 443.1 GiB
DeepSeek-R1 · Q4_K_M · 443.1 GiB
DeepSeek-V3-0324 · Q4_K_M · 443.1 GiB
DeepSeek-R1-0528 · Q4_K_M · 443.7 GiB
DeepSeek-V3.1 · Q4_K_M · 444.2 GiB
Muse Spark Open · NVFP4 · 508.9 GiB
DeepSeek-V4.1-Flash · FP8 · 558.8 GiB
Ling-3.1-flash · FP8 · 628.3 GiB
Kimi-K2-Instruct · Q4_K_M · 679.5 GiB
Kimi-K2-Thinking · Q4_K_M · 680 GiB
GLM-5.3 · Q8_0 · 877 GiB

Detailed checks

Known deployments on this hardware

1 known setups

Real deployment setups tested by maintainers and community labs on this hardware platform.

Qwen3.8-27B

1× GeForce RTX 5090 32GB (desktop)

Measured by upstream maintainer

Quant:NVFP4 safetensors(4 bpw)

Runtime:vLLM

Tested:

✓ generation
✓ 262,144 context
✓ tool calling
✓ speculative decoding

Measured:160 tok/s· 1 stream

Single NVIDIA RTX 5090 (32 GB, sm_120) deployment serving RadixArk/Qwen3.8-27B-NVFP4 with vLLM 0.27.1, native 256K context, TurboQuant 4-bit KV cache, and MTP-3 speculative decoding.

Use this setup Quick start Source ↗

Memory

NVIDIAGeForce RTX 5090 32GB desktop
Memory kindDiscrete VRAM
Marketed32 GB GDDR7
Gross32.00 GiB (34,359,738,368 bytes)
Default reserve2.00 GiB (2,147,483,648 bytes)
Net budget30.00 GiB (32,212,254,720 bytes)

The reserve is an adjustable assumption. On a headless machine you can lower it in the calculator.

Two of these cards do not add up to one bigger card. Each copy of a model has to fit on one card unless the engine splits it with tensor parallelism.

Open the hardware-first calculator with this device Calculator

Sources

NVIDIA GeForce RTX 5090 product page — Standard Memory Config 32 GB GDDR7

Calculator

Loading catalog snapshot…

Your GPU or Mac

Model

Context length

Longer context needs more memory for the KV cache, the model's working memory for the conversation.

More options

Show historical models

Runtime

Engine and KV cache type for this estimate.

Quantization filter Allow a substitute if the filter has no match

Off: GGUF, Q8-class, and other filters stay strict. On: a replacement is shown with requested, used, and reason.

EngineKV cache typeRecurrent / conv state dtype Include theoretical estimates (expert)

Workload

Overhead scenario and the prefill workspace.

Overhead scenario Include heavier prefill workspace

Topology

How the model is split across devices. Memory is not added together.

DeploymentIntra-replica splitDevice count

Replica count is an advanced deployment option. Quantity is nodes or GPUs, not replicas.

Replica groupsDevices per replica

Budget

System reserve and safety margin. An empty number keeps the previous value.

System / display reserve

GiB (2^30 bytes)

Budget already excludes system reserveExtra safety margin

Custom bits per weightDisplay unit

Pick a GPU or Mac to see which models fit, or pick a model to see how much memory it needs.