Can an RTX 4090 run Qwen2.5-32B-Instruct?
Just barely Qwen2.5-32B-Instruct at Q4_K_M needs 20.61 GiB–21.50 GiB at 8,192 tokens of context. The RTX 4090 has about 22 GiB usable.
5 files, 19.9 GB in total.
These are estimates from the model's config and file sizes, not measurements. How we calculate
What to change
- Shorten the context to about 1,536 tokens for a clean fit.
- Close other apps that hold GPU memory, or lower the reserve if the machine runs headless.
Other files on this GPU
| Quant | File size | Engine | Total | Fit |
|---|---|---|---|---|
| BF16 safetensors | 65.5 GB | vllm | 63.7 GiB–65.6 GiB | Does not fit |
| GGUF Q8_0 (Qwen) | 34.8 GB | llama.cpp | 34.6 GiB–35.7 GiB | Does not fit |
| GGUF Q6_K (Qwen) | 26.9 GB | llama.cpp | 27.2 GiB–28.2 GiB | Does not fit |
| GGUF Q5_K_M (Qwen) | 23.3 GB | llama.cpp | 23.8 GiB–24.7 GiB | Does not fit |
| GGUF Q4_K_M This page | 19.9 GB | llama.cpp | 20.6 GiB–21.5 GiB | Tight fit |
Same context (8,192 tokens), one chat at a time, each file with its usual engine.
Settings used
- ModelQwen2.5-32B-Instruct
- HardwareGeForce RTX 4090 24GB (desktop)
- QuantizationGGUF Q4_K_M
- Context length (cached tokens)8,192
- Engine
llama.cpp - Concurrent sequences1
- KV cache typefp16
Technical identifiers and share data
- Model
qwen2.5-32b-instruct - Hardware
nvidia-geforce-rtx-4090-24gb-desktop - Quantization
gguf-q4_k_m - Share fragment for this identity
v2.eyJzY2hlbWFWZXJzaW9uIjoyLCJjYWxjdWxhdG9yVmVyc2lvbiI6IjAuMy4wIiwiZGF0YXNldFZlcnNpb24iOiIyMDI2LTEwLTAyLjEiLCJjb25maWciOnsibW9kZWxJZCI6InF3ZW4yLjUtMzJiLWluc3RydWN0IiwiaGFyZHdhcmVJZCI6Im52aWRpYS1nZWZvcmNlLXJ0eC00MDkwLTI0Z2ItZGVza3RvcCIsImRldmljZUNvdW50IjoxLCJxdWFudGl6YXRpb25JZCI6ImdndWYtcTRfa19tIiwiY29udGV4dFRva2VucyI6ODE5MiwiY29uY3VycmVudFNlcXVlbmNlcyI6MSwiZW5naW5lIjoibGxhbWEuY3BwIiwia3ZUeXBlIjoiZnAxNiJ9fQ
How this number is calculated
Total = weight file + KV cache + runtime buffers. The range spans low to high runtime overhead for the engine shown.
- Weights (file)
- 18.49 GiB (19,851,336,384 bytes)
- KV cache
- 2.00 GiB (2,147,483,648 bytes)
- Runtime buffers
- 0.13 GiB–1.01 GiB
- Total
- 20.61 GiB–21.50 GiB
- Usable memory
- 22.00 GiB
- OS / display reserve
- 2.00 GiB
- Left over
- 0.50 GiB–1.39 GiB
21.50 GiB high estimate vs 22.00 GiB net
Enough memory does not mean this engine, quant, and OS will run. Support does not mean it will be fast.
- Requested context is within the recorded native window of 32768 tokens.
- Full-resident weight estimates use total parameters, including inactive MoE experts.
- Weight bytes come from the selected weight files only, not from every file in the repository.
- Download file size is not peak GPU memory. Runtime layout, allocator padding, and KV are extra.
- Counted files: qwen2.5-32b-instruct-q4_k_m-00001-of-00005.gguf, qwen2.5-32b-instruct-q4_k_m-00002-of-00005.gguf, qwen2.5-32b-instruct-q4_k_m-00003-of-00005.gguf, qwen2.5-32b-instruct-q4_k_m-00004-of-00005.gguf, qwen2.5-32b-instruct-q4_k_m-00005-of-00005.gguf.
- Context budget is the number of cached tokens per sequence (prompt plus reserved generation).
- Concurrent sequences share one model replica; weights are not multiplied by sequence count.
- KV dtype is independent of weight quantization unless you change it.
Related pages
Sources
- Qwen2.5-32B-Instruct model card
- config.json — revision 5ede1c97bbab6ce5cda5812749b4c0bdf79b18dd
- safetensors index — metadata.total_size=65527752704
- Artifact repo Qwen/Qwen2.5-32B-Instruct-GGUF
- NVIDIA GeForce RTX 4090 product page — Standard Memory Config 24 GB GDDR6X
Calculator
This combo page keeps the route model and device. Customize to change them on the main calculator; the share then keeps that identity.
Loading catalog snapshot…
Pick a model to see how much memory it needs.