Qwen2.5-7B-Instruct at 8,192 tokens, 1 sequence, FP16 KV, llama.cpp, needs 4.92 to 5.42 GiB in official GGUF Q4_K_M (4,683,073,632 bytes). Q5_K_M is 5.63 to 6.14 GiB, Q6_K is 6.39 to 6.91 GiB, Q8_0 is 8.10 to 8.66 GiB, and BF16 safetensors are 14.75 to 15.44 GiB. The KV term stays 0.44 GiB on every row. These are formula estimates from calculator 0.3.0, dataset 2026-10-02.1. Runtime process memory for these five files is Unknown. Software support is unverified.
The 4-bit vs 8-bit vs FP16 page compares naive bits-per-weight with Q4_K_M and Q8_0. This page fills in Q5_K_M and Q6_K from the same official Qwen GGUF listing, then shows where the next quant crosses a 24 GB card.
Qwen2.5-7B-Instruct, no device
Repo Qwen/Qwen2.5-7B-Instruct-GGUF for the GGUF rows and Qwen/Qwen2.5-7B-Instruct for BF16. Scenario: 8,192 tokens, 1 sequence, FP16 KV, llama.cpp, typical extras. No GPU is selected, so the status is demand-only. The byte totals are still the calculator’s complete range.
| File | Bytes | Weights GiB | KV GiB | Total GiB |
|---|---|---|---|---|
| Q4_K_M | 4,683,073,632 | 4.36 | 0.44 | 4.92 to 5.42 |
| Q5_K_M | 5,444,831,360 | 5.07 | 0.44 | 5.63 to 6.14 |
| Q6_K | 6,254,198,880 | 5.82 | 0.44 | 6.39 to 6.91 |
| Q8_0 | 8,098,525,408 | 7.54 | 0.44 | 8.10 to 8.66 |
| BF16 safetensors | 15,231,233,024 | 14.19 | 0.44 | 14.75 to 15.44 |
Q5_K_M is 761,077,728 bytes above Q4_K_M. Q6_K is another 809,367,520 bytes. Q8_0 is another 1,844,326,528 bytes. None of those steps change the KV formula: 28 layers, 4 KV heads, head dim 128, FP16, one sequence.
Qwen3-8B, same ladder, larger KV
Official Qwen/Qwen3-8B-GGUF. Qwen3-8B has 36 layers and 8 KV heads, so FP16 KV at 8,192 tokens is 1.13 GiB, not 0.44 GiB.
| File | Bytes | Weights GiB | KV GiB | Total GiB |
|---|---|---|---|---|
| Q4_K_M | 5,027,783,488 | 4.68 | 1.13 | 5.93 to 6.48 |
| Q5_K_M | 5,851,112,224 | 5.45 | 1.13 | 6.70 to 7.26 |
| Q6_K | 6,725,899,040 | 6.26 | 1.13 | 7.51 to 8.09 |
| Q8_0 | 8,709,518,112 | 8.11 | 1.13 | 9.36 to 9.98 |
| BF16 safetensors | 16,381,470,720 | 15.26 | 1.13 | 16.51 to 17.27 |
Where Q5 leaves a 24 GB card
Official Qwen/Qwen2.5-32B-Instruct-GGUF on a GeForce RTX 4090 24GB. The catalog capacity is 24 GiB with a 2 GiB reserve, so the net budget is 22.00 GiB. Same 8,192-token llama.cpp scenario.
| File | Bytes | Total GiB | Verdict |
|---|---|---|---|
| Q4_K_M | 19,851,336,384 | 20.61 to 21.50 | tight |
| Q5_K_M | 23,262,157,568 | 23.79 to 24.74 | insufficient |
| Q6_K | 26,886,155,104 | 27.16 to 28.18 | insufficient |
| Q8_0 | 34,820,885,632 | 34.55 to 35.72 | insufficient |
Q4_K_M’s high end, 21.50 GiB, is under 22.00 GiB and above 90% of that net budget, so the verdict is tight. Q5_K_M’s low end is already over 22.00 GiB. A tight or insufficient memory verdict is not a statement about CUDA kernels. Software support on this page is unverified. Tokens per second are Unknown.
Context changes the same files again. See context length VRAM tradeoff and the KV cache calculator. Model pages: Qwen2.5-7B-Instruct, Qwen3-8B, Qwen2.5-32B-Instruct.
FAQ
Does a higher GGUF quant shrink the KV cache?
No. On Qwen2.5-7B-Instruct at 8,192 tokens and FP16 KV, Q4_K_M through BF16 all keep a 0.44 GiB KV term. Only the weight file changes.
Will Qwen2.5-32B Q5_K_M fit an RTX 4090?
The calculator says insufficient. Official Q5_K_M at 8,192 tokens is 23.79 to 24.74 GiB against a 22.00 GiB net budget. Q4_K_M is 20.61 to 21.50 GiB and the verdict is tight. Software support is unverified.
Are these rows measured?
No. They are formula estimates from calculator 0.3.0. Process GPU memory for these files is Unknown. The measured runs in this repository are Qwen2.5-3B-Instruct Q4_K_M on an RTX 4090.