Qwen3-32B official GGUF Q4_K_M keeps 18.40 GiB of weights (19,762,149,024 bytes) at every KV dtype. At 32,768 tokens, 1 sequence, llama.cpp, on a GeForce RTX 4090 (22.00 GiB net), FP16 KV makes the total 26.53 to 27.83 GiB (insufficient) and Q4_0 KV makes it 20.78 to 21.68 GiB (tight). Formula estimates, calculator 0.3.0. Software support for these KV types on this file is unverified. Quality and tokens per second are Unknown.
The element sizes live in src/calc/memory/kv.ts. FP16 and BF16 are 2 bytes. FP8 and INT8 are 1 byte. Q8_0 is 1 byte plus a 2-byte scale per 32 values (6.25%). Q4_0 is 0.5 bytes plus a 2-byte scale per 32 packed values, stored as a 12.5% extra on that half-byte payload.
One file, four KV dtypes
RTX 4090, 32,768 tokens, typical extras.
| KV dtype | Raw KV GiB | Block extra GiB | Total GiB | Verdict |
|---|---|---|---|---|
| FP16 | 8.00 | 0.00 | 26.53 to 27.83 | insufficient |
| FP8 | 4.00 | 0.00 | 22.53 to 23.55 | insufficient |
| Q8_0 | 4.00 | 0.25 | 22.78 to 23.82 | insufficient |
| Q4_0 | 2.00 | 0.25 | 20.78 to 21.68 | tight |
FP8 and Q8_0 both miss the 22.00 GiB budget on the low end. Q4_0’s high end, 21.68 GiB, is under 22.00 GiB and above 90% of it, so the verdict is tight. At 8,192 tokens the same weights with default FP16 KV are 20.53 to 21.41 GiB, also tight, which is the row on context length VRAM tradeoff.
Changing the weight quant does not apply these KV sizes. Q4_K_M, Q5_K_M, Q6_K and Q8_0 weight files at FP16 KV are on Q4 vs Q5 vs Q6 vs Q8. The raw GQA formula is on context length and KV cache.
FAQ
Does Q4_K_M weights mean the KV is also 4-bit?
No. The default KV type in this calculator is FP16. Qwen3-32B Q4_K_M at 32,768 tokens stores 8.00 GiB of FP16 KV on top of 18.40 GiB of weights.
Why is Q8_0 KV larger than FP8 KV?
Both use 1 byte per element, so raw KV is 4.00 GiB. Q8_0 adds a 2-byte scale per 32 values, which is 0.25 GiB here. FP8 has no block-scale term in this calculator, so the total is 22.53 to 23.55 GiB versus 22.78 to 23.82 GiB for Q8_0.
Is Q4_0 KV safe for quality?
Unknown. This repository has no measurement of output quality or of process GPU bytes for quantized KV on Qwen3-32B.