Qwen2.5-32B-Instruct quantization: file size and VRAM for Q4, Q5, Q6, Q8 and BF16
The published Q4_K_M file for Qwen2.5-32B-Instruct is 19,851,336,384 bytes (18.5 GiB), 30% of the 61.0 GiB BF16 checkpoint. Q8_0 is 32.4 GiB (53%). Add the KV cache and runtime overhead and one stream at 8,192 tokens needs 20.6 GiB–21.5 GiB with Q4_K_M or 34.6 GiB–35.7 GiB with Q8_0.
The KV cache is 2.00 GiB at 8,192 tokens in fp16 and does not change with the weight quant. For Q4_K_M it is about 9% of the estimated total, which is why the totals shrink less than the files do.
Every published file, side by side
| File | Bytes | GiB | Bits/weight | vs BF16 | Total at 8K | Smallest card that fits |
|---|---|---|---|---|---|---|
| BF16 | 65,527,752,704 | 61.03 GiB | 16.0 | 100% | 63.7 GiB–65.6 GiB | NVIDIA A100 80GB SXM |
| Q8_0 | 34,820,885,632 | 32.43 GiB | 8.5 | 53% | 34.6 GiB–35.7 GiB | NVIDIA L40S 48GB |
| Q6_K | 26,886,155,104 | 25.04 GiB | 6.5 | 41% | 27.2 GiB–28.2 GiB | NVIDIA L40S 48GB |
| Q5_K_M | 23,262,157,568 | 21.66 GiB | 5.5 | 35% | 23.8 GiB–24.7 GiB | GeForce RTX 5090 32GB (desktop) |
| Q4_K_M | 19,851,336,384 | 18.49 GiB | 4.5 | 30% | 20.6 GiB–21.5 GiB | GeForce RTX 5090 32GB (desktop) |
File bytes are the published GGUF (or safetensors for BF16) sizes, not peak GPU memory. Totals are calculator estimates at 8,192 tokens, one sequence, fp16 KV. "Fits" means the high estimate is within 90% of the card's usable memory; the catalog devices are sorted by usable memory, so the first match is the smallest.
What to download for common cards
Highest-quality GGUF that fits at 8,192 tokens, llama.cpp, and the longest context that still fits with that file.
| Card | Usable memory | Pick | Estimated total | Max context |
|---|---|---|---|---|
| GeForce RTX 4060 8GB (desktop) | 7.0 GiB | Nothing fits | n/a | n/a |
| GeForce RTX 3060 12GB (desktop) | 11.0 GiB | Nothing fits | n/a | n/a |
| GeForce RTX 4060 Ti 16GB (desktop) | 15.0 GiB | Nothing fits | n/a | n/a |
| GeForce RTX 4090 24GB (desktop) | 22.0 GiB | Q4_K_M (tight) | 20.6 GiB–21.5 GiB | 13,873 |
| GeForce RTX 5090 32GB (desktop) | 30.0 GiB | Q5_K_M | 23.8 GiB–24.7 GiB | 16,851 |
What this page does not say
This site sizes memory. It does not rank output quality between quants, so there is no "best quant" claim here beyond which file fits. For the arithmetic behind bits per weight, read 4-bit vs 8-bit vs FP16.
Data sources
- Qwen/Qwen2.5-32B-Instruct @ 5ede1c9 — BF16 file bytes, fetched 2026-09-20
- Qwen/Qwen2.5-32B-Instruct-GGUF @ a15e3cc — Q8_0 file bytes, fetched 2026-09-24
- Qwen/Qwen2.5-32B-Instruct-GGUF @ a15e3cc — Q6_K file bytes, fetched 2026-09-24
- Qwen/Qwen2.5-32B-Instruct-GGUF @ a15e3cc — Q5_K_M file bytes, fetched 2026-09-24
- Qwen/Qwen2.5-32B-Instruct-GGUF @ a15e3cc — Q4_K_M file bytes, fetched 2026-09-20