Qwen2.5-14B-Instruct quantization: file size and VRAM for Q4, Q5, Q6, Q8 and BF16
The published Q4_K_M file for Qwen2.5-14B-Instruct is 8,988,110,496 bytes (8.4 GiB), 30% of the 27.5 GiB BF16 checkpoint. Q8_0 is 14.6 GiB (53%). Add the KV cache and runtime overhead and one stream at 8,192 tokens needs 10.0 GiB–10.6 GiB with Q4_K_M or 16.2 GiB–17.0 GiB with Q8_0.
The KV cache is 1.50 GiB at 8,192 tokens in fp16 and does not change with the weight quant. For Q4_K_M it is about 15% of the estimated total, which is why the totals shrink less than the files do.
Every published file, side by side
| File | Bytes | GiB | Bits/weight | vs BF16 | Total at 8K | Smallest card that fits |
|---|---|---|---|---|---|---|
| BF16 | 29,540,067,328 | 27.51 GiB | 16.0 | 100% | 29.6 GiB–31.6 GiB | NVIDIA L40S 48GB |
| Q8_0 | 15,701,597,984 | 14.62 GiB | 8.5 | 53% | 16.2 GiB–17.0 GiB | Radeon RX 7900 XTX 24GB (desktop) |
| Q6_K | 12,124,684,064 | 11.29 GiB | 6.5 | 41% | 12.9 GiB–13.6 GiB | Radeon RX 7900 XT 20GB (desktop) |
| Q5_K_M | 10,508,873,376 | 9.79 GiB | 5.5 | 36% | 11.4 GiB–12.1 GiB | Radeon RX 7800 XT 16GB (desktop) |
| Q4_K_M | 8,988,110,496 | 8.37 GiB | 4.5 | 30% | 10.0 GiB–10.6 GiB | Radeon RX 7800 XT 16GB (desktop) |
File bytes are the published GGUF (or safetensors for BF16) sizes, not peak GPU memory. Totals are calculator estimates at 8,192 tokens, one sequence, fp16 KV. "Fits" means the high estimate is within 90% of the card's usable memory; the catalog devices are sorted by usable memory, so the first match is the smallest.
What to download for common cards
Highest-quality GGUF that fits at 8,192 tokens, llama.cpp, and the longest context that still fits with that file.
| Card | Usable memory | Pick | Estimated total | Max context |
|---|---|---|---|---|
| GeForce RTX 4060 8GB (desktop) | 7.0 GiB | Nothing fits | n/a | n/a |
| GeForce RTX 3060 12GB (desktop) | 11.0 GiB | Q4_K_M (tight) | 10.0 GiB–10.6 GiB | 13,676 |
| GeForce RTX 4060 Ti 16GB (desktop) | 15.0 GiB | Q5_K_M | 11.4 GiB–12.1 GiB | 15,399 |
| GeForce RTX 4090 24GB (desktop) | 22.0 GiB | Q8_0 | 16.2 GiB–17.0 GiB | 22,377 |
| GeForce RTX 5090 32GB (desktop) | 30.0 GiB | Q8_0 | 16.2 GiB–17.0 GiB | 32,768 |
What this page does not say
This site sizes memory. It does not rank output quality between quants, so there is no "best quant" claim here beyond which file fits. For the arithmetic behind bits per weight, read 4-bit vs 8-bit vs FP16.
Data sources
- Qwen/Qwen2.5-14B-Instruct @ cf98f3b — BF16 file bytes, fetched 2026-09-20
- Qwen/Qwen2.5-14B-Instruct-GGUF @ b466e1f — Q8_0 file bytes, fetched 2026-09-20
- Qwen/Qwen2.5-14B-Instruct-GGUF @ b466e1f — Q6_K file bytes, fetched 2026-09-24
- Qwen/Qwen2.5-14B-Instruct-GGUF @ b466e1f — Q5_K_M file bytes, fetched 2026-09-24
- Qwen/Qwen2.5-14B-Instruct-GGUF @ b466e1f — Q4_K_M file bytes, fetched 2026-09-20