Going from 16-bit to 4-bit cuts a model’s weight file to roughly a third, not a quarter. For Qwen2.5-7B-Instruct, the official files are 14.19 GiB in BF16, 7.54 GiB at Q8_0 and 4.36 GiB at Q4_K_M. Real 4-bit files are bigger than “parameters × half a byte” because they carry per-block scales and keep some tensors at higher precision.
To size a model, use the actual file you’ll download, not the bit width on the label. Then add the KV cache and runtime overhead; see why file size isn’t the VRAM you need.
Qwen2.5-7B: textbook math vs real files
| What | Bytes | Notes |
|---|---|---|
| 4-bit math | 3,807,808,256 | 7,615,616,512 parameters × 4 / 8. No block scales. |
| Official GGUF Q4_K_M | 4,683,073,632 | Two published shards |
| 8-bit math | 7,615,616,512 | Parameters × 8 / 8 |
| Official GGUF Q8_0 | 8,098,525,408 | Three published shards |
| BF16 safetensors | 15,231,233,024 | The total_size in the official index |
Q4_K_M comes out about 23% above the plain 4-bit figure, and Q8_0 about 6% above 8-bit. The gap comes from three things:
- Block scales. K-quants group weights into blocks and store a scale (and sometimes an offset) per block at higher precision.
- Tensors kept at higher precision. Embeddings and some output and norm tensors usually aren’t packed as tightly as the rest.
- File format overhead. GGUF and safetensors carry metadata the simple formula ignores.
AWQ, GPTQ and EXL3 files differ from GGUF Q4_K_M too, and none of them is exactly half a byte per parameter. The calculator always uses the published file size when one exists.
What fits where (Qwen2.5-7B, 8K context)
| Format | Total at 8K context (GiB) |
|---|---|
| Q4_K_M | 4.92 to 5.42 |
| Q8_0 | 8.10 to 8.66 |
| BF16 | 14.75 to 15.44 |
One sequence, FP16 KV cache, llama.cpp. On an 8 GB card (about 7 GiB usable), Q4_K_M fits and Q8_0 doesn’t. On 12 GB, Q8_0 fits. BF16, at up to 15.44 GiB, is more than a 16 GB card has free (14 to 15 GiB), so plan on 24 GB. The in-between steps (Q5_K_M, Q6_K) are in Q4 vs Q5 vs Q6 vs Q8.
Memory only: none of this says which bit width gives better answers, and there’s no “near-lossless” or “sweet spot” claim here. Compare quality on your own prompts.
Weight quantization doesn’t set the KV cache
A Q4_K_M model does not automatically mean 4-bit KV cache. Most engines default to FP16 KV unless you change it:
- llama.cpp:
-ctk/-ctv(defaultf16, per the llama.cpp CLI README, checked 2026-09-21) - vLLM:
--kv-cache-dtype(defaultauto, per the vLLM engine args docs, checked 2026-09-21)
What a quantized cache saves is in FP16 vs Q8_0 vs Q4_0 KV cache, and you can size the cache for your context in the KV cache calculator.
No silent swaps
If you ask the calculator for a quantization that has no published file, it says the file is missing. It won’t quietly substitute another format and show you its numbers. The full file list for this model is on the Qwen2.5-7B-Instruct page, and the same comparison for another model is on Qwen3-8B.
Numbers are estimates from model configs and published file sizes, not measurements. How we calculate.
FAQ
How much smaller is a 4-bit model than FP16?
Roughly a third of the size, not a quarter. Qwen2.5-7B-Instruct is 15,231,233,024 bytes in BF16 and 4,683,073,632 bytes as an official Q4_K_M GGUF.
Why is a Q4_K_M file bigger than parameters × 4 bits?
Q4_K_M stores scale values for each block of weights and keeps some tensors, such as embeddings, at higher precision. For Qwen2.5-7B that makes the file about 23% larger than the plain 4-bit math.
Does a 4-bit model use a 4-bit KV cache?
No. The KV cache has its own setting and usually defaults to FP16, whatever the weight format.
Which is better quality, Q4 or Q8?
File size can't tell you that. These numbers cover memory only; test quality on your own prompts.
Numbers from calculator 0.3.0, model data 2026-09-20.1. Sources last checked 2026-09-21.