A model file is only the weights. To run it, your GPU also has to hold the KV cache (the memory for your context), the engine’s working memory, and whatever the desktop and driver already use. A 4.36 GiB Qwen2.5-7B Q4_K_M file needs 4.92 to 5.42 GiB at 8K context, and up to 6.82 GiB at 32K. On an 8 GB card with about 7 GiB free, that’s the difference between a comfortable fit and a tight one.
Rule of thumb for a 7B at 8K context: plan on roughly 0.6 to 1.1 GiB on top of the file, and more as the context grows.
What uses GPU memory
- The weight file you picked. Only the shards for your chosen quantization, not every file in the repo.
- KV cache. Grows in a straight line with context length and with the number of parallel requests. For Qwen2.5-7B it’s 0.44 GiB at 8K and 1.75 GiB at 32K.
- Engine workspace. Scratch memory for computing each step. We use a low-to-high range per engine rather than a single number.
- Desktop and driver reserve. Memory you can’t use for the model. We subtract 1 GiB on an 8 GB card such as the RTX 4060, leaving 7 GiB. If your idle usage is different, change the reserve in the calculator.
Hugging Face’s model memory estimator makes the same point: the size to load a model isn’t what it uses during inference.
Example: Qwen2.5-7B-Instruct Q4_K_M on an RTX 4060 8GB
| 8K context | 32K context (native max) | 64K context | |
|---|---|---|---|
| Weight file | 4,683,073,632 bytes (4.36 GiB) | same | same |
| KV cache | 469,762,048 bytes (0.44 GiB) | 1,879,048,192 bytes (1.75 GiB) | grows further; past the native window |
| Total estimate | 5,287,053,408 to 5,816,251,408 bytes (4.92 to 5.42 GiB) | 6,696,339,552 to 7,324,187,582 bytes (6.24 to 6.82 GiB) | over the budget |
| On RTX 4060 (7 GiB usable) | Fits | Tight | Does not fit |
One sequence, FP16 KV cache, llama.cpp.
At 32K, this model is tight on 8 GB, not already OOM: it should load if nothing else is using the GPU. 64K is past the 32,768-token window the model was built for, so the estimate is only partial; “does not fit” there is a prediction from the memory math, not a guaranteed CUDA crash. Our data has no extended-context setting for this checkpoint, so the calculator doesn’t assume one.
Prefill (reading a long prompt) can briefly need more workspace than generating does. If you want to see the worst case, switch on high extras in the calculator.
If it still runs out of memory
- Lower the context length to what you actually need. The KV cache calculator shows what each length costs.
- Quantize the KV cache or drop a weight quant level: 4-bit vs 8-bit vs FP16.
- Close other apps that use the GPU, including browsers with hardware acceleration.
- Work through VRAM OOM troubleshooting for engine-specific flags.
For this model on specific cards, see Qwen2.5-7B on an RTX 4060. Model pages split the estimate the same way, for example Qwen2.5-14B-Instruct and Qwen3-8B.
Numbers are estimates from model configs and published file sizes, not measurements. How we calculate.
FAQ
If the GGUF file is smaller than my VRAM, will it run?
Not necessarily. You also need room for the KV cache, which grows with context, plus engine workspace, and the desktop already uses some memory. A 4.36 GiB Qwen2.5-7B file needs 4.92 to 5.42 GiB at 8K context.
How much memory does the desktop take?
We assume 1 GiB on most 8 and 12 GB cards and 2 GiB on most 16 to 32 GB cards. Check your own idle usage in nvidia-smi and adjust the reserve in the calculator.
Can I run 32K context on an 8 GB card with a 7B Q4 model?
Probably, but it's tight. Qwen2.5-7B Q4_K_M at 32K needs 6.24 to 6.82 GiB against about 7 GiB usable. Close other GPU apps first.
Numbers from calculator 0.3.0, model data 2026-09-20.1. Sources last checked 2026-09-21.