A 4.36 GiB GGUF can fail on an 8 GB card. The file is the selected quantization payload. Runtime still allocates KV cache, compute buffers, and often a CUDA/Metal/Vulkan context.
Example, calculator 0.1.0 / dataset 2026-09-19.1:
- Model: Qwen2.5-7B-Instruct
- Quant: official GGUF Q4_K_M shards only (4,683,073,632 bytes)
- Context: 32,768 cached tokens
- Sequences: 1
- KV: FP16
- Engine: llama.cpp, typical extras
Raw KV scales linearly with context. At 8,192 tokens this model’s raw KV is 469,762,048 bytes (0.44 GiB). At 32,768 tokens it is 1,879,048,192 bytes (1.75 GiB). Workspace also grows in the high scenario. An 8 GB desktop GPU is treated as 8 GiB minus a default 1 GiB display/system reserve, so the budget is 7 GiB unless you say the budget is already net.
Hugging Face’s own model memory estimator is explicit: loading-size estimates are not inference occupancy.
Do not add every GGUF in a repo. This site only sums the shards of the quantization you picked.