The KV cache is the memory a model uses to remember the tokens it has already processed, and it grows in a straight line with context length. For standard grouped-query-attention (GQA) models:
KV bytes = 2 × layers × KV heads × head dim × tokens × sequences × bytes per value
For Qwen2.5-7B-Instruct in FP16, that’s 0.44 GiB at 8K tokens and 1.75 GiB at 32K. For Qwen2.5-32B-Instruct it’s 2 GiB at 8K and 8 GiB at 32K. The weights don’t change with context; only this term does.
To try your own numbers, use the KV cache calculator.
What each term means
- 2: one tensor for keys, one for values.
- layers: the number of transformer layers (
num_hidden_layersinconfig.json). - KV heads:
num_key_value_heads, not the number of attention heads. - head dim:
head_dimif the config has it, otherwise hidden size ÷ attention heads. - tokens: everything in the context per sequence: the prompt plus what the model generates.
- sequences: requests running in parallel on the same loaded model.
- bytes per value: 2 for FP16 or BF16, 1 for FP8 or INT8.
Worked examples
Qwen2.5-7B-Instruct has 28 attention heads but only 4 KV heads, 28 layers and a head dim of 128. Using the 28 attention heads instead of the 4 KV heads would make the cache 7× too big.
| Model | Layers × KV heads × head dim | 8K tokens, FP16 | 32K tokens, FP16 |
|---|---|---|---|
| Qwen2.5-7B-Instruct | 28 × 4 × 128 | 469,762,048 bytes (0.44 GiB) | 1,879,048,192 bytes (1.75 GiB) |
| Qwen2.5-32B-Instruct | 64 × 8 × 128 | 2,147,483,648 bytes (2.00 GiB) | 8,589,934,592 bytes (8.00 GiB) |
One sequence. If you’ve seen half these figures quoted for Qwen2.5-32B, that’s the result for a 32-layer model; this one has 64.
At 131,072 tokens the 7B’s cache would be 7,516,192,768 bytes (7.00 GiB). But Qwen2.5-7B’s native window is 32,768 tokens, and our data has no extended-context setting for it, so treat 128K as math rather than a tested configuration.
Parallel requests
Four parallel 8K sequences on Qwen2.5-7B hold 4 × 0.4375 = 1.75 GiB of KV. The weights are still loaded once. That’s a memory estimate, not a promise of how many users the model can serve at a usable speed. More on this in parallel requests and VRAM.
When this formula doesn’t apply
- MLA (DeepSeek-V2 and V3) compresses the cache differently. We don’t apply the GQA formula to it.
- Sliding-window models such as Gemma 3 and gpt-oss keep only a short window in most layers. See KV cache size by model.
- Hybrid models such as Qwen3.8 only cache attention layers; their recurrent layers keep a fixed-size state. See hybrid attention VRAM.
- Quantized weights aren’t a quantized cache. A Q4_K_M model still uses an FP16 cache unless you set
-ctk/-ctvin llama.cpp. See FP16 vs Q8_0 vs Q4_0 KV cache.
Hugging Face explains cache layouts in more depth in the Transformers KV cache guide. For how this adds up with weights on real cards, see context length VRAM tradeoff.
Numbers are estimates from model configs and published file sizes, not measurements. How we calculate.
FAQ
How do I calculate KV cache size?
Multiply 2 × layers × KV heads × head dimension × tokens × parallel sequences × bytes per value (2 for FP16). For Qwen2.5-7B at 8K tokens that's 469,762,048 bytes, or 0.44 GiB.
Should I use attention heads or KV heads?
KV heads. Most current models use grouped-query attention, where several query heads share one KV head. Using the query-head count for Qwen2.5-7B would overstate its cache 7×.
Does doubling the context double the KV cache?
For standard attention, yes. Models with sliding-window or hybrid layers grow more slowly; Gemma 3 and Qwen3.8 are examples.
Numbers from calculator 0.3.0, model data 2026-09-20.1. Sources last checked 2026-09-21.