KV cache calculator: how to calculate KV cache size
KV cache size = 2 × layers × KV heads × head dim × tokens × sequences × bytes per element. The 2 is for keys and values, and FP16 uses 2 bytes per element.
Qwen2.5-7B-Instruct has 28 layers, 4 KV heads and a head dim of 128. At 8,192 tokens in FP16:2 × 28 × 4 × 128 × 8,192 × 1 × 2 = 469,762,048 bytes (0.4375 GiB). At 32,768 tokens it is 1,879,048,192 bytes (1.7500 GiB). The model weights stay the same size.
These are estimates from each model's config, not measurements. How we calculate.
Calculate it for a model
2 × 28 × 4 × 128 × 8192 × 1 × 2
469,762,048 bytes
- Attention: gqa. Calculator support: full. GQA formula applies.
- Query heads 28 versus KV heads 4. Using query heads would count 3.06 GiB instead.
- Context classification: native. Native window 32,768. No extended window is recorded.
- This result is an estimate, not a measured trace.
What each term means
Use the KV head count from num_key_value_heads in the model's config.json. Most current models use grouped-query attention, so they have far fewer KV heads than query heads; plugging in the query heads overstates the cache several times over.
Worked GQA examples
| Model | Layers × KV heads × head dim | 8,192 FP16 | 32,768 FP16 |
|---|---|---|---|
| Qwen2.5-7B-Instruct | 28 × 4 × 128 | 469,762,048 (0.4375 GiB) | 1,879,048,192 (1.7500 GiB) |
| Qwen2.5-14B-Instruct | 48 × 8 × 128 | 1,610,612,736 (1.5000 GiB) | 6,442,450,944 (6.00 GiB) |
| Qwen2.5-32B-Instruct | 64 × 8 × 128 | 2,147,483,648 (2.00 GiB) | 8,589,934,592 (8.00 GiB) |
| Qwen3-8B | 36 × 8 × 128 | 1,207,959,552 (1.1250 GiB) | 4,831,838,208 (4.5000 GiB) |
One sequence, FP16. More concurrent sequences multiply the KV cache, not the weights. A 4-bit model does not get a 4-bit KV cache unless you set the KV cache type separately.
Models the basic formula does not cover
- Sliding-window layers (gpt-oss, Gemma 3): those layers only keep the last window of tokens. The calculator uses
2 × kvHeads × headDim × sequences × bytesPerElement × (fullAttentionLayers × context + slidingAttentionLayers × min(context, slidingWindow)). - MLA (DeepSeek-V2/V3): the cache holds a compressed latent, so the KV-head formula does not apply.
- Hybrid layers (Qwen3.8): only the attention layers keep KV. The other layers hold a fixed state,
sequences × layers × V_heads × K_dim × V_dim × state_dtype, that does not grow with context.
| Model | Attention layers | KV at 8,192 | KV at 32,768 | Fixed state |
|---|---|---|---|---|
| Qwen3.8-27B | 16 of 64 | 536,870,912 | 2,147,483,648 | 150,994,944 |
| Qwen3.8-Flash-Next | 12 of 48 | 201,326,592 | 805,306,368 | 56,623,104 |
See Qwen3.8-27B VRAM and Qwen3.8-Flash-Next VRAM for the full totals, or the DGX Spark page for a 128 GB unified-memory budget.
FAQ
How do you calculate KV cache size?
Multiply 2 × layers × kvHeads × headDim × cachedTokens × sequences × bytesPerElement. For Qwen2.5-7B-Instruct at 8,192 tokens in FP16 (2 bytes per element) that is 469,762,048 bytes.
How big is the Qwen2.5-7B KV cache at 8K and 32K tokens?
469,762,048 bytes at 8,192 tokens and 1,879,048,192 bytes at 32,768 tokens, for one sequence in FP16. The weights do not change.
Do quantized weights shrink the KV cache?
No. A Q4 model still stores its KV cache in FP16 unless you also set a quantized KV cache type (for example FP8 in vLLM or q8_0 in llama.cpp).
Does this formula work for DeepSeek or Qwen3.8 models?
Not directly. DeepSeek's MLA attention stores a compressed latent instead of full keys and values. Qwen3.8 hybrids keep KV only on their attention layers, plus a fixed-size state that does not grow with context.