The KV cache of a 70B Llama model at 131,072 tokens is 40.00 GiB, roughly a third of its BF16 weights. The KV cache of Gemma 3 27B at the same length is 10.41 GiB, even though Gemma 3 27B has 62 layers. This guide lists the numbers for the models people ask about most, and explains the one architectural switch that separates the two curves.
All values are FP16 KV for one sequence, calculated by the calculator 0.3.0 kv.ts and sliding-mixed.ts adapters from official architecture values. They are Formula Estimates. No row was measured on an inference engine.
The formula and where its inputs come from
For each layer that caches the full context:
2 (K and V) × kvHeads × headDim × tokens × 2 bytes
Layers, KV heads and head dimension came from these places:
- Llama 3.x: Meta’s
llama-modelsrepository,sku_list.pyat commit0e0b8c5(n_layers,n_kv_heads,dim,n_heads). The head dimension isdim ÷ n_heads, because the file has no separate key. - Gemma 3: Google’s
gemma_pytorchrepository,config.pyat commitcb7c015(layer count, KV heads, head dimension, sliding window, and the layer pattern of five local layers per global layer). - Qwen3, Mistral, phi-4, gpt-oss: each model’s
config.json.
The Hugging Face config.json files of the Llama and Gemma repositories are gated, so the Llama and Gemma values were read from the publishers’ code and cross-checked against the metadata header of a GGUF file for each size (a non-official file used only for comparison). All of them matched.
KV cache in GiB, FP16, 1 sequence
| Model | Layout | Native window (catalog) | 4,096 | 8,192 | 32,768 | 131,072 |
|---|---|---|---|---|---|---|
| Llama 3.2 1B Instruct | 16 full layers, 8 KV heads × 64 | 131,072 | 0.13 | 0.25 | 1.00 | 4.00 |
| Llama 3.2 3B Instruct | 28 full layers, 8 KV heads × 128 | 131,072 | 0.44 | 0.88 | 3.50 | 14.00 |
| Llama 3.1 8B Instruct | 32 full layers, 8 KV heads × 128 | 131,072 | 0.50 | 1.00 | 4.00 | 16.00 |
| Llama 3.3 70B Instruct | 80 full layers, 8 KV heads × 128 | 131,072 | 1.25 | 2.50 | 10.00 | 40.00 |
| Gemma 3 1B IT | 4 full + 22 sliding (window 512), 1 KV heads × 256 | 32,768 | 0.03 | 0.04 | 0.14 | n/a |
| Gemma 3 4B IT | 5 full + 29 sliding (window 1024), 4 KV heads × 256 | 131,072 | 0.19 | 0.27 | 0.74 | 2.61 |
| Gemma 3 12B IT | 8 full + 40 sliding (window 1024), 8 KV heads × 256 | 131,072 | 0.56 | 0.81 | 2.31 | 8.31 |
| Gemma 3 27B IT | 10 full + 52 sliding (window 1024), 16 KV heads × 128 | 131,072 | 0.72 | 1.03 | 2.91 | 10.41 |
| Qwen3-8B | 36 full layers, 8 KV heads × 128 | 40,960 | 0.56 | 1.13 | 4.50 | n/a |
| Qwen3-14B | 40 full layers, 8 KV heads × 128 | 40,960 | 0.63 | 1.25 | 5.00 | n/a |
| Qwen3-32B | 64 full layers, 8 KV heads × 128 | 40,960 | 1.00 | 2.00 | 8.00 | n/a |
| Qwen3-30B-A3B | 48 full layers, 4 KV heads × 128 | 40,960 | 0.38 | 0.75 | 3.00 | n/a |
| gpt-oss-20b | 12 full + 12 sliding (window 128), 8 KV heads × 64 | 131,072 | 0.10 | 0.19 | 0.75 | 3.00 |
| gpt-oss-120b | 18 full + 18 sliding (window 128), 8 KV heads × 64 | 131,072 | 0.15 | 0.29 | 1.13 | 4.50 |
| Mistral-Small-24B-Instruct-2501 | 40 full layers, 8 KV heads × 128 | 32,768 | 0.63 | 1.25 | 5.00 | n/a |
| phi-4 | 40 full layers, 10 KV heads × 128 | 16,384 | 0.78 | 1.56 | n/a | n/a |
“n/a” means the context is above the native window recorded in the catalog. Qwen3 rows stop at the 40,960 max_position_embeddings in each config, and the dataset has no extended-context variants. Gemma 3 1B lists 32,768, and phi-4 lists 16,384.
Full-attention models grow in a straight line
Every Llama and Qwen3 row doubles when the context doubles, because every layer caches every token. The slope is set by layers × KV heads × head dimension:
| Model | Layers × KV heads × head dim | Bytes per token | KiB per token |
|---|---|---|---|
| Llama 3.2 1B Instruct | 16 × 8 × 64 | 32,768 | 32 |
| Llama 3.2 3B Instruct | 28 × 8 × 128 | 114,688 | 112 |
| Llama 3.1 8B Instruct | 32 × 8 × 128 | 131,072 | 128 |
| Qwen2.5-7B-Instruct | 28 × 4 × 128 | 57,344 | 56 |
| Qwen3-32B | 64 × 8 × 128 | 262,144 | 256 |
| Llama 3.3 70B Instruct | 80 × 8 × 128 | 327,680 | 320 |
Llama 3.1 8B and Qwen3-8B have the same eight KV heads of width 128, so the difference between them (128 against 144 KiB per token) is the layer count, 32 against 36. Qwen2.5-7B costs 56 KiB per token, less than half of Llama 3.1 8B, because it has 4 KV heads. Llama 3.3 70B has the same 8 KV heads as the 8B model and 80 layers instead of 32, so its cost per token is 2.5 times higher. A larger model therefore does not always carry a proportionally larger cache.
Sliding-window layers flatten the curve
Gemma 3 mixes local and global attention. In the 27B model, 10 layers are global and 52 are local with a 1,024-token window (the 1B model uses 512). A local layer needs at most window × KV cost, however long the prompt is. The calculator charges each global layer for the full context and each local layer for min(context, window).
For Gemma 3 27B IT that gives two parts: global layers add 81,920 bytes per token, and the local layers saturate at 436,207,616 bytes (416 MiB) once the context passes 1,024. At 131,072 tokens the two parts sum to 10.41 GiB. If all 62 layers kept the full length, the same model would need 62.00 GiB. For Gemma 3 12B the windowed figure is 8.31 GiB against 48.00 GiB.
gpt-oss alternates local and global layers with a window of 128, according to layer_types and sliding_window in its config. For gpt-oss-20b the windowed estimate at 131,072 tokens is 3.00 GiB, against 6.00 GiB with full-length layers, because half of the layers are global. The gpt-oss guide shows how the smaller cache combines with the weight file.
Gemma 3 4B against Qwen3-8B at long context
| Context | Gemma 3 4B IT | Qwen3-8B |
|---|---|---|
| 8,192 tokens | 0.27 GiB | 1.13 GiB |
| 32,768 tokens | 0.74 GiB | 4.50 GiB |
| 131,072 tokens | 2.61 GiB | n/a (native 40,960) |
At 8,192 tokens the gap is 0.86 GiB. At 32,768 tokens it is 3.76 GiB. Weights and extras still add on top; see how much VRAM a local LLM needs.
What changes the numbers
- Several sequences. Multiply by the number of concurrent sequences. Four 8,192-token requests on Llama 3.1 8B hold 4.00 GiB of KV.
- KV quantization. The calculator counts q8_0 KV at 1 byte per element plus 6.25% block overhead, which is about 53% of FP16, and q4_0 at 0.5 bytes. Quality effects are not estimated here.
- Engine behavior. The sliding-window rows assume the engine stores local layers at window length. An engine that allocates every layer at full length uses the full-length figures instead. I have not measured which engines do what for Gemma 3 or gpt-oss, so treat the windowed numbers as the lower bound.
- Paged or pre-allocated caches. Servers such as vLLM reserve a fraction of GPU memory for cache blocks, so reported usage can exceed the formula even with a short prompt.
To try your own context length, open the KV cache calculator. The older context length and KV cache guide explains the formula with a Qwen2.5 example. Model pages with the full architecture records: Llama 3.1 8B, Llama 3.3 70B, Gemma 3 27B and gpt-oss-20b.
FAQ
How big is the KV cache for Llama 3.1 8B?
128 KiB per token in FP16: 1.00 GiB at 8,192 tokens, 4.00 GiB at 32,768 and 16.00 GiB at 131,072, for one sequence. Architecture values come from Meta's llama-models repository.
Why is Gemma 3 KV cache smaller than Llama's at the same context?
Gemma 3 27B keeps a full-length cache in 10 of its 62 layers and a 1,024-token window in the other 52. At 131,072 tokens the calculator gives 10.41 GiB, against 62.00 GiB if every layer kept the full length.