How much VRAM do you need to run an LLM locally?

Local LLM VRAM requirements come down to three things: the weight file, the KV cache for your context, and some runtime overhead. For Qwen2.5-7B-Instruct at 4-bit (Q4_K_M) with an 8K context, that adds up to 4.92 to 5.42 GiB, so it fits an 8 GB card. The same model unquantized (BF16) needs 14.75 to 15.44 GiB. A 27B at 4-bit needs about 18 GiB, and a 70B in BF16 about 137 GiB.

Quick sizes

One sequence, FP16 KV cache, llama.cpp. Then compare the high end with what your card has free:

The usable figure is the card minus a reserve for the desktop and driver: 1 GiB on most 8 and 12 GB cards, 2 GiB on most 16, 24 and 32 GB cards. Full per-tier tables are in what fits on 12, 16 and 24 GB.

Part 1: weights, from the file size

For BF16, bytes are close to parameters × 2. Llama 3.1 8B Instruct has 8,030,261,248 parameters, so 2 bytes each is 16,060,522,496; the four shards add up to 16,060,556,376 bytes, 33,880 bytes more because of shard headers. For quantized files, the parameter count tells you much less. Read the actual file size.

Qwen2.5-7B comes from its model.safetensors.index.json (revision a09a35458c702b33eeacc393d103063234e8bc28). The Llama and Gemma BF16 rows are sums of the safetensors sizes in the Hugging Face listings, because those repos are gated. The Gemma Q4_0 files are Google’s quantization-aware-trained GGUFs. Meta doesn’t publish GGUFs for Llama 3.x, so there’s no Llama GGUF row. Why 4-bit files are bigger than you’d expect: 4-bit vs 8-bit vs FP16.

Part 2: the KV cache grows with every token

For standard grouped-query-attention models, the FP16 cache is:

2 × layers × KV heads × head dim × tokens × sequences × 2 bytes

So each model has a fixed cost per token:

Llama values come from Meta’s llama-models repository (sku_list.py, commit 0e0b8c5), with head dim = dim ÷ n_heads. Qwen values come from each config.json.

At 8K tokens, Llama 3.1 8B needs 1.00 GiB of KV and Llama 3.3 70B 2.50 GiB. At 32K that’s 4.00 and 10.00 GiB. Gemma 3 grows more slowly because five of every six layers keep only a 1,024-token window; see KV cache size by model. A Q4_K_M weight file doesn’t shrink this term; the cache stays FP16 unless you set it otherwise (-ctk / -ctv in llama.cpp). See FP16 vs Q8_0 vs Q4_0 KV cache.

Part 3: runtime overhead

Engines need working memory on top of weights and cache. We use a range rather than a fixed multiplier. For Gemma 3 27B Q4_0 on an RTX 4090 at 8K with llama.cpp, the activation workspace is 67,108,864 to 690,538,804 bytes and the engine reserve 67,108,864 to 268,435,456 bytes. vLLM reserves more; see llama.cpp vs vLLM vs Ollama.

How well does this match reality? We have three real measurements: llama.cpp running Qwen2.5-3B-Instruct Q4_K_M on an RTX 4090 (calibrations rt-qwen2.5-3b-instruct-q4km-4090-ctx512, -ctx2048, -ctx8192). Process GPU memory was 2.31, 2.37 and 2.59 GiB at 512, 2,048 and 8,192 tokens. The estimated ranges were 2.10 to 2.52, 2.16 to 2.57 and 2.37 to 2.80 GiB, so each measurement landed inside. That’s one model on one card, so treat it as a sanity check, not proof. We haven’t measured any Llama or Gemma model.

Big models

Llama 3.3 70B in BF16 doesn’t fit one 80 GB H100 (78 GiB usable) or one DGX Spark (112 GiB usable); the weights alone are 131.42 GiB. Separate GPUs don’t pool memory by default, so two 24 GB cards aren’t one 48 GB card. See two GPUs and unified memory.

The largest open mixture-of-experts models are far beyond one consumer card. Ling-3.1-flash has an official mixed-FP8 index of 573,753,281,152 bytes, and its model card lists 8 × H200 or 4 × GB300. Muse Spark Open has BF16 shards totaling 1,644,333,393,864 bytes and NVFP4 shards totaling 533,011,023,472 bytes; Meta’s card states at least 2 TB of VRAM for BF16 and 700 GB for NVFP4. Every expert has to be in memory even though only a few run per token: MoE models load every expert.

Size any model in four steps

1. Pick the exact file you’ll download and note its size.
2. Work out the KV cache at the context you’ll really use, times the number of parallel requests.
3. Add the overhead range for your engine.
4. Compare the high end with your card’s size minus the reserve, then check it in the calculator with that model and card.

Model pages for the examples above: Qwen2.5-7B-Instruct, Llama 3.1 8B Instruct, Gemma 3 27B IT, Llama 3.3 70B Instruct. If a model fits on paper and still runs out of memory, read why file size isn’t the VRAM you need.

Numbers are estimates from model configs and published file sizes, not measurements, except where measured runs are named. How we calculate.

Hybrid layouts are not the GQA table

Qwen3.8-27B and Qwen3.8-Flash-Next mix a few attention layers with Gated DeltaNet layers. Only the attention layers keep a KV cache that grows with context; the Gated DeltaNet layers keep a fixed-size state. Applying the KV formula above to every layer would overstate their cache. Being mixture-of-experts doesn’t shrink the Flash-Next file either. The table below shows the calculator’s split for both, at 8K and 32K, one sequence, FP16 KV cache. We haven’t measured either model. More in hybrid attention VRAM, on the Qwen3.8-27B and Qwen3.8-Flash-Next pages, or in the KV cache calculator.

Numbers from calculator 0.3.0, model data 2026-10-02.1. Sources last checked 2026-10-28.