Qwen2.5-72B-Instruct VRAM requirements

Qwen2.5-72B-Instruct needs about 45 GiB of GPU memory at Q4_K_M with an 8K context. That is tight on 48 GB cards like the L40S; 80 GB cards like the A100 have room for it. At 32K context it needs about 53 GiB, which fits 80 GB cards.

These are estimates from the model's config and file sizes, not measurements. How we calculate

VRAM by quantization

QuantFile sizeTotal at 8KTotal at 32K
BF16 safetensors145.4 GB138.6 GiB–140.5 GiB146.1 GiB–148.2 GiB
GGUF Q8_0 (Qwen)77.5 GB74.8 GiB–76.8 GiB82.3 GiB–84.8 GiB
GGUF Q6_K (Qwen)59.9 GB58.4 GiB–60.0 GiB65.9 GiB–68.1 GiB
GGUF Q5_K_M (Qwen)51.7 GB50.8 GiB–52.3 GiB58.3 GiB–60.3 GiB
GGUF Q4_K_M44.0 GB43.6 GiB–45.0 GiB51.1 GiB–53.0 GiB

Names like Q4_K_M and Q8_0 are quantization levels: fewer bits per weight means a smaller file and a small loss of quality. Totals add the KV cache (memory that grows with context length) and runtime buffers, for one chat at a time. Download files

Which GPUs can run it

MemoryUsableExamplesAt 8KAt 32K
8 GB7.0 GiBRTX 4060, RTX 3060, RTX 5060 Ti +1 moreDoes not fitDoes not fit
10 GB9.0 GiBRTX 3080Does not fitDoes not fit
12 GB11.0 GiBRTX 4070, RTX 3060, RTX 5070 +2 moreDoes not fitDoes not fit
16 GB14.0 GiBRTX 4080 SUPER, RTX 5080, RX 7800 XT +5 moreDoes not fitDoes not fit
16 GB15.0 GiBRTX 4060 Ti, RTX 4070 Ti SUPERDoes not fitDoes not fit
20 GB18.0 GiBRX 7900 XTDoes not fitDoes not fit
24 GB22.0 GiBRTX 4090, RX 7900 XTX, RTX 3090Does not fitDoes not fit
32 GB30.0 GiBRTX 5090Does not fitDoes not fit
48 GB46.0 GiBL40STight fitDoes not fit
80 GB78.0 GiBA100, H100Runs comfortablyRuns comfortably
64 GB · unified54.0 GiBApple M2 MaxFitsTight fit
128 GB · unified96.0 GiBRyzen AI Max+ 395Runs comfortablyRuns comfortably
128 GB · unified112.0 GiBDGX Spark (GB10)Runs comfortablyRuns comfortably
128 GB · unified116.0 GiBApple M4 MaxRuns comfortablyRuns comfortably

GGUF Q4_K_M, one chat at a time. Usable memory is the total minus a default reserve for the OS and display.

Detailed checks

Architecture

The KV cache grows with context: about 2.5 GiB at 8K and 10 GiB at 32K in FP16. The model has 8 KV heads for 64 query heads (grouped-query attention), which keeps that cache small.

FAQ

Is the download size the same as VRAM use?

No. The Q4_K_M file is about 41 GiB, but with an 8K context the total is about 45 GiB, because the KV cache and runtime buffers sit on top of the weights.

How much memory does a longer context add?

For Qwen2.5-72B-Instruct the KV cache is about 2.5 GiB at 8K tokens and 10 GiB at 32K in FP16. The weights stay the same size.

Which quantization should I use?

Start with Q4_K_M (about 45 GiB at 8K). If you have the memory, BF16 needs about 140.5 GiB and stays closer to the original weights.

Download files

QuantFilesFile sizeEngine
BF16 safetensors
37 safetensors shards
First shard: model-00001-of-00037.safetensors
Last shard: model-00037-of-00037.safetensors
View all 37 shard files
  • model-00001-of-00037.safetensors
  • model-00002-of-00037.safetensors
  • model-00003-of-00037.safetensors
  • model-00004-of-00037.safetensors
  • model-00005-of-00037.safetensors
  • model-00006-of-00037.safetensors
  • model-00007-of-00037.safetensors
  • model-00008-of-00037.safetensors
  • model-00009-of-00037.safetensors
  • model-00010-of-00037.safetensors
  • model-00011-of-00037.safetensors
  • model-00012-of-00037.safetensors
  • model-00013-of-00037.safetensors
  • model-00014-of-00037.safetensors
  • model-00015-of-00037.safetensors
  • model-00016-of-00037.safetensors
  • model-00017-of-00037.safetensors
  • model-00018-of-00037.safetensors
  • model-00019-of-00037.safetensors
  • model-00020-of-00037.safetensors
  • model-00021-of-00037.safetensors
  • model-00022-of-00037.safetensors
  • model-00023-of-00037.safetensors
  • model-00024-of-00037.safetensors
  • model-00025-of-00037.safetensors
  • model-00026-of-00037.safetensors
  • model-00027-of-00037.safetensors
  • model-00028-of-00037.safetensors
  • model-00029-of-00037.safetensors
  • model-00030-of-00037.safetensors
  • model-00031-of-00037.safetensors
  • model-00032-of-00037.safetensors
  • model-00033-of-00037.safetensors
  • model-00034-of-00037.safetensors
  • model-00035-of-00037.safetensors
  • model-00036-of-00037.safetensors
  • model-00037-of-00037.safetensors
145.41 GB
145,412,407,296 bytes
vllm, sglang
GGUF Q8_0 (Qwen)
21 safetensors shards
First shard: qwen2.5-72b-instruct-q8_0-00001-of-00021.gguf
Last shard: qwen2.5-72b-instruct-q8_0-00021-of-00021.gguf
View all 21 shard files
  • qwen2.5-72b-instruct-q8_0-00001-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00002-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00003-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00004-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00005-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00006-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00007-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00008-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00009-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00010-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00011-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00012-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00013-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00014-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00015-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00016-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00017-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00018-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00019-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00020-of-00021.gguf
  • qwen2.5-72b-instruct-q8_0-00021-of-00021.gguf
77.53 GB
77,530,000,800 bytes
llama.cpp, ollama
GGUF Q6_K (Qwen)
16 safetensors shards
First shard: qwen2.5-72b-instruct-q6_k-00001-of-00016.gguf
Last shard: qwen2.5-72b-instruct-q6_k-00016-of-00016.gguf
View all 16 shard files
  • qwen2.5-72b-instruct-q6_k-00001-of-00016.gguf
  • qwen2.5-72b-instruct-q6_k-00002-of-00016.gguf
  • qwen2.5-72b-instruct-q6_k-00003-of-00016.gguf
  • qwen2.5-72b-instruct-q6_k-00004-of-00016.gguf
  • qwen2.5-72b-instruct-q6_k-00005-of-00016.gguf
  • qwen2.5-72b-instruct-q6_k-00006-of-00016.gguf
  • qwen2.5-72b-instruct-q6_k-00007-of-00016.gguf
  • qwen2.5-72b-instruct-q6_k-00008-of-00016.gguf
  • qwen2.5-72b-instruct-q6_k-00009-of-00016.gguf
  • qwen2.5-72b-instruct-q6_k-00010-of-00016.gguf
  • qwen2.5-72b-instruct-q6_k-00011-of-00016.gguf
  • qwen2.5-72b-instruct-q6_k-00012-of-00016.gguf
  • qwen2.5-72b-instruct-q6_k-00013-of-00016.gguf
  • qwen2.5-72b-instruct-q6_k-00014-of-00016.gguf
  • qwen2.5-72b-instruct-q6_k-00015-of-00016.gguf
  • qwen2.5-72b-instruct-q6_k-00016-of-00016.gguf
59.86 GB
59,861,035,808 bytes
llama.cpp, ollama
GGUF Q5_K_M (Qwen)
14 safetensors shards
First shard: qwen2.5-72b-instruct-q5_k_m-00001-of-00014.gguf
Last shard: qwen2.5-72b-instruct-q5_k_m-00014-of-00014.gguf
View all 14 shard files
  • qwen2.5-72b-instruct-q5_k_m-00001-of-00014.gguf
  • qwen2.5-72b-instruct-q5_k_m-00002-of-00014.gguf
  • qwen2.5-72b-instruct-q5_k_m-00003-of-00014.gguf
  • qwen2.5-72b-instruct-q5_k_m-00004-of-00014.gguf
  • qwen2.5-72b-instruct-q5_k_m-00005-of-00014.gguf
  • qwen2.5-72b-instruct-q5_k_m-00006-of-00014.gguf
  • qwen2.5-72b-instruct-q5_k_m-00007-of-00014.gguf
  • qwen2.5-72b-instruct-q5_k_m-00008-of-00014.gguf
  • qwen2.5-72b-instruct-q5_k_m-00009-of-00014.gguf
  • qwen2.5-72b-instruct-q5_k_m-00010-of-00014.gguf
  • qwen2.5-72b-instruct-q5_k_m-00011-of-00014.gguf
  • qwen2.5-72b-instruct-q5_k_m-00012-of-00014.gguf
  • qwen2.5-72b-instruct-q5_k_m-00013-of-00014.gguf
  • qwen2.5-72b-instruct-q5_k_m-00014-of-00014.gguf
51.67 GB
51,673,983,488 bytes
llama.cpp, ollama
GGUF Q4_K_M
12 safetensors shards
First shard: qwen2.5-72b-instruct-q4_k_m-00001-of-00012.gguf
Last shard: qwen2.5-72b-instruct-q4_k_m-00012-of-00012.gguf
View all 12 shard files
  • qwen2.5-72b-instruct-q4_k_m-00001-of-00012.gguf
  • qwen2.5-72b-instruct-q4_k_m-00002-of-00012.gguf
  • qwen2.5-72b-instruct-q4_k_m-00003-of-00012.gguf
  • qwen2.5-72b-instruct-q4_k_m-00004-of-00012.gguf
  • qwen2.5-72b-instruct-q4_k_m-00005-of-00012.gguf
  • qwen2.5-72b-instruct-q4_k_m-00006-of-00012.gguf
  • qwen2.5-72b-instruct-q4_k_m-00007-of-00012.gguf
  • qwen2.5-72b-instruct-q4_k_m-00008-of-00012.gguf
  • qwen2.5-72b-instruct-q4_k_m-00009-of-00012.gguf
  • qwen2.5-72b-instruct-q4_k_m-00010-of-00012.gguf
  • qwen2.5-72b-instruct-q4_k_m-00011-of-00012.gguf
  • qwen2.5-72b-instruct-q4_k_m-00012-of-00012.gguf
44.01 GB
44,010,465,472 bytes
llama.cpp, ollama
How this number is calculated

Total = weight file + KV cache + runtime buffers. The range spans low to high runtime overhead for the engine shown.

GGUF Q4_K_M · 8,192 tokens · llama.cpp

Weights (file)
40.99 GiB (44,010,465,472 bytes)
KV cache
2.50 GiB (2,684,354,560 bytes)
Runtime buffers
0.13 GiB–1.49 GiB
Total
43.61 GiB–44.98 GiB
  • Requested context is within the recorded native window of 32768 tokens.
  • Full-resident weight estimates use total parameters, including inactive MoE experts.
  • Weight bytes come from the selected weight files only, not from every file in the repository.
  • Download file size is not peak GPU memory. Runtime layout, allocator padding, and KV are extra.
  • Weight bytes come from 12 published safetensors shards totaling 44010465472 bytes.
  • Context budget is the number of cached tokens per sequence (prompt plus reserved generation).
  • Concurrent sequences share one model replica; weights are not multiplied by sequence count.
  • KV dtype is independent of weight quantization unless you change it.

How we calculate

Open the calculator with this model

Related pages

Sources

Calculator

Loading catalog snapshot…

Context length

Longer context needs more memory for the KV cache, the model's working memory for the conversation.

More options
Quantization
Runtime

Engine and KV cache type for this estimate.

Workload

Overhead scenario and the prefill workspace.

Topology

How the model is split across devices. Memory is not added together.

Replica count is an advanced deployment option. Quantity is nodes or GPUs, not replicas.

Budget

System reserve and safety margin. An empty number keeps the previous value.

GiB (2^30 bytes)

%

Custom model

Import a config or type the architecture. Nothing is scanned until you import.

Parsed in the browser. The file is not uploaded.

Only huggingface.co, config.json, no token. Private repos are rejected.

Pick a model to see how much memory it needs.