Can an H100 run Qwen3-32B?
Just barely Qwen3-32B at BF16 needs 69.65 GiB–71.76 GiB at 32,768 tokens of context. The H100 has about 78 GiB usable.
17 files, 65.5 GB in total.
These are estimates from the model's config and file sizes, not measurements. How we calculate
What to change
- Shorten the context to about 25,600 tokens for a clean fit.
- Use GGUF Q8_0 (Qwen) instead: about 42.1 GiB at the same context (Runs comfortably).
- Close other apps that hold GPU memory, or lower the reserve if the machine runs headless.
Other files on this GPU
| Quant | File size | Engine | Total | Fit |
|---|---|---|---|---|
| BF16 safetensors This page | 65.5 GB | vllm | 69.6 GiB–71.8 GiB | Tight fit |
| GGUF Q8_0 (Qwen) | 34.8 GB | llama.cpp | 40.6 GiB–42.1 GiB | Runs comfortably |
| GGUF Q6_K (Qwen) | 26.9 GB | llama.cpp | 33.2 GiB–34.6 GiB | Runs comfortably |
| GGUF Q5_K_M (Qwen) | 23.2 GB | llama.cpp | 29.7 GiB–31.1 GiB | Runs comfortably |
| GGUF Q4_K_M | 19.8 GB | llama.cpp | 26.5 GiB–27.8 GiB | Runs comfortably |
Same context (32,768 tokens), one chat at a time, each file with its usual engine.
Settings used
- ModelQwen3-32B
- HardwareNVIDIA H100 80GB SXM
- QuantizationBF16 safetensors
- Context length (cached tokens)32,768
- Engine
vllm - Concurrent sequences1
- KV cache typefp16
Technical identifiers and share data
- Model
qwen3-32b - Hardware
nvidia-h100-80gb-sxm - Quantization
bf16 - Share fragment for this identity
v2.eyJzY2hlbWFWZXJzaW9uIjoyLCJjYWxjdWxhdG9yVmVyc2lvbiI6IjAuMy4wIiwiZGF0YXNldFZlcnNpb24iOiIyMDI2LTEwLTAyLjEiLCJjb25maWciOnsibW9kZWxJZCI6InF3ZW4zLTMyYiIsImhhcmR3YXJlSWQiOiJudmlkaWEtaDEwMC04MGdiLXN4bSIsImRldmljZUNvdW50IjoxLCJxdWFudGl6YXRpb25JZCI6ImJmMTYiLCJjb250ZXh0VG9rZW5zIjozMjc2OCwiY29uY3VycmVudFNlcXVlbmNlcyI6MSwiZW5naW5lIjoidmxsbSIsImt2VHlwZSI6ImZwMTYifX0
How this number is calculated
Total = weight file + KV cache + runtime buffers. The range spans low to high runtime overhead for the engine shown.
- Weights (file)
- 61.02 GiB (65,524,246,528 bytes)
- KV cache
- 8.00 GiB (8,589,934,592 bytes)
- Runtime buffers
- 0.63 GiB–2.74 GiB
- Total
- 69.65 GiB–71.76 GiB
- Usable memory
- 78.00 GiB
- OS / display reserve
- 2.00 GiB
- Left over
- 6.24 GiB–8.35 GiB
71.76 GiB high estimate vs 78.00 GiB net
Enough memory does not mean this engine, quant, and OS will run. Support does not mean it will be fast.
- Requested context is within the recorded native window of 40960 tokens.
- Full-resident weight estimates use total parameters, including inactive MoE experts.
- Weight bytes come from the selected weight files only, not from every file in the repository.
- Download file size is not peak GPU memory. Runtime layout, allocator padding, and KV are extra.
- Weight bytes come from 17 published safetensors shards totaling 65524246528 bytes.
- Context budget is the number of cached tokens per sequence (prompt plus reserved generation).
- Concurrent sequences share one model replica; weights are not multiplied by sequence count.
- KV dtype is independent of weight quantization unless you change it.
Related pages
Sources
- Qwen3-32B model card
- config.json — revision 9216db5781bf21249d130ec9da846c4624c16137
- safetensors index — metadata.total_size=65524246528
- Artifact repo Qwen/Qwen3-32B-GGUF
- NVIDIA H100 product page — H100 SXM GPU Memory 80GB
Calculator
This combo page keeps the route model and device. Customize to change them on the main calculator; the share then keeps that identity.
Loading catalog snapshot…
Pick a model to see how much memory it needs.