Qwen2.5-72B-Instruct official GGUF Q4_K_M is 44,010,465,472 bytes. One GeForce RTX 4090, vLLM, 8,192 tokens, 1 sequence, FP16 KV, is 44.11 to 46.06 GiB, insufficient against a 22.00 GiB net budget. Two RTX 4090s as independent replicas still each need that full copy. llama.cpp tensor parallel reports 21.87 to 23.24 GiB per device (tight) and leaves the unsplit total at 43.61 to 44.98 GiB. Formula estimates, calculator 0.3.0, dataset 2026-10-02.1. Software support is unverified. Tokens per second are Unknown. A separate NCCL or PCIe buffer measurement is Unknown. The split below is the calculator’s even shard of weights and KV.
The rule, also on two GPUs and unified memory: device capacities are not added. Fit is judged per device after the split you asked for.
Qwen2.5-72B Q4_K_M, 8,192 tokens, RTX 4090
Net budget 22.00 GiB on every row. The vLLM single-device and replica rows use the vLLM engine reserve. The llama.cpp tensor-parallel row uses the llama.cpp reserve, so its unsplit total is not the same number as the vLLM unsplit total.
| Layout | Per-device total GiB | Unsplit total GiB | Per-device weights GiB | Per-device KV GiB | Verdict |
|---|---|---|---|---|---|
| 1× vLLM | 44.11 to 46.06 | 44.11 to 46.06 | 40.99 | 2.50 | insufficient |
| 2× replicas, vLLM | 44.11 to 46.06 | 44.11 to 46.06 | 40.99 | 2.50 | insufficient |
| 2× tensor parallel, vLLM | 22.37 to 24.32 | 44.11 to 46.06 | 20.49 | 1.25 | insufficient |
| 2× tensor parallel, llama.cpp | 21.87 to 23.24 | 43.61 to 44.98 | 20.49 | 1.25 | tight |
vLLM tensor parallel halves weights and KV and still keeps a 0.50 to 2.00 GiB engine reserve on each device, so the per-device low end is 22.37 GiB, over the 22.00 GiB budget. llama.cpp tensor parallel uses a smaller reserve scenario. Its low end, 21.87 GiB, is inside the budget and its high end, 23.24 GiB, is over it, so the verdict is tight. Tight is not a measured multi-GPU run. This repository has no calibration for 72B on two 4090s.
KV heads for this model are 8, which divides by 2, so the calculator shards KV. If the head count did not divide evenly, KV would stay replicated on each rank. That case is not this file.
Two 8 GB cards still miss 32B
Qwen2.5-32B-Instruct Q4_K_M, 19,851,336,384 bytes, vLLM, 8,192 tokens. Each RTX 4060 8GB has a 7.00 GiB net budget.
| Layout | Per-device total GiB | Verdict |
|---|---|---|
| 1× 4060 | 21.11 to 23.05 | insufficient |
| 2× tensor parallel | 10.87 to 12.80 | insufficient |
Halving the 32B file still leaves more than 7.00 GiB on each 8 GB card. The same Q4 file on one RTX 4090, llama.cpp, is the tight 20.61 to 21.50 GiB row on Q4 vs Q5 vs Q6 vs Q8.
Engine reserves without a second GPU are on llama.cpp vs vLLM vs Ollama. Model pages: Qwen2.5-72B-Instruct, Qwen2.5-32B-Instruct, RTX 4090.
FAQ
Do two 24 GB RTX 4090s make 48 GB of VRAM?
No. The calculator keeps a 22.00 GiB net budget on each card. Qwen2.5-72B Q4_K_M as two independent replicas still needs 44.11 to 46.06 GiB on each card under the vLLM scenario.
What does tensor parallel change?
llama.cpp tensor parallel on two RTX 4090s reports 21.87 to 23.24 GiB per device for that 72B Q4 file at 8,192 tokens, verdict tight. The unsplit total stays 43.61 to 44.98 GiB. Weights and KV are split evenly. The engine reserve stays on each device.
Will two 8 GB cards run Qwen2.5-32B Q4?
Not in the vLLM tensor-parallel scenario. Each RTX 4060 8GB has a 7.00 GiB net budget. The per-device total is 10.87 to 12.80 GiB, which is still insufficient. A single card is 21.11 to 23.05 GiB.