Qwen2.5-14B-Instruct Q4_K_M keeps 8.37 GiB of weights (8,988,110,496 bytes, Qwen/Qwen2.5-14B-Instruct-GGUF) whether you run 1 sequence or 8. At 8,192 tokens on a GeForce RTX 4090, one sequence is 10.00 to 10.64 GiB and eight sequences are 20.50 to 21.88 GiB. The net budget is 22.00 GiB. One, two and four sequences are inside that budget. Eight is tight. Formula estimates, calculator 0.3.0, llama.cpp, FP16 KV, typical extras. Software support is unverified. Tokens per second are Unknown.
KV bytes are 2 × layers × kvHeads × headDim × tokens × sequences × 2 for this GQA model. Sequences scale the cache. They do not scale the weight file inside one replica.
RTX 4090, 8,192 tokens
| Sequences | KV GiB | Total GiB | Verdict |
|---|---|---|---|
| 1 | 1.50 | 10.00 to 10.64 | comfortable |
| 2 | 3.00 | 11.50 to 12.25 | comfortable |
| 4 | 6.00 | 14.50 to 15.46 | fit |
| 8 | 12.00 | 20.50 to 21.88 | tight |
Comfortable means the high end is at or under 70% of the 22.00 GiB net budget. The four-sequence high end, 15.46 GiB, is over that 70% line and under 90%, so the status is likely-fits. The eight-sequence high end, 21.88 GiB, is under 22.00 GiB and above 90%, so the verdict is tight.
A longer context does the same multiplication on one sequence. Qwen2.5-7B at 2K, 8K and 32K is on context length VRAM tradeoff. Two cards that each hold a shard are on two RTX 4090s and tensor parallel.
FAQ
Do four sequences load the model four times?
Not inside one replica. Qwen2.5-14B Q4_K_M weights stay 8.37 GiB at 1, 2, 4 and 8 sequences. KV goes from 1.50 GiB to 12.00 GiB.
Is eight sequences the same as eight GPUs?
No. These rows are one RTX 4090 and one replica. Independent replicas each need a full weight copy. Tensor parallel is a different split. See the two-RTX-4090 page.
Were these sequence counts measured?
No. They are formula estimates. Process GPU memory and tokens per second at these batch sizes are Unknown.