At Q4_K_M with an 8K context, Mistral Small 24B needs 14.72 to 15.45 GiB and fits a 24 GB RTX 4090 comfortably. Mixtral 8x7B needs 27.62 to 28.59 GiB and doesn’t fit, even though it only uses about 12.9B parameters per token. The weight file is what decides it: Mixtral’s Q4_K_M GGUF is 28,448,468,384 bytes, all of which has to be loaded.
Four Mistral models at Q4_K_M
| Model | GGUF repo | File size (bytes) | Weights (GiB) | KV cache (GiB) | Total at 8K context (GiB) | RTX 4090 (22 GiB usable) |
|---|---|---|---|---|---|---|
| Mistral-7B-Instruct-v0.3 | bartowski/Mistral-7B-Instruct-v0.3-GGUF | 4,372,812,000 | 4.07 | 1.00 | 5.20 to 5.72 | comfortable |
| Mistral-Nemo-Instruct-2407 | unsloth/Mistral-Nemo-Instruct-2407-GGUF | 7,477,204,512 | 6.96 | 1.25 | 8.34 to 8.94 | comfortable |
| Mistral-Small-24B-Instruct-2501 | unsloth/Mistral-Small-24B-Instruct-2501-GGUF | 14,333,909,248 | 13.35 | 1.25 | 14.72 to 15.45 | fits |
| Mixtral-8x7B-Instruct-v0.1 | mradermacher/Mixtral-8x7B-Instruct-v0.1-GGUF | 28,448,468,384 | 26.49 | 1.00 | 27.62 to 28.59 | won’t fit |
8K context (8,192 tokens), one chat, FP16 KV cache, llama.cpp.
Which card for which model
- 8 GB (RTX 4060, 7 GiB usable): Mistral 7B fits. Mistral Nemo, at 8.34 GiB and up, doesn’t. See Mistral 7B on an RTX 4060.
- 12 GB (RTX 4070, 11 GiB usable): Mistral Nemo fits. See Mistral Nemo on an RTX 4070.
- 16 GB: Mistral Small 24B is tight on an RTX 4060 Ti 16GB (15 GiB usable) and doesn’t fit cards that reserve 2 GiB, such as the RTX 4080 SUPER.
- 24 GB (RTX 4090 or 3090): Mistral Small 24B fits with room for more context. See Mistral Small 24B on an RTX 4090.
- Mixtral 8x7B needs more than one 24 GB card. A 32 GB RTX 5090 (30 GiB usable) is tight, and a 64 GB unified-memory Mac has plenty of room on memory alone. See Mixtral on an RTX 4090 and Mixtral on an RTX 5090.
Why Mixtral is so much bigger than its active size
Mixtral has 46,702,792,704 parameters in total, split across 8 experts per layer, and routes each token to 2 of them. The 12.9B active figure tells you how much arithmetic each token costs, not how much memory the model takes. The router picks again at every token and every layer, so all experts stay resident.
Some runtimes can keep idle experts in system RAM, at a speed cost. The catalog marks expert offload as unspecified for Mixtral, and how many bytes you could move off the GPU that way is unknown, so the calculator does not assume it. More on this in why MoE models load every expert.
The KV cache isn’t the problem: Mixtral’s is 1.00 GiB at 8K, the same as Mistral 7B. A longer context only grows that column; see context length vs VRAM. Two GPUs don’t simply add up their memory either: two RTX 4090s and tensor parallel covers what splitting a model actually does.
Numbers are estimates from model configs and published file sizes, not measurements. How we calculate.
Model pages: Mistral-7B-Instruct-v0.3, Mistral-Nemo-Instruct-2407, Mistral-Small-24B-Instruct-2501, Mixtral-8x7B-Instruct-v0.1.
FAQ
How much VRAM does Mixtral 8x7B need at Q4_K_M?
27.62 to 28.59 GiB at 8K context. That won't fit a 24 GB RTX 4090 (22 GiB usable) and is tight on a 32 GB RTX 5090.
Mixtral only uses 2 of its 8 experts per token. Why does it need memory for all of them?
The router can pick any expert for the next token, so every expert has to stay loaded. The Q4_K_M file is the full 46.7B-parameter checkpoint even though about 12.9B parameters are active per token.
Can I run Mistral Small 24B on a 24 GB GPU?
Yes. At Q4_K_M with an 8K context it needs 14.72 to 15.45 GiB, which fits an RTX 4090 or RTX 3090 with room left for a longer context.
What is the smallest GPU for Mistral 7B?
An 8 GB card. Mistral-7B-Instruct-v0.3 at Q4_K_M needs 5.20 to 5.72 GiB at 8K context, which fits the 7 GiB usable on an RTX 4060.
Numbers from calculator 0.3.0, model data 2026-10-02.1. Sources last checked 2026-10-29.