Which local LLMs fit on an RTX 4060 (8 GB GDDR6)?

The RTX 4060 has 8 GB GDDR6 of discrete VRAM. After a 1 GiB reserve for the OS and display, about 7 GiB is usable for a model. At an 8K context the largest model in our list that fits is Qwen2-7B-Instruct at Q5_K_M (about 6.1 GiB). GLM-4-9B-0414 at Q4_K_M is tight (about 6.2 GiB–6.7 GiB).

These are estimates from the model's config and file sizes, not measurements. How we calculate

Which models fit at 8K

Each model uses the highest-precision file that fits, one chat at a time.

Fits (28)

ModelQuantTotal
DeepSeek-R1-Distill-Llama-8BGGUF Q4_K_M (Unsloth)5.7 GiB–6.2 GiB
Qwen2.5-Math-7B-InstructGGUF Q4_K_M (bartowski)4.9 GiB–6.2 GiB
Qwen2.5-Coder-7BGGUF Q5_K_M (mradermacher)5.6 GiB–6.1 GiB
Qwen2.5-7B-InstructGGUF Q5_K_M5.6 GiB–6.1 GiB
Qwen2.5-Coder-7B-InstructGGUF Q5_K_M (Qwen)5.6 GiB–6.1 GiB
DeepSeek-R1-Distill-Qwen-7BGGUF Q5_K_M (Unsloth)5.6 GiB–6.1 GiB
OpenMath-Nemotron-7BGGUF Q5_K_M (mradermacher)5.6 GiB–6.1 GiB
Qwen2.5-VL-7B-InstructGGUF Q5_K_M (Unsloth)5.6 GiB–6.1 GiB
Qwen2-7B-InstructGGUF Q5_K_M (Qwen)5.6 GiB–6.1 GiB
Phi-3.5-mini-instructGGUF Q4_K_M (bartowski)5.4 GiB–6.0 GiB
Qwen3-VL-4B-InstructGGUF Q8_0 (Qwen)5.2 GiB–5.8 GiB
Qwen3-4B-Instruct-2507GGUF Q8_0 (Unsloth)5.2 GiB–5.8 GiB
Qwen3-4BGGUF Q8_05.2 GiB–5.8 GiB
Mistral-7B-Instruct-v0.3GGUF Q4_K_M (bartowski)5.2 GiB–5.7 GiB
Phi-4-mini-instructGGUF Q8_0 (Unsloth)4.9 GiB–5.5 GiB
Qwen2.5-Coder-3B-InstructGGUF Q8_0 (Qwen)3.8 GiB–4.2 GiB
Qwen2.5-3B-InstructGGUF Q8_03.8 GiB–4.2 GiB
Qwen2.5-VL-3B-InstructGGUF Q8_0 (Unsloth)3.5 GiB–3.9 GiB
SmolLM2-1.7B-InstructGGUF Q8_0 (Unsloth)3.3 GiB–3.8 GiB
Gemma 3 4B ITGGUF Q4_0 (Google QAT)3.3 GiB–3.8 GiB
Qwen3-1.7B-BaseGGUF Q8_0 (mradermacher)2.7 GiB–3.2 GiB
Qwen3-1.7BGGUF Q8_02.7 GiB–3.2 GiB
Llama 3.2 1B InstructOfficial BF16 safetensors2.7 GiB–3.1 GiB
Qwen2.5-Coder-1.5B-InstructGGUF Q8_0 (Qwen)2.1 GiB–2.5 GiB
Qwen2.5-1.5B-InstructGGUF Q8_0 (Qwen)2.1 GiB–2.5 GiB
Qwen3-0.6BGGUF Q8_0 (Qwen)1.6 GiB–2.0 GiB
Gemma 3 1B ITGGUF Q4_0 (Google QAT)1.1 GiB–1.5 GiB
Qwen2.5-0.5B-InstructGGUF Q8_0 (Qwen)0.8 GiB–1.2 GiB

Tight fit (6)

ModelQuantTotal
Llama 3.2 3B InstructOfficial BF16 safetensors7.0 GiB–7.5 GiB
GLM-4-9B-0414GGUF Q4_K_M (Unsloth)6.2 GiB–6.7 GiB
granite-3.3-8b-instructGGUF Q4_K_M (IBM)6.0 GiB–6.5 GiB
Qwen3-VL-8B-InstructGGUF Q4_K_M (Qwen)5.9 GiB–6.5 GiB
Qwen3-8B-BaseGGUF Q4_K_M (mradermacher)5.9 GiB–6.5 GiB
Qwen3-8BGGUF Q4_K_M5.9 GiB–6.5 GiB
Won't fit (52)

Detailed checks

Memory

The reserve is an adjustable assumption. On a headless machine you can lower it in the calculator.

Two of these cards do not add up to one bigger card. Each copy of a model has to fit on one card unless the engine splits it with tensor parallelism.

Open the hardware-first calculator with this deviceCalculator

Related pages

Sources

Calculator

Loading catalog snapshot…

Context length

Longer context needs more memory for the KV cache, the model's working memory for the conversation.

More options
Runtime

Engine and KV cache type for this estimate.

Off: GGUF, Q8-class, and other filters stay strict. On: a replacement is shown with requested, used, and reason.

Workload

Overhead scenario and the prefill workspace.

Topology

How the model is split across devices. Memory is not added together.

Replica count is an advanced deployment option. Quantity is nodes or GPUs, not replicas.

Budget

System reserve and safety margin. An empty number keeps the previous value.

GiB (2^30 bytes)

%

Pick a GPU or Mac to see which models fit, or pick a model to see how much memory it needs.