Guides

Practical notes tied to the same calculator core.

• 4-bit, 8-bit, and FP16 memory are not the same kind of number

  Theoretical bits/parameter, real GGUF files, and BF16 safetensors answer different questions.

• How to read 12 GB, 16 GB, and 24 GB cards for local LLMs

  Treat marketing GB as GiB, subtract a reserve, then compare a range — not weights versus the box label.

• How context length changes KV cache memory

  Raw KV is linear in cached tokens and in concurrent sequences. Weights stay put for a shared replica.

• Why a model file can fit and runtime still runs out of memory

  Selected weight files are not peak occupancy. KV cache, workspace, and allocator behavior sit on top.

• How much VRAM does a local LLM need?

  Weight files, KV cache, and runtime extras add up. Walk through Qwen2.5-7B-Instruct at 8,192 tokens.

• Two GPUs are not one bigger GPU, and unified memory is not extra RAM stacked on VRAM

  Independent replicas, tensor parallel, and DGX Spark unified memory are different topologies.