Side-by-side comparison of the two most popular llama.cpp quantization formats for Llama 3 8B on a consumer RTX 4090 (24GB). See how file size, VRAM headroom, decode throughput, and quality differ — with transparent formulas, not hand-wavy claims.
| Spec | Q4_K_M | Q5_K_M | Winner |
|---|---|---|---|
| Bits per weight (bpw) | ~4.8 bpw | ~5.7 bpw | Q5_K_M |
| Model file size (8B params) | 4.4 GB | 5.4 GB | Q4_K_M |
| VRAM after weights (24GB GPU) | 19.6 GB free | 18.6 GB free | Q4_K_M |
| Dequantization overhead | ~10% | ~8% | Q5_K_M |
| Quality (chat) | Excellent | Near-lossless | Q5_K_M |
| Quality (code/math) | Minor artifacts | Lossless-grade | Q5_K_M |
| Recommended use case | General chat on consumer GPU | Quality-critical local apps | Tie |
| llama.cpp default | Yes (recommended) | Optional | Q4_K_M |
Q4_K_M is the recommended default for local inference — it's llama.cpp's own recommended sweet spot. For an 8B model, the file is ~4.4 GB vs Q5_K_M's ~5.4 GB, leaving ~1 GB more VRAM for KV cache.
Q5_K_M quality is near-lossless for chat but the difference matters for code, math, and long-context reasoning. If your workflow involves structured output (JSON mode) or multi-step reasoning, Q5_K_M is worth the ~1 GB extra.
Q4_K_M is slightly faster decode because the smaller weights mean less memory bandwidth pressure on memory-bound decode. The difference is 5-10% — measurable but not dramatic for an 8B model.
Context length is where Q4 wins — at 32K context on a 24 GB GPU, Q5_K_M runs out of VRAM about 4-8K tokens earlier than Q4_K_M. For long-context RAG/agent work, Q4_K_M lets you fit more.
Start with Q4_K_M — it's the llama.cpp default and works on virtually all consumer GPUs (8GB+). Upgrade to Q5_K_M only if you notice quality issues with code/math/structured output, and your GPU has the VRAM headroom.
For chat and general Q&A, no — Q4_K_M is indistinguishable. For coding, math, long-context reasoning, or JSON-mode structured output, yes — Q5_K_M's extra precision reduces artifacts. Test on your specific workload.
At Q4_K_M: ~4.4 GB weights + KV cache (varies with context). At 8K context: ~6 GB total. At 32K context: ~10 GB. At 64K context: ~14 GB. A 16 GB GPU (RTX 4060 Ti) comfortably handles 32K; a 24 GB GPU (RTX 4090) handles 64K+.
Yes — both are different .gguf files. llama.cpp loads whichever file you point it at. You can keep both on disk (~10 GB total) and swap based on workload: Q4 for chat, Q5 for code review.
Q3_K_M (~3.9 bpw) is too lossy for production — only use when VRAM is critically short. Q6_K (~6.6 bpw) and Q8_0 (~8.5 bpw) are near-lossless but 60-100% larger than Q4_K_M with diminishing quality returns. Stick to Q4_K_M or Q5_K_M for the sweet spot.
Related comparisons: