LLM Quantization Explained: GGUF Levels, Quality Loss and File Sizes

LLM quantization explained: what GGUF levels like Q4_K_M and Q8_0 mean, how much quality each one loses, and how big the files really are for 8B and 70B models.

By Galactic News Editorial · Published 2026-11-10T16:22:44+00:00 · Updated 2026-11-10T16:22:44+00:00

LLM Quantization Explained: GGUF Levels, Quality Loss and File Sizes

Quantization is how an open model shrinks from 16 gigabytes to under 5 and still runs well on a laptop. It stores each weight with fewer bits. The file name tells you how much was cut: Q4_K_M, Q6_K, Q8_0, and the rest. This guide explains GGUF quantization levels, what each one costs in quality, and how big the files really are. All size and quality figures come from llama.cpp's own published measurements.

Quantization explained in one minute

A model is a pile of numbers called weights. At full precision, each weight is a 16-bit float. That is 2 bytes per weight. An 8-billion-parameter model needs about 16 gigabytes just for weights. A 70-billion-parameter model needs about 140 gigabytes. Few people have that much memory. Quantization stores each weight with fewer bits. At 4 bits, the same 8-billion model needs under 5 gigabytes. The trade is precision. A 4-bit weight is a rough approximation of the original 16-bit value. The surprise is how little that costs. Down to about 4 bits, the quality loss is small. Below 4 bits, it bends sharply. That knee in the curve, at 4 bits, is the whole story of quantization.

What GGUF is, and what it is not

GGUF is a file format, not a quantization method. It is the single-file format of llama.cpp, the C and C++ inference project. Ollama, LM Studio, llamafile, and KoboldCpp all run on it. A GGUF file can hold full 16-bit weights, 8-bit weights, or anything down to roughly 2 bits per weight. The quantization type is a separate choice. It is usually written at the end of the filename. So GGUF and quantization are two different things. You always need a GGUF file. The quantization level inside it sets the balance between size, speed, and quality.

How to read a GGUF quant name

Take Q4_K_M apart piece by piece. The Q means quantized. The 4 means about 4 bits per weight on average. The K means k-quant, a later generation of formats that groups weights into importance-aware blocks. K-quants give measurably better perplexity than the older fixed-block formats at the same bit width. The M means medium. S, M, and L stand for small, medium, and large. They describe how many sensitive tensors get extra bits. In Q4_K_M, half of the attention value tensors and half of the feed-forward down-projection tensors are stored at 6 bits. Everything else is stored at 4 bits. Older names work differently. Q8_0, Q4_0, and Q5_1 are the original formats. Weights are split into small blocks that share a scale. Q8_0 is still popular because at 8 bits the simple approach is already close to lossless. I-quants, such as IQ3_XS and IQ2_M, use an importance matrix. They shine below 4 bits, where an IQ3_M can beat a Q3_K_M of the same size.

GGUF quant levels compared

llama.cpp publishes measurements for Llama 3.1 8B. The table below shows bits per weight and file size for each level. Quality cost is the perplexity increase against full 16-bit precision, from the same project's measurements. Lower is better.

Read the table as a curve, not a ladder. The jump from F16 to Q8_0 costs almost nothing. The jump to Q6_K costs a little more. Q4_K_M is where most of the saving happens for a small cost. Below that, the cost rises fast. Q2_K costs about 20 times the perplexity increase of Q4_K_M for a saving of only 1.6 gigabytes on an 8B model.

Quality loss: what the numbers say

Perplexity is the main yardstick llama.cpp publishes. It measures how surprised the model is by real text. A small increase means the quantized model still predicts text almost as well as the original. On Llama 3 8B, the increases are +0.003 for Q8_0, +0.022 for Q6_K, and +0.175 for Q4_K_M. Q2_K jumps to +3.52. Perplexity is not a task benchmark, so treat it as directional. It tells you the shape of the trade, not how the model will do on your prompts. Benchmark suites tell a similar story. On Llama 3.1 8B Instruct, Q8_0 scored 69.41 against 69.47 at 16-bit on a published benchmark run. That is effectively identical. Q4_K_M costs a percent or two of relative perplexity while cutting size to roughly a third. The practical rule is simple. Use the highest quant that fits your memory. Stop worrying at Q5_K_M or above.

File sizes: the formula and real examples

File size follows a simple formula. File size in gigabytes is roughly parameters in billions, times bits per weight, divided by 8. An 8B model at Q4_K_M is about 8 times 4.85 divided by 8, or about 4.9 gigabytes. The same model at Q8_0 is about 8.5 gigabytes. At full F16 it is about 16 gigabytes. For a 70B model, the numbers scale up: about 43 gigabytes at Q4_K_M, about 75 gigabytes at Q8_0, and about 140 gigabytes at F16. Actual files run a few percent heavier than the raw math, because GGUF files also carry metadata, the tokenizer, and k-quant scaling data. Memory needs are larger than the file. You also need room for the KV cache, which holds your context, plus about 0.5 to 1 gigabyte of overhead. A rough rule: VRAM needed equals file size plus KV cache plus overhead. The KV cache is quantized separately from the weights. llama.cpp stores it at 16-bit by default. The -ctk and -ctv flags switch keys and values to q8_0 or q4_0. That is often the cheapest way to fit a longer context.

How to pick the right quant

1. Measure your memory first. Check how much VRAM or RAM you have free. That is the hard limit.
2. Pick the largest model that fits at Q4_K_M or better. Down to about 4 bits, a bigger model usually beats a smaller model at higher precision.
3. Spend spare memory on quality. If Q4_K_M fits with room left over, step up to Q5_K_M, Q6_K, or Q8_0. Each step buys a smaller gain than the last.
4. Drop below 4 bits only to fit a much larger model. Use I-quants such as IQ3_XS or IQ2_M there. They beat K-quants at the same size.
5. Test with your own prompts. Perplexity is a guide. Run your real tasks side by side at two levels and compare the answers.
6. Quantize the KV cache before dropping the weights further. A q8_0 KV cache often frees more memory than a lower weight quant, with less quality cost.

For most people, the answer is Q4_K_M. It is the default in most local tools, and it is the quant llama.cpp uses in its own examples. Q5_K_M or Q6_K are worth it when you have spare memory. Q8_0 is the choice when quality matters most and memory is not tight.

How to quantize a model yourself

You can make a GGUF quant with llama.cpp's own tool. Start from 16-bit or 32-bit weights, not from another quant. The project warns that requantizing from an already quantized file can severely reduce quality. The basic command is:

llama-quantize my-model-f16.gguf my-model-Q4_K_M.gguf Q4_K_M

Replace Q4_K_M with any level from the table. The tool writes a new GGUF file at that level. Then load it with llama-cli:

llama-cli -m my-model-Q4_K_M.gguf

Most people never run this command. They download a pre-quantized GGUF file from Hugging Face instead. Open the model page, open the Files tab, and pick the quant from the list. The file name ends with the quant type, such as Q4_K_M or Q8_0. Download only the file you need. A full GGUF repository can hold twenty files.

Bottom line

Quantization is lossy compression for model weights. Q8_0 is practically free in quality at about half the size of full precision. Q4_K_M is the default for most people: about a third of the size for a small, measurable quality cost. Below 4 bits, quality falls fast. The rule is to run the highest quant that fits your memory, with room left for context. If you have memory to spare, spend it on a higher quant or a larger model.