Quantization explained

Quantization stores the weights of a model with fewer bits so that it fits in less memory.

Read on 14 November 2026. This field changes fast: check the sources.

What the project says

The llama.cpp quantization README says that quantization shrinks a model's size and can speed up inference, and that it may introduce some accuracy loss, usually measured in perplexity and/or Kullback-Leibler divergence. It says this loss can be minimized by using a suitable imatrix file. Quantization is done in two phases: first convert the original model to the GGUF format, then quantize the GGUF file. The README's example quantizes to Q4_K_M with a command of the form llama-quantize input.gguf output.gguf Q4_K_M.

It also warns that requantizing a file that is already quantized (--allow-requantize) can severely reduce quality compared to quantizing from a 16-bit or 32-bit file.

What it means for you

1. Fewer bits, less memory. Memory for weights shrinks in step with the bits: see the table on the memory page.
2. Quality can drop. How much depends on the model and the method, so test your own task on the file you will use.
3. Names. A name such as Q4_K_M identifies a method in the llama.cpp family; the README lists them. We give no figure for bits per weight of each, because the README text we read gives none.
4. Do not stack. Download files made from the original precision, not quantized again.

Choosing

Start with the largest quantization that fits your memory with room for the context, then try the next one down if it does not. Run a few prompts that matter to you and compare the answers.

This is an independent guide. It is not connected with Ollama, the llama.cpp project or any company named here. This field changes quickly: statements about Ollama and llama.cpp were read on 14 November 2026 from the pages named under Sources. The memory numbers are estimates from arithmetic. We did not run any model for these pages.