Quantization stores a model’s numbers with fewer bits. That reduces the space needed for the weights and can help a large model fit your computer. It changes how precisely the numbers are represented; it does not simply remove most of the model’s parameters.
A rounding analogy
Imagine replacing a long decimal with a shorter approximation. The shorter number is easier to store, but loses detail. Real quantization is more sophisticated than rounding every value the same way, and some model parts may keep higher precision.
Why the download is not exactly one quarter
In a simplified calculation, four bits per weight use a quarter of the space of sixteen bits per weight. Real files also include scales, metadata, and sometimes higher-precision sections. Running the model needs extra working memory as well.
What is the tradeoff?
Less precision can change answers or reduce quality. The effect depends on the model, the quantization method, and the task. Speed may improve, but depends on the hardware and software. A smaller file is not proof of a faster experience.
Try this
Compare two versions of the same model using a task you can check, such as extracting dates from your own notes. Keep the prompt and settings the same. Look for missing or changed facts as well as the time taken.
4-bit is a useful starting point for fitting larger models. Treat it as a candidate to test, not a universal best-quality setting.
Sources & further reading
These references go into more detail; some are written for developers.