Quantization
Faster inference.
Same ideas.↘
What does quantization change about how a model’s
numbers are stored, and what do you trade for it?
Quantization stores each weight using fewer bits, for example FP16 to INT8 or INT4. This reduces the memory required to store the model and can also make inference faster, at the cost of a small drop in numerical precision, which can slightly hurt quality.
the architecture?”
→ No. It changes the numerical
representation. The layers stay.
A 70B model goes from FP16 → INT4. How much memory do you save?
70B × 2 bytes = ~140 GB
At INT4 (0.5 bytes each):
70B × 0.5 bytes = ~35 GB
now fits on a single 48 GB GPU.
~48 GB
~48 GB
~48 GB
~35 GB
48 GB GPU ✓
Why does using fewer bits hurt quality, and how do GPTQ and AWQ
reduce the damage?
Using fewer bits introduces quantization error because we round continuous values to a smaller set of discrete values. This can hurt quality, especially for important weights.
GPTQ reduces the damage by quantizing weights and then compensating for the error using second-order information. AWQ reduces the damage by identifying important weights and protecting them with activation-aware scaling before quantizing the rest.
remaining weights
weights
them
carefully
This is a small sample.
600+ interview questions live inside Skillumen.
Every concept card carries its own interview questions, graded as you answer.
Free to start.