Quantization interview questions

Quantization stores each weight using fewer bits — for example going from 16-bit (fp16) down to 8-bit or 4-bit integers. It doesn't remove any weights or change the architecture; it just rounds each number to a coarser scale, keeping a scale factor to stretch it back at run time. The trade is a much smaller, faster model in exchange for a small rounding error that can chip at quality — usually negligible at 8-bit, small at 4-bit with good methods, and larger below that.

Roughly 4×. Each weight goes from 16 bits to 4 bits, so a 70-billion-parameter model drops from about 140 GB in fp16 to about 35 GB in 4-bit. That matters because 140 GB needs two or three data-centre GPUs, while 35 GB fits on a single 48 GB card. It also speeds up inference: these models are memory-bound, so moving fewer bytes per weight means each token comes out quicker. Same model, a fraction of the cost.

Fewer bits means fewer buckets, so every weight gets rounded slightly off its true value — rounding error. Individually tiny, but it can accumulate, and a few unusually large outlier weights carry a lot of the model's behaviour. GPTQ rounds one column at a time and compensates the remaining weights to offset the error it just introduced. AWQ identifies the most important weights and scales them so they survive quantization with more precision. Both mean the damage lands where it matters least.