What changes#
Lower precision reduces memory use and can improve throughput. Whether quality changes depends on the model, quantization method and task.
What to measure#
Compare the same checkpoint, prompts and generation settings. Record exact quantization and runtime versions. Test retrieval fidelity, structured extraction and tool arguments rather than a few pleasant chat responses.
Our reporting standard#
Check the actual serving precision for the selected deployment. An unspecified value does not establish BF16, FP8 or any other quantization; confirm it before comparing memory use or output quality.