Deployment of Large Language Models (LLMs) on memory-constrained edge devices relies heavily on aggressive post-training quantization. However, the evaluation of these quantized models is still largely based on zero-shot task accuracy, which depends solely on argmax predictions and is therefore insensitive to changes in the underlying predictive distribution. As a result, accuracy can exhibit unstable and non-monotonic behavior under progressive quantization, masking substantial fidelity loss relative to the BFloat16 (BF16) uncompressed base model and potentially providing misleading signals for deployment decisions. In this work, we introduce a distribution-sensitive evaluation framework that quantifies information loss in quantized LLMs as the divergence between full-vocabulary predictive distributions at the token decision boundary. Specifically, we compute statistical distances, including Jensen-Shannon Divergence and Total Variation Distance, between the outputs of full-precision and quantized models, enabling a fine-grained analysis of distributional shift. Using this framework, we quantify probability mass displacement and distributional drift relative to the BF16 reference. These measures capture changes in the predictive distribution that may not be reflected in top-1 task accuracy. We conduct a 120-run experimental matrix across five foundation model architectures and four reasoning benchmarks under progressive quantization regimes, from uncompressed BF16 to Q2\\_K, providing a systematic fidelity analysis. Our results show that divergence metrics generally increase under stronger quantization in the evaluated settings, complementing task accuracy with a fidelity signal relative to the BF16 reference. Across the tested llama.cpp quantization schemes, mixed-precision Q4\\_K generally yields lower divergence than uniform Q4\\_0 at similar memory footprints. These findings motivate distribution-aware evaluation as a practical diagnostic complement to task accuracy; they do not directly establish correctness, calibration, safety, or user-perceived quality.<\/p>\n","protected":false},"excerpt":{"rendered":"
Deployment of Large Language Models (LLMs) on memory-constrained edge devices relies heavily on aggressive post-training quantization. However, the evaluation of these quantized models is still largely based on zero-shot task accuracy, which depends solely on argmax predictions and is therefore insensitive to changes in the underlying predictive distribution. As a result, accuracy can exhibit unstable and non-monotonic behavior under progressive quantization, masking substantial fidelity loss relative to the BFloat16 (BF16) […]<\/p>\n","protected":false},"author":14,"featured_media":0,"template":"","meta":{"_acf_changed":false,"footnotes":""},"publication-type":[32],"publication-year":[1680],"class_list":["post-43800","publication","type-publication","status-publish","hentry","publication-type-inproceedings","publication-year-1680"],"acf":[],"publishpress_future_workflow_manual_trigger":{"enabledWorkflows":[]},"_links":{"self":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/43800","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication"}],"about":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/types\/publication"}],"author":[{"embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/users\/14"}],"version-history":[{"count":0,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication\/43800\/revisions"}],"wp:attachment":[{"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/media?parent=43800"}],"wp:term":[{"taxonomy":"publication-type","embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication-type?post=43800"},{"taxonomy":"publication-year","embeddable":true,"href":"https:\/\/lamarr-institute.org\/de\/wp-json\/wp\/v2\/publication-year?post=43800"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}