When Quantization Preserves Accuracy but Not Evidence: Explanation-Aware Post-Training Quantization for Medical LLMs

Problem

This work addresses a gap in the preservation of answer-supporting rationales during post-training quantization (PTQ) for medical large language models (LLMs). The authors highlight that while existing quantization methods can maintain task accuracy, they often fail to retain the evidence that supports the answers provided by these models. This is particularly critical in medical applications where the rationale behind a decision can be as important as the decision itself. The paper is a preprint and has not undergone peer review.

Method

The authors propose an explanation-aware objective for transformation-based PTQ. The method involves creating an offline faithfulness cache derived from the rationales of a full-precision teacher model. The optimization process focuses on preserving tokens that support the answers and ensuring that the behavior of the evidence-conditioned answers remains intact. The proposed method is instantiated on OSTQuant, utilizing W4A4KV4 quantization. The evaluation includes four medical and instruction-tuned LLMs, each with parameter sizes ranging from 7 billion to 8 billion. The datasets used for evaluation are MedExQA, MedExpQA, and ChallengeClinicalQA.

Results

The results indicate that while the OSTQuant baseline maintains task accuracy, it compromises the integrity of answer-supporting rationales when compared to a same-calibration baseline. In contrast, the proposed method demonstrates superior performance in preserving the answer behavior of the full-precision model and the relationship between rationales and answers when compared to the full-precision model itself. However, the available text does not report quantitative results.

Limitations

The authors do not report any limitations in their work. However, it is worth noting that the absence of reported limitations may suggest a lack of comprehensive evaluation across diverse scenarios or datasets.

Why it matters

This research has significant implications for the deployment of medical LLMs, where the interpretability and explainability of model outputs are crucial. By ensuring that quantization methods can preserve the supporting rationales, this work paves the way for more reliable and trustworthy AI systems in healthcare, potentially enhancing clinician decision-making and patient outcomes.