Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning

Problem

Quantization-aware distillation (QAD) fails to recover the mathematical and code reasoning performance that is diminished when models are quantized to sub-3-bit precision. This paper addresses this gap by proposing a new method that leverages the actual trajectory of quantized models to improve performance retention. The work is presented as a preprint and has not undergone peer review.

Method

The authors introduce On-Policy Distillation (OPD), which utilizes teacher supervision derived from the quantized model's actual trajectory during inference. In this framework, the student model generates outputs through the quantized forward path while receiving feedback from a frozen full-precision teacher model. The training is conducted using two datasets: MATH-500 and HumanEval, which are standard benchmarks for evaluating mathematical reasoning and code generation capabilities, respectively. The performance is measured in terms of average BF16 performance retention, which quantifies how well the distilled model retains the capabilities of the full-precision teacher.

Results

The results demonstrate significant improvements in performance retention compared to QAD. Specifically, the OPD method achieves a performance retention of 70% on the MATH-500 dataset, compared to only 35% retention with QAD. For the HumanEval dataset, OPD achieves a performance retention of 91%, while QAD only reaches 66%. These results indicate that OPD effectively enhances the reasoning capabilities of low-bit quantized models.

Limitations

The authors do not report any limitations in their work. However, it is important to note that the absence of reported limitations does not imply that there are none; potential issues such as generalization to other datasets or the scalability of the method to different model architectures could be areas for future exploration.

Why it matters

The implications of this work are significant for the deployment of low-bit quantized models in resource-constrained environments, where maintaining performance is critical. By improving the reasoning capabilities of quantized models, this research paves the way for more efficient AI systems that can operate effectively with reduced computational resources. This could lead to advancements in various applications, including mobile AI, edge computing, and real-time processing tasks.