Coupled Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback

Problem

The paper addresses the systematic bias and errors that arise in large language model (LLM) distillation due to direct imitation under covariate shift. This issue is particularly pronounced when the teacher model's outputs are directly imitated by the student model, leading to suboptimal performance in target domains. The authors highlight that existing methods often rely on target-domain reward feedback, which may not always be available or feasible, thus motivating the need for a novel approach. This work is presented as a preprint and has not undergone peer review.

Method

The authors propose an algorithm called Coupled Calibration and Learning (CCL). This method couples the calibration of the teacher model with updates to the student model through a mechanism of token-level branching. The training process involves providing reward feedback solely on source questions, which are used to calibrate the teacher model in each iteration. Subsequently, the student model is trained on target questions. The authors provide a convergence proof demonstrating that the expected average Kullback-Leibler divergence between the student and an oracle student converges to zero at a polynomial rate as the number of iterations increases. The oracle student is defined as one that maximizes the true reference-regularized target reward within the student class, ensuring that the student learns effectively from the calibrated teacher.

Results

The paper reports that the expected average Kullback-Leibler divergence to the oracle student converges to zero, indicating effective learning and calibration. However, the available text does not report quantitative results against specific baselines or benchmarks. The authors also quantify the error in teacher calibration and establish a separation from regularized direct matching, although specific error metrics are not detailed.

Limitations

The authors do not report any limitations in their work. However, the lack of quantitative results and comparisons to existing methods could be seen as a limitation in assessing the practical effectiveness of the proposed approach.

Why it matters

The implications of this work are significant for downstream applications in LLM distillation, particularly in scenarios where target-domain feedback is scarce or unavailable. By mitigating teacher bias through the proposed CCL method, the research opens avenues for more robust and effective model training in diverse applications, potentially enhancing the performance of LLMs in real-world tasks.