When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignment

Problem

This work addresses a gap in the literature regarding the guidance of individual tokens in reinforcement learning with verifiable rewards (RLVR). The authors highlight the lack of effective methods for credit assignment at the token level, which is crucial for improving the performance of reinforcement learning agents. The paper is a preprint and has not undergone peer review.

Method

The authors propose a novel framework called Unified Entropy-Calibrated Credit Redistribution for GRPO (UECR-GRPO). This method integrates verifier and teacher signals within a single update mechanism that follows a GRPO-style approach, operating at both the response and token levels. Key components of the method include:

• Path-Utility Unification (PUU): This technique combines the verifier reward with the teacher-to-anchor path log-ratio, formulated within a KL-regularized objective to enhance the learning signal.
• On-policy implementation: The method employs a length-normalized teacher score, which is combined with rewards prior to group normalization and PPO clipping, ensuring that the updates are stable and effective.
• Entropy-Calibrated Redistribution (ECR): This component utilizes the signed gap between the teacher's old policy and the current token to redistribute the verifier-derived components, allowing for more nuanced credit assignment.
• Full-vocabulary teacher entropy: This mechanism is designed to mitigate the effects of uncertain guidance from the teacher, thereby improving the reliability of the learning process.
• Response-wise zero-sum projection: This ensures that the total task credit is preserved while maintaining the token-wise sign before applying clipping, which is critical for maintaining the integrity of the learning signal.

Results

The proposed method was evaluated on the Qwen3 model, yielding the following results:

• Avg@12 Accuracy (Qwen3-1.7B): 17.21% — outperforming the strongest baseline by 0.89 percentage points.
• Avg@12 Accuracy (Qwen3-4B): 65.09% — also surpassing the strongest baseline by 0.56 percentage points. These results indicate a significant improvement in accuracy, demonstrating the effectiveness of the proposed UECR-GRPO framework in enhancing token-level guidance in RLVR settings.

Limitations

The authors do not report any limitations in their work, and no obvious limitations are identified in the available text.

Why it matters

The implications of this research are substantial for downstream work in reinforcement learning, particularly in scenarios where precise credit assignment is critical. By providing a unified framework that effectively integrates multiple signals for guidance, this work paves the way for more robust and efficient learning algorithms that can better leverage the strengths of both teacher and verifier signals. This could lead to advancements in various applications of RL, including those requiring fine-grained control and decision-making.