RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning

Problem

The paper addresses a gap in the capability of self on-policy distillation (OPD) for multi-turn agents in reinforcement learning (RL). The authors highlight that existing methods are hindered by unreliable teacher information and the stage-dependent benefits of supervision. This work is particularly relevant as it is presented as a preprint and has not undergone peer review.

Method

The core technical contribution is the Self-Retiring On-Policy Distillation (RetireOPD) algorithm. The architecture consists of a decoupled, skill-conditioned teacher and a skill-free student. A notable feature of RetireOPD is the Adaptive Retirement mechanism, which allows the student agent to independently drop the teacher based on performance metrics, enhancing the learning process. The data utilized for training includes Qwen2.5 models, which range from 1.5 billion to 7 billion parameters. However, the loss function and specific training compute details are not disclosed in the paper.

Results

The results demonstrate significant improvements over a reinforcement learning baseline. In the ALFWorld benchmark, the success rate improved from 14.1% to 18.8%. Similarly, in the WebShop benchmark, accuracy increased from 11.8% to 19.0%. Furthermore, the student agent consistently outperformed the skill-conditioned teacher across all settings, indicating the effectiveness of the RetireOPD approach.

Limitations

The authors do not report any limitations in the study. However, the lack of specified loss functions and training compute details may limit reproducibility and understanding of the training dynamics.

Why it matters

The implications of this work are significant for downstream applications in reinforcement learning, particularly in scenarios requiring multi-turn interactions. By improving the reliability of on-policy distillation and enabling agents to adaptively manage their learning sources, RetireOPD could enhance the performance of RL agents in complex environments, paving the way for more robust and efficient learning strategies.