Agent-Editing World Model: Rethinking World Modeling for LLM Agents

Problem

Task-state contamination in agents adversely affects their decision-making capabilities. This paper addresses this issue by proposing a novel framework, the Agent-Editing World Model (AEWM), which aims to enhance the reasoning and action processes of agents. The work is presented as a preprint and has not undergone peer review.

Method

The core contribution of this paper is the AEWM, which integrates several components to improve agent performance:

• Action Judge: This component classifies decisions into three categories: Critical, Exploratory, and Noisy. This classification helps in understanding the nature of decisions made by the agent.
• State Revision: This mechanism allows the model to edit noisy reasoning-action continuations based on historical observations, thereby refining the decision-making process.
• EditAct: This feature combines the agent's capabilities with real execution, enabling the model to modify the underlying state effectively.
• Training Regimen: The model was trained across various domains, including Search, Terminal, and Software Engineering, utilizing mid-training and supervised fine-tuning techniques to enhance its performance.

Results

The AEWM demonstrates significant improvements over existing baselines:

• The Action Judge achieved a macro-F1 score of 70.5%, marking a 10.6 points improvement over the strongest frontier baseline.
• The EditAct component showed an average score improvement ranging from 3.2 to 6.7 points across six benchmarks and three different agent backbones compared to the strongest baseline.
• The AEWM-RFT variant exhibited an improvement of 2.2 to 2.6 points over the Self-RFT method across three domains without the guidance of online AEWM.

Limitations

The authors do not report any limitations in the study. However, the absence of reported limitations may suggest a lack of comprehensive evaluation across diverse scenarios or potential overfitting to the training data.

Why it matters

The implications of this work are significant for the development of more robust LLM agents. By addressing task-state contamination, the AEWM framework can lead to improved decision-making processes in various applications, enhancing the reliability and effectiveness of AI systems in real-world tasks. This research opens avenues for further exploration into agent architectures and their training methodologies, potentially influencing future advancements in AI agent design.