Grow the Harness, Not the Context: From Strategy-Free Scaffolds to Reusable Specialist Agents

Problem

This work addresses the inefficiency of standard harnesses in large language model (LLM) agents, which necessitate repeated control decisions for each task. The authors propose a solution to streamline this process, thereby improving the overall performance of LLM agents. The paper is a preprint and has not undergone peer review.

Method

The authors introduce the Growing Harness, a failure-guided training paradigm designed to optimize the performance of LLM agents. Key components of this method include:

• Scaffold: A strategy-free scaffold that exposes fixed model and tool interfaces, allowing for a more structured approach to agent design.
• Execution Traces: Function-level execution traces that help localize failures to a bounded code surface, facilitating targeted repairs.
• Optimizer: An optimizer that repairs a window of failures jointly, enhancing the robustness of the agent.
• Gate Mechanism: A success-first held-out gate that rolls back harmful repair sequences, ensuring that only beneficial changes are retained.
• Harness Structure: Accepted edits accumulate in a shared harness, promoting reusability and efficiency across tasks.

Results

The proposed method demonstrates significant improvements across various benchmarks:

• Mean Success Rate: Achieves the highest mean success in five out of six benchmark-model settings, with a 0.7 percentage point improvement in the sixth setting compared to the best mean.
• LLM Calls Reduction: Reduces the number of LLM calls by 76.0% to 91.8% when compared to the Tool-Calling agent.
• Inference Cost Reduction: The deployed-agent inference cost is reduced by 74.4% to 98.6%, indicating substantial efficiency gains.
• WebArena-Verified Success Rate: Maintains a success rate of 44.7% to 45.3% across different model scales, in stark contrast to the Tool-Calling agent, which achieves only 6.7% success with the 4B model.

Limitations

The authors do not report any limitations in their work, and no obvious limitations are identified in the available text.

Why it matters

This research has significant implications for the development of more efficient LLM agents, particularly in contexts where repeated control decisions are costly. By optimizing the harness structure and reducing inference costs, the proposed method can enhance the scalability and applicability of LLMs in real-world tasks, paving the way for more sophisticated and resource-efficient AI systems.