Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It

Problem

The paper addresses a significant gap in the capability of serving systems, specifically the inaccurate estimation of tool call durations. This inaccuracy leads to inefficient management of key-value (KV) caches, which can degrade overall system performance. The authors highlight that existing predictors do not adequately capture the dynamics of tool calls, necessitating a more reliable approach. Notably, this work is presented as a preprint and has not yet undergone peer review.

Method

The authors propose a novel approach called Tool Call Reporting, where tool calls explicitly report their progress during execution. This method leverages a census of four public agent corpora to gather data on tool call performance. The core technical contribution lies in the integration of this reporting mechanism into a production engine with minimal changes, allowing for real-time updates on tool call progress. The performance of the proposed method is measured by evaluating the accuracy of the reported progress against existing predictors, which are currently the best published in the literature.

Results

The results demonstrate that the accuracy of progress reporting using the proposed method is several times to an order of magnitude more accurate than the best published predictors. Additionally, the authors report a 20.7% reduction in Time to First Token (TTFT) when using High Bandwidth Memory (HBM) only, compared to the Least Recently Used (LRU) caching strategy. When both HBM and DRAM are utilized, the TTFT improvement is 20.8% over LRU. These results indicate a significant enhancement in the efficiency of KV cache management through improved tool call duration estimates.

Limitations

The authors do not report any limitations in their work. However, it is important to note that the lack of reported limitations may suggest a need for further validation in diverse operational environments or with different types of tool calls.

Why it matters

This work has important implications for the design and optimization of serving systems, particularly in contexts where tool calls are frequent and performance is critical. By providing a more accurate method for estimating tool call durations, the proposed approach can lead to more efficient resource allocation and improved system responsiveness. This advancement could pave the way for further research into adaptive caching strategies and more intelligent serving architectures.