SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving

Problem

The paper identifies a significant gap in existing benchmarks for production inference engineering tasks, particularly in evaluating the effectiveness of various models and configurations in real-world scenarios. The authors argue that current benchmarks do not adequately address the complexities involved in production environments, which necessitates a more robust evaluation framework. This work is presented as a preprint and has not yet undergone peer review.

Method

The authors propose SWE-Serve, a comprehensive benchmark designed to assess 53 repository-grounded tasks that span from production changes to SGLang. The benchmark encompasses six distinct inference engineering families, allowing for a diverse evaluation of model performance. The execution environment for the benchmark can utilize either a CPU or a single GPU (specifically the H100).

Evaluation is conducted through a series of hidden functional and regression tests, alongside end-to-end (E2E) serving tests that incorporate calibrated performance gates. To ensure the reliability of the results, the authors implement several controls, including executable no-op and oracle controls, adversarial verifier reviews, and closed-book execution. The benchmark evaluates 11 models across 31 model-effort configurations, providing a comprehensive landscape for performance assessment.

Results

The benchmark yields a mean Pass@1 score of 75% for the best-performing configuration, indicating a solid level of task completion. However, the results also reveal a concerning E2E test rejection rate, where 45.9% of patches that pass other tests are ultimately rejected in the E2E tests. This rejection rate is notably higher at 69.4% when E2E tests are excluded, underscoring the importance of E2E evaluations in ensuring production readiness.

Limitations

The authors acknowledge a substantial gap between local task completion rates and actual production correctness. This discrepancy highlights the challenges in translating model performance from controlled environments to real-world applications. Additionally, the paper does not address potential scalability issues or the impact of varying deployment conditions on the benchmark's applicability.

Why it matters

The introduction of SWE-Serve has significant implications for downstream work in production inference engineering. By providing a structured and rigorous evaluation framework, it enables researchers and practitioners to better understand the performance of different models in production settings. This benchmark can facilitate the development of more reliable AI systems, ultimately leading to improved deployment strategies and enhanced operational efficiency in real-world applications.