ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

Problem

This work addresses a gap in the evaluation of coding agents, specifically focusing on their performance in inferring features from incomplete applications. The authors highlight the need for a systematic approach to assess coding agents based on real-world software behaviors, which has not been adequately explored in existing literature. The paper is a preprint and has not undergone peer review.

Method

The authors propose a new benchmark called ProgramDistill, which involves a comprehensive feature discovery process that factors applications into features of varying granularities. The methodology includes a Behavior Discovery Pipeline termed mine-craft-patch, which successfully identifies and verifies behaviors from the analyzed applications. In total, 26 applications were analyzed, leading to the discovery of 1,975 replay-verified behaviors. From these behaviors, the authors constructed 4,063 tasks to evaluate the performance of nine frontier coding agents, including notable models such as GPT-6 Astra and Claude Opus 5.

Results

The evaluation results indicate a Cumulative Workflow Success Rate of 49.2% for GPT-6 Astra and 28.8% for Claude Opus 5, with no baseline performance reported for comparison. Additionally, the Partial-Application Reconstruction Success Rate ranged from 100% to 64.0% as the restoration depth increased from 1 to 8 for GPT-6 Astra, while Claude Opus 5 achieved a success rate of 96% to 32% under the same conditions. The available text does not report quantitative results for other baselines.

Limitations

The authors do not report any limitations in their study, and no obvious limitations are identified in the provided text.

Why it matters

The introduction of ProgramDistill has significant implications for the evaluation of coding agents, as it provides a structured framework for assessing their capabilities in real-world scenarios. This benchmark can facilitate further research into improving coding agents' performance and understanding their limitations in software development tasks, ultimately contributing to advancements in automated software engineering.