Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen

Problem

The paper addresses the inadequacy of compile rate as a metric for evaluating single-function vulnerability repair in large language models (LLMs). The authors argue that compile rate can be misleading, as it does not accurately reflect model performance due to various confounding factors. This work is particularly relevant as it is presented as a preprint and has not undergone peer review, indicating that the findings should be interpreted with caution.

Method

The authors conducted five controlled experiments involving 203 vulnerable functions sourced from the Big-Vul dataset. They evaluated three open-source code LLMs with parameter sizes ranging from 350 million to 6.7 billion. The study employed three distinct prompting strategies to assess the models' performance. The evaluation metrics included compile rate, reference-similarity metrics, whole-function CodeBLEU, and diff_F1, allowing for a comprehensive analysis of the models' capabilities in repairing code vulnerabilities.

Results

The results indicate that 64% of compile failures were not attributable to the model's quality, suggesting that the compile rate is an unreliable indicator of performance. The compile rate exhibited a shift of 1.8 to 2.7 times on identical patches when evaluated under a single compiler-standard flag, with no regressions noted. Interestingly, the ranking of the three models based on compile rate was found to be in the opposite order compared to rankings derived from reference-similarity metrics. Additionally, a whole-function CodeBLEU score showed that an unchanged copy of the vulnerable input outperformed all models tested. The diff_F1 metric was found to give zero credit for no-operations (no-ops) and near-zero credit for certain deletion-based patches, further questioning its utility as a measure of repair quality.

Limitations

The authors acknowledge that the compile rate is influenced by artifacts from the evaluation harness and the dataset used, which may skew results. Furthermore, they highlight that diff_F1 is not a reliable metric for assessing repair quality due to its inherent limitations, particularly in how it evaluates no-ops and deletion-based changes.

Why it matters

This work has significant implications for the evaluation of LLMs in code repair tasks. By highlighting the shortcomings of traditional metrics like compile rate and diff_F1, the authors advocate for a more nuanced understanding of model performance. The proposed change-aware evaluation framework could lead to improved methodologies for assessing LLMs in software engineering contexts, ultimately enhancing the reliability of automated code repair systems.