How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure

Problem

This work addresses the gap in understanding the reproducibility of evaluation conclusions for large language models (LLMs) when based on small prompt sets. The authors conduct a self-audit to assess how reliable the evaluation metrics are across different model variants, particularly in the context of LLMs, where evaluation confidence can be precarious due to the limited scope of prompts used.

Method

The study evaluates eight open model variants from five families, with parameter sizes ranging from 8 billion to 675 billion. Caching was disabled to ensure that the evaluations were not influenced by previously computed results. The authors collected 293 raw intermediate representations during their experiments. The primary evaluation metric employed was the Mean Node-Set Jaccard, which ranged from 0.39 to 0.96. To analyze reproducibility, a joint cluster bootstrap method was applied over the prompts, allowing for a robust assessment of the stability of model rankings across different evaluations.

Results

The results indicate a wide range of Mean Node-Set Jaccard scores, from 0.39 to 0.96, reflecting varying degrees of overlap in the inferred prompt structures. The reproducibility of rankings showed that the bottom two models maintained their rank in 99% and 86% of the replicates, respectively. In contrast, the middle four models exhibited rank consistency in only 27% to 48% of the cases, while the top two models retained their ranks in 68% of the replicates. Notably, the study found that four out of the eight models experienced changes in their rankings, with two specific rules for merging repeated campaigns resulting in a shift of the overall headline by 7 percentage points.

Limitations

The authors acknowledge several limitations in their study. Firstly, identical calls to the models do not consistently yield identical structures, which raises concerns about the reliability of the evaluations. Additionally, four of the eight endpoints evaluated were withdrawn within ten weeks of measurement, potentially affecting the generalizability of the findings. Lastly, the authors caution that small-sample evaluations may present a misleading sense of definitiveness that is not supported by the underlying evidence.

Why it matters

This research has significant implications for the field of LLM evaluation, particularly in highlighting the need for more robust methodologies when drawing conclusions from limited prompt sets. The findings suggest that reliance on small prompt evaluations can lead to misleading results, which may affect downstream applications and research. By emphasizing the variability in model rankings and the reproducibility of evaluations, this work encourages the development of more comprehensive evaluation frameworks that can better capture the performance and reliability of LLMs.