JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

Problem

This work addresses the challenge of evaluation cost and the reliability of confidence in large language models (LLMs) when used as judges. The authors highlight the need for a more efficient evaluation mechanism that can maintain accuracy while reducing operational costs. The paper is a preprint and has not undergone peer review.

Method

The core technical contribution is the introduction of the JEV-as-a-judge architecture, which focuses on decision-only judgments. This model is compared against sixteen generative and reward-model judges, as well as blinded human adjudication, to assess its performance. Notably, JEV operates at only 0.36% of the cost of the strongest comparator, demonstrating significant cost efficiency. The model is specifically designed to handle low-confidence decisions, which is a critical aspect of its functionality.

Results

The JEV-as-a-judge model achieves an accuracy that is within three percentage points of the state-of-the-art LLM judge when compared to the strongest comparator. Furthermore, it retains 99% of the comparator's accuracy while operating at a substantially lower cost, showcasing its effectiveness in maintaining performance while reducing financial overhead.

Limitations

The authors acknowledge that the JEV model may struggle with larger gaps in judgments that require derivation checks or involve complex incorrect answers. This limitation suggests that while JEV is effective for straightforward decision-making, it may not be as reliable in more intricate evaluative scenarios.

Why it matters

The implications of this work are significant for downstream applications in AI evaluation, particularly in contexts where cost efficiency is paramount. By providing a model that can deliver high accuracy at a fraction of the cost, JEV-as-a-judge could facilitate broader adoption of LLMs in various domains, enabling more scalable and accessible AI solutions.