A Zeroth-Order Paradigm for LLM Preference Alignment

Problem

The paper addresses a gap in the capability of existing methods to extract information from preference pairs with small likelihood margins. This is particularly relevant in the context of aligning large language models (LLMs) with user preferences, where traditional methods may struggle to effectively utilize subtle differences in preference data. The authors propose a novel approach, termed Comparison-based Preference Optimization (ComPO), as a solution to this challenge. Notably, this work is presented as a preprint and has not undergone peer review.

Method

The core technical contribution is the introduction of ComPO, a zeroth-order alignment method that extracts directional information from preference pairs without relying on a differentiable preference loss. This method operates under specific conditions, including smoothness, gradient sparsity, and compatibility between the oracle and a latent objective, which provide a convergence guarantee.

Additionally, the authors present an online variant of ComPO that retains the offline comparison mechanism while utilizing unlabeled policy generations for reverse-KL control relative to a reference policy. The performance guarantee for this method is established for a basic constrained scheme, focusing on local coverage and in-distribution pairwise reward accuracy.

Results

The paper reports improvements of ComPO over existing direct alignment methods, although specific metrics are not disclosed. The authors mention enhancements in length-controlled win rates and provide pair-level diagnostics that indicate a consistent mitigation of likelihood displacement. However, the available text does not report quantitative results, making it difficult to assess the magnitude of these improvements relative to named baselines or benchmarks.

Limitations

The authors do not explicitly state any limitations of their approach. However, potential limitations inherent to zeroth-order methods, such as sensitivity to noise in preference data or challenges in scalability, are not discussed. This omission may warrant further investigation in future work.

Why it matters

The implications of this research are significant for downstream work in preference alignment for LLMs. By providing a method that can effectively utilize preference pairs with small likelihood margins, ComPO may enhance the ability of LLMs to align more closely with user preferences, potentially leading to improved user satisfaction and engagement. Furthermore, the establishment of performance guarantees under specific conditions may pave the way for more robust applications of zeroth-order methods in various machine learning contexts.