Higher-order pruning of experts in mixture-of-experts language models

Problem

Large parameter counts in Mixture-of-Experts (MoE) language models create significant memory bottlenecks, hindering their deployment and efficiency. This paper addresses this issue by proposing a novel pruning technique that aims to reduce the model size while maintaining performance. The work is presented as a preprint and has not yet undergone peer review.

Method

The authors introduce HOPE (Higher-Order Pruning of Experts), which employs a second-order pruning objective that minimizes an upper bound on pruning error. This method is a generalization of REAP, a first-order pruning technique, and is designed to enhance the efficiency of MoE models. The experiments were conducted on three MoE models with parameter counts reaching up to 122 billion. Two distinct calibration sets were utilized to evaluate the effectiveness of the pruning method across various tasks. The benchmarks included a diverse set of challenges such as mathematical reasoning, instruction following, coding tasks, and an agentic suite, ensuring a comprehensive assessment of the model's capabilities.

Results

HOPE achieved a pruning rate of 50%, demonstrating a significant reduction in model size. In terms of performance, HOPE attained an average rank of 1.58 out of five methods tested, outperforming REAP, which had an average rank of 2.42. Notably, HOPE provided a performance gain of up to +6.1% on the agentic coding tasks. Overall, HOPE achieved the best average rank across all conditions and surpassed other methods in the majority of head-to-head comparisons, indicating its effectiveness in maintaining performance while reducing model complexity.

Limitations

The authors did not report any limitations in their study, suggesting that the method may be robust across the tested scenarios. However, the absence of reported limitations may also indicate a lack of exploration into potential weaknesses or edge cases of the proposed method.

Why it matters

The implications of this work are significant for the deployment of large-scale language models, particularly in resource-constrained environments. By effectively reducing the memory footprint of MoE models without sacrificing performance, HOPE paves the way for more efficient AI systems that can be utilized in a wider range of applications. This advancement could lead to improved accessibility and scalability of AI technologies, fostering further research and development in the field.