Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem

Multiverse Computing has introduced a novel approach to pruning large language models (LLMs) by framing the block removal process as an Ising optimization problem. This method was applied to the Llama-3.3-70B-Instruct model, which consists of 80 blocks. The researchers successfully removed 32 blocks, achieving a compression rate of 50%. This pruning technique demonstrated a notable improvement in the MMLU benchmark, yielding a score of 76.6 for the pruned model compared to the original model's score of 82.2.

The study highlights that the energy of the spin system used in the Ising model serves as an effective and inexpensive proxy for predicting the performance of the pruned model on various benchmarks. This insight suggests that the optimization process can be guided by the energy states of the model, allowing for more strategic block removals. The researchers also noted that the optimal configuration is not necessarily the ground state but rather an excited state, which can lead to better performance outcomes.

In terms of comparative performance, the pruned Llama-3.3-70B-Instruct model outperformed other models, such as Qwen3-14B, by approximately 10 points on the MMLU benchmark when 12 out of 40 blocks were removed. The study also explored configurations for removing different numbers of blocks, revealing a configuration space of 29 billion possibilities for the removal of 8 blocks. The most challenging cases for block removal took about two days to compute, indicating the complexity of the optimization problem.

Overall, the findings from this research suggest that applying principles from physics to the pruning of LLMs can yield significant benefits in terms of model efficiency and performance, paving the way for more effective strategies in model optimization.