Physics-based pruning improves deep LLM block removal
A paper presents a method for pruning transformer models by treating block selection as a constrained binary optimization problem. Each block is assigned a binary variable indicating whether it should be kept or removed. A second-order Taylor expansion of the loss produces a Hessian whose diagonal represents individual block importance, while its off-diagonal terms capture interactions between removal decisions.
The resulting energy function can be used to rank candidate pruning configurations without running the model or evaluating benchmarks for each one. The Hessian is calculated once from forward and backward passes on a small calibration dataset and can then be reused for different compression targets. Candidate configurations can be searched by brute force for smaller cases or with classical, quantum and quantum-inspired solvers, including tabu search.
Tests on Llama-3.1-8B-Instruct, Qwen3-14B and Llama-3.3-70B-Instruct found results comparable to or better than existing block-removal methods, with larger gains at stronger compression. Without retraining, removing 40 of 80 blocks from Llama-3.3-70B-Instruct produced an MMLU score of 76.9, compared with 54.0 for block influence. The method also improved selected results on the hybrid NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 model. The authors report that high-performing configurations can come from low-energy excited states rather than the single lowest-energy solution.
Written by Sakhanda Wire AI from reporting by Hugging Face. Facts, quotes and full context are in the original article below.