Sakhanda Wire
NVDA $153.77 -1.52% MSFT $504.91 -2.53% GOOGL $281.52 -0.78% META $620.99 -0.87% AMZN $256.80 -0.37%
← Back to the news

HOPE prunes MoE experts while preserving cooperative groups

A method called HOPE aims to reduce the memory footprint of mixture-of-experts (MoE) language models by removing less useful experts while accounting for relationships between them. MoE models may contain hundreds of experts, but only a small subset is activated for each token. In Qwen3.5-122B, for example, 256 experts are present in each of 48 MoE layers, while eight are used per layer for an individual token. Expert MLPs account for about 95% of the model’s roughly 250 GB parameter memory footprint.

Unlike first-order pruning techniques, which score experts independently, HOPE uses a matrix that captures pairwise, gate-weighted interactions between experts that are activated together. The method selects experts for removal by solving a quadratic program after one calibration pass over a dataset.

Tests on Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air compared HOPE with four first-order methods across pruning rates from 10% to 50%. HOPE achieved the best average rank across the tested conditions. At 50% pruning, its average rank was 1.58, compared with 2.42 for REAP. On SWE-Bench Pro, Qwen3.5-35B-A3B scored 42.3 after HOPE pruning, versus 41.8 for the unpruned model and 36.2 for REAP.

Written by Sakhanda Wire AI from reporting by Hugging Face. Facts, quotes and full context are in the original article below.

Originally published by Hugging Face on

Read the original on Hugging Face ↗

The reporting and the image belong to Hugging Face and are credited and linked above; the text on this page is Sakhanda Wire's own summary.

← Back to the news

More stories

All the latest news