Sakhanda Wire
NVDA $154.00 -1.37% MSFT $506.28 -2.26% GOOGL $280.87 -1.00% META $620.34 -0.97% AMZN $256.88 -0.34%
← Back to the news

Comparison maps routing choices across 10 open MoE models

A configuration-verified comparison examines how 10 open mixture-of-experts (MoE) models assign tokens to experts. The models range from Qwen3.6-35B-A3B, with 35B total and 3B active parameters, to DeepSeek-V4-Pro, with 1.6T total and 49B active. Their routed expert counts span 8 to 384, while top-k selection ranges from one expert in Llama 4 Maverick to eight in several models.

All models use token-choice routing, but differ in scoring, shared experts and load balancing. Seven retain one shared expert; Mixtral, gpt-oss and Qwen3 do not. DeepSeek-V3 uses sigmoid scoring with bias-based, auxiliary-loss-free balancing, while DeepSeek-V4-Pro uses sqrtsoftplus, six selected experts and hash routing in its first three MoE layers. Kimi K2 uses 384 experts and top-8 routing; its report found that higher sparsity required fewer FLOPs at the same validation loss.

The comparison also notes operational differences. gpt-oss-120b uses top-4 routing across 128 experts and can fit on one 80 GB GPU with MXFP4 weights. DeepSeek-V4-Pro and Kimi K2 require multi-GPU expert parallelism. Gating details for Llama 4 Maverick and Qwen3.6, and balancing details for Llama 4 Maverick and gpt-oss, are not published in their configurations.

Written by Sakhanda Wire AI from reporting by MarkTechPost. Facts, quotes and full context are in the original article below.

Originally published by MarkTechPost on

Read the original on MarkTechPost ↗

The reporting and the image belong to MarkTechPost and are credited and linked above; the text on this page is Sakhanda Wire's own summary.

← Back to the news

More stories

All the latest news