Comparison maps routing choices across 10 open MoE models
A configuration-verified comparison examines how 10 open mixture-of-experts (MoE) models assign tokens to experts. The models range from Qwen3.6-35B-A3B, with 35B total and 3B active parameters, to DeepSeek-V4-Pro, with 1.6T total and 49B active. Their routed expert counts span 8 to 384, while top-k selection ranges from one expert in Llama 4 Maverick to eight in several models.
All models use token-choice routing, but differ in scoring, shared experts and load balancing. Seven retain one shared expert; Mixtral, gpt-oss and Qwen3 do not. DeepSeek-V3 uses sigmoid scoring with bias-based, auxiliary-loss-free balancing, while DeepSeek-V4-Pro uses sqrtsoftplus, six selected experts and hash routing in its first three MoE layers. Kimi K2 uses 384 experts and top-8 routing; its report found that higher sparsity required fewer FLOPs at the same validation loss.
The comparison also notes operational differences. gpt-oss-120b uses top-4 routing across 128 experts and can fit on one 80 GB GPU with MXFP4 weights. DeepSeek-V4-Pro and Kimi K2 require multi-GPU expert parallelism. Gating details for Llama 4 Maverick and Qwen3.6, and balancing details for Llama 4 Maverick and gpt-oss, are not published in their configurations.
Written by Sakhanda Wire AI from reporting by MarkTechPost. Facts, quotes and full context are in the original article below.