Sakhanda Wire
NVDA $152.94 -2.05% MSFT $505.64 -2.38% GOOGL $280.11 -1.27% META $619.47 -1.11% AMZN $256.54 -0.47%
← Back to the news

Olmo-core 3 targets trillion-parameter MoE training

Olmo-core 3, a new version of the framework used to develop Olmo language models, introduces an open training system for large mixture-of-experts (MoE) models. Its developers say the design is intended to scale into the trillion-parameter range while limiting the computational and communication costs associated with routing data among experts.

In one benchmark, the expert pool grew from 8 to 128 while four experts remained active for each token. Active parameters stayed at about 3.2B per token, while total capacity rose from 4.6B to 47B and throughput declined by less than 5%. On eight NVIDIA B300 GPUs, the new system processed 52,000 tokens per second per GPU for a 47B-parameter MoE, compared with 19,400 for the earlier implementation.

The framework combines expert, pipeline and distributed-optimizer parallelism with GPU-resident routing, grouped GEMM and other optimizations. MXFP8 increased throughput by about 21% over BF16 in a four-GPU test and reduced peak active memory from 103 GiB to 95 GiB. Tests included a 1.2T-parameter model across 512 GPUs, while a separate capacity test reached 2.38T parameters. Olmo-core 3 is intended for the next generation of Olmo and is available for researchers and developers to adapt and use.

Written by Sakhanda Wire AI from reporting by Hugging Face. Facts, quotes and full context are in the original article below.

Originally published by Hugging Face on

Read the original on Hugging Face ↗

The reporting and the image belong to Hugging Face and are credited and linked above; the text on this page is Sakhanda Wire's own summary.

← Back to the news

More stories

All the latest news