Olmo-core 3 targets trillion-parameter MoE training
Olmo-core 3, a new version of the framework used to develop Olmo language models, introduces an open training system for large mixture-of-experts (MoE) models. Its developers say the design is intended to scale into the trillion-parameter range while limiting the computational and communication costs associated with routing data among experts.
In one benchmark, the expert pool grew from 8 to 128 while four experts remained active for each token. Active parameters stayed at about 3.2B per token, while total capacity rose from 4.6B to 47B and throughput declined by less than 5%. On eight NVIDIA B300 GPUs, the new system processed 52,000 tokens per second per GPU for a 47B-parameter MoE, compared with 19,400 for the earlier implementation.
The framework combines expert, pipeline and distributed-optimizer parallelism with GPU-resident routing, grouped GEMM and other optimizations. MXFP8 increased throughput by about 21% over BF16 in a four-GPU test and reduced peak active memory from 103 GiB to 95 GiB. Tests included a 1.2T-parameter model across 512 GPUs, while a separate capacity test reached 2.38T parameters. Olmo-core 3 is intended for the next generation of Olmo and is available for researchers and developers to adapt and use.
Written by Sakhanda Wire AI from reporting by Hugging Face. Facts, quotes and full context are in the original article below.