NeoMME introduces compact multilingual multimodal encoders
Hugging Face has introduced NeoMME, a family of multilingual multimodal encoders with 260M and 800M parameters. Unlike architectures that combine a pretrained vision tower with a causal language model, NeoMME uses one bidirectional Transformer for text tokens and raw 32×32 image patches. The models were trained from scratch with a masked discrete-diffusion objective and support text, images, code, mathematics and document pages.
A fine-tuned version, NeoMME-Retriever, is designed for visual document retrieval. It processes document pages as images, avoiding OCR preprocessing, and produces dense and late-interaction embeddings in one forward pass. On ViDoRe v3, the 260M model recorded 0.523 nDCG@10, while the 800M version reached 0.556. The article places both models on the benchmark’s model-size Pareto frontier.
At 2048×2048 resolution on an NVIDIA L40S, NeoMME-Retriever-260M encoded about 51 pages per second. Hierarchical token pooling and asymmetric quantization reduced late-interaction storage from about 1.5 MB to 6 kB per page in one tested configuration, while retaining more than 95% of baseline nDCG@10. The models and checkpoints are available in Hugging Face Transformers under the Apache 2.0 license.
Written by Sakhanda Wire AI from reporting by Hugging Face. Facts, quotes and full context are in the original article below.