Sakhanda Wire
NVDA $154.51 -1.04% MSFT $506.49 -2.22% GOOGL $281.27 -0.86% META $621.51 -0.78% AMZN $256.90 -0.33%
← Back to the news

Tokenizers v1 reports major encoding speed gains

The release candidate for tokenizers v1 is designed to make text processing scale better in machine-learning workloads. Benchmarks across ten model families show encoding speeds 3 to 30 times higher than v0.23 with one thread on an Apple M4 Max. Performance reached 76% of linear scaling across eight workers, according to the measurements described in the project’s benchmark suite.

The new version is intended to preserve the existing vocabulary, merge ranks, API and token IDs, so the same inputs produce the same outputs. It continues to support BPE, WordPiece and Unigram tokenizers. The release candidate replaces some regular-expression splitting with SIMD-based bitstream operations, stores merge data in caller-provided memory, and uses a linked structure to reduce movement during BPE processing.

It also adds a thread-local word cache for repeated pre-tokens and allows one tokenizer to encode from multiple threads without a shared lock. A release candidate is available through crates.io using Cargo. Encoding calls remain unchanged, while encode_batch is the interface used for multi-core scaling. The reported figures measure the Rust crate and exclude additional overhead from Python bindings.

Written by Sakhanda Wire AI from reporting by Hugging Face. Facts, quotes and full context are in the original article below.

Originally published by Hugging Face on

Read the original on Hugging Face ↗

The reporting and the image belong to Hugging Face and are credited and linked above; the text on this page is Sakhanda Wire's own summary.

← Back to the news

More stories

All the latest news