Tokenizers v1 reports major encoding speed gains
The release candidate for tokenizers v1 is designed to make text processing scale better in machine-learning workloads. Benchmarks across ten model families show encoding speeds 3 to 30 times higher than v0.23 with one thread on an Apple M4 Max. Performance reached 76% of linear scaling across eight workers, according to the measurements described in the project’s benchmark suite.
The new version is intended to preserve the existing vocabulary, merge ranks, API and token IDs, so the same inputs produce the same outputs. It continues to support BPE, WordPiece and Unigram tokenizers. The release candidate replaces some regular-expression splitting with SIMD-based bitstream operations, stores merge data in caller-provided memory, and uses a linked structure to reduce movement during BPE processing.
It also adds a thread-local word cache for repeated pre-tokens and allows one tokenizer to encode from multiple threads without a shared lock. A release candidate is available through crates.io using Cargo. Encoding calls remain unchanged, while encode_batch is the interface used for multi-core scaling. The reported figures measure the Rust crate and exclude additional overhead from Python bindings.
Written by Sakhanda Wire AI from reporting by Hugging Face. Facts, quotes and full context are in the original article below.