Pooling-aware training improves compressed multi-vector retrieval
Multi-vector embedding models represent a document with separate contextual vectors for its tokens. This can improve search because different query terms may match different document passages, but the approach requires substantial storage. Token pooling reduces that cost by merging similar vectors. HierarchicalTokenPooling uses Ward clustering and represents each resulting group by its mean vector.
In an example with pool_factor=4, an ordinarily trained model preserved 94.6% of its unpooled NanoBEIR nDCG@10 while reducing the number of document vectors fourfold. Pooling-aware training increased retention to 97.8% with the same storage budget. The method adds a retrieval loss for the original document representations and another after pooling, then averages the two losses.
The training process uses a straight-through estimator so retrieval errors from pooled vectors can update the shared encoder without differentiating the clustering assignments. Sentence Transformers provides MultiVectorTokenPoolingLoss for this setup. Pooling can also be controlled with max_tokens or a distance threshold; the latter can be calibrated for a target average vector count. The post includes reproducible training code, optional kernel acceleration, and evaluation examples.
Written by Sakhanda Wire AI from reporting by Hugging Face. Facts, quotes and full context are in the original article below.