# Token Pooling for Multi-Vector Search: Reduce Index Size Without Guessing

Sentence Transformers makes compressed retrieval easier to evaluate, but fewer vectors do not automatically mean the same reduction in your search bill.

By Paras Madan | 2026-11-12

Token pooling reduces the number of document vectors stored by a multi-vector search system. Sentence Transformers now documents pooling-aware training alongside downloadable example models, giving teams a practical way to test whether retrieval quality survives that compression. Start with an index-size experiment, not a promise of four-times-cheaper RAG.

In the [November 11 walkthrough](https://huggingface.co/blog/token-pooling), Tom Aarsen reports that pooling-aware training retained 97.8% of its model's unpooled NanoBEIR nDCG@10 score at a factor-four pooling setting. The ordinary-training baseline retained 94.6% of its own unpooled score. Those are retrieval benchmark results, not answer accuracy or guaranteed savings on a production system.

## First identify which part of the index is expensive

A single-vector encoder represents a document with one vector. Multi-vector retrieval keeps several contextual vectors, allowing different query terms to match different parts of the document. That flexibility can make the stored representation much larger.

There are three distinct compression knobs. Quantization uses fewer bits per coordinate. Dimensionality reduction keeps fewer coordinates per vector. Pooling keeps fewer vectors per document. They can be combined, but changing them simultaneously makes it harder to discover which one damaged a useful match.

Consider a hypothetical corpus of 500,000 documents, each represented by 160 vectors of 128 dimensions at four bits per coordinate. The vector payload alone is 5.12 GB in decimal units. Reducing that to 40 vectors per document would make the same payload 1.28 GB. This is arithmetic, not a measured index: metadata, identifiers, search-engine structures and retained source documents still need space.

**Budget against the complete index, not only its vector payload.** If vectors account for half your stored bytes, reducing that half by 75% cuts total storage by 37.5%, not 75%. Measure the actual contribution before forecasting infrastructure savings.

## Choose one pooling budget

The [Sentence Transformers guide](https://sbert.net/examples/multi_vector_encoder/training/token_pooling/README.html) describes three alternative controls:

| Control | What you are asking for | Main trade-off |
| --- | --- | --- |
| `pool_factor=4` | Roughly one quarter of each document's vectors | Longer documents retain more capacity |
| `max_tokens=16` | At most 16 vectors per document | A long document gets the same ceiling as a short one |
| `distance_threshold` | Merge according to embedding similarity | Calibrate an average budget rather than expecting a fixed count |

Configure one control at a time. Query vectors stay unpooled by default, and the first document token is protected. The encoder still processes the document through its full backbone; this is not a technique for skipping three quarters of the encoding work.

For an initial experiment, proportional pooling is easier to interpret than an adaptive threshold. After establishing that baseline, adaptive pooling can allocate more representation to documents whose embeddings contain more distinct information. Its calibrated average can change when the corpus changes, so a threshold fitted on short support articles is not a storage guarantee for lengthy technical manuals.

## Compare full and compressed representations on the same queries

The released [pooling-aware checkpoint](https://huggingface.co/tomaarsen/mve-ModernBERT-base-token-pooling) provides a starting point. The following documentation-derived example compares counts and scores without inventing an expected result. It requires a Sentence Transformers installation exposing `MultiVectorEncoder` and `HierarchicalTokenPooling`; the accompanying upstream training example declares version 6.3.0 or later.

```python
from sentence_transformers import MultiVectorEncoder
from sentence_transformers.multi_vector_encoder.modules import HierarchicalTokenPooling

model = MultiVectorEncoder("tomaarsen/mve-ModernBERT-base-token-pooling")
queries = ["Can an annual plan be refunded after renewal?"]
documents = [
    "Annual renewals can be refunded within seven days.",
    "Monthly subscriptions can be cancelled before the next invoice.",
    "Annual plans include a discount compared with monthly billing.",
]
q = model.encode_query(queries)
full = model.encode_document(documents)
pooled = model.encode_document(
    documents, token_pooling=HierarchicalTokenPooling(pool_factor=4)
)
print("Full counts:", [len(x) for x in full])
print("Pooled counts:", [len(x) for x in pooled])
print("Full scores:", model.similarity(q, full))
print("Pooled scores:", model.similarity(q, pooled))
```

This example has not been executed here. Its purpose is to expose what changes, not establish that three synthetic documents are a meaningful benchmark. Replace them with a held-out set of real queries and relevance labels, then inspect failures involving small but important distinctions such as renewal versus initial purchase.

Run the comparison at unchanged precision and vector width. Record retrieval quality, actual stored bytes, indexing time and query latency separately. A smaller representation is useful only if the savings survive the search engine around it and the lost matches stay within your application's tolerance.

## Train only after measuring the compression penalty

If pooling harms retrieval too much, the [upstream training script](https://github.com/huggingface/sentence-transformers/blob/main/examples/multi_vector_encoder/training/token_pooling/training_cached_contrastive.py) provides matched ordinary-loss and pooling-aware runs. The latter combines losses from full and compressed representations, teaching one encoder to work with both.

Review that script before running it on company data. It saves a local model and then attempts a Hugging Face Hub upload using available authentication. Remove or disable that upload step if your intended experiment is local-only. Training a model and publishing its weights are separate decisions.

The reference experiment uses 120,000 MS MARCO triplets and matched initialization, so it isolates the loss change rather than proving the best possible configuration for every corpus. It is a starting comparison, not a universal replacement for a stronger retrieval model.

The useful decision is whether a smaller index preserves the distinctions your users search for. Set that acceptance threshold before tuning compression. Otherwise, it is easy to celebrate a tidy storage number while quietly making the search product less dependable.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help technical teams explain retrieval systems, infrastructure trade-offs and product decisions in content developers can actually use.

[Book a 30-minute call](https://cal.com/paras-madan/30min)