# Atto Embeddings: A Smaller Model Can Make Your Vector Index Bigger

Compare NeuML's 90K and 14M Atto encoders by index size, retrieval quality and CPU latency before you replace an existing embedding model.

By Paras Madan | 2026-11-20


A tiny embedding model can produce a surprisingly large search index. NeuML's [November 19 Atto release](https://huggingface.co/blog/NeuML/introducing-atto) makes that tradeoff easy to see: a 90,000-parameter encoder returns 2,048 numbers per chunk, while a 14-million-parameter sibling returns 384.

If you are choosing a retrieval model for a laptop or a small server, decide which part is under pressure first. A smaller encoder can help when every query competes for CPU. A narrower vector can matter more when you need to keep a large corpus searchable in memory.

## Put a price on each vector

For uncompressed float32 vectors, multiply the number of chunks by the embedding dimension and four bytes. Here is the arithmetic for one million chunks, not a measurement of either model:

| Atto dense checkpoint | Dimensions | Bytes per chunk | Raw vectors for 1M chunks |
| --- | ---: | ---: | ---: |
| 90K | 2,048 | 8,192 | 8.192 GB |
| 14M | 384 | 1,536 | 1.536 GB |

Those totals use decimal GB and exclude search structures, document metadata and replicas. If each document contributes multiple chunks, count the chunks, not the documents. Quantization or compression can change the totals, but compare those configurations at the retrieval quality you need rather than assuming the savings are free.

For a documentation assistant with a million chunks and a tight memory budget, the 14M checkpoint is the better first candidate on index size alone. For a small local collection where encoder startup and CPU use dominate, the 90K checkpoint deserves a separate test. The table cannot tell you which model actually encodes faster on your hardware.

## Treat the published scores as a starting point

NeuML reports average nDCG@10 of about 0.426 for the 90K dense checkpoint and 0.489 for the 14M dense checkpoint across ArguAna, NFCorpus and SciFact. This measures how well those tests rank relevant items near the top. It is not a guarantee about your own corpus.

Keep exact names, product codes and policy exceptions in your evaluation. A model can do well on broadly relevant results while repeatedly missing the one paragraph that changes the answer. Check the proportion of queries that retrieve the required passage in the first few results, not just whether an answer sounds useful after generation.

## Load the dense variants deliberately

The [90K card](https://huggingface.co/NeuML/atto-90k-2048d-embeddings) and [14M card](https://huggingface.co/NeuML/atto-14m-384d-embeddings) describe English, Apache-2.0 checkpoints. Their examples support Sentence Transformers and txtai and opt into custom model code with `trust_remote_code=True`.

Review the model code and pin the revision before allowing it to run. Use an isolated environment and public sample text for the first load, not a worker with production documents or service credentials.

The cards record distinct `query: ` and `document: ` training prompts. Decide how your library applies those prompts and keep that choice explicit. Start by checking the emitted vector dimensions for a query and a document. Then measure the time for one query and for a document batch separately; those are different jobs.

## Use a separate index for the trial

Re-embed a fixed corpus snapshot into an isolated index under each checkpoint. Even when two encoders happen to output the same width, their vectors are not interchangeable. Keep the incumbent unchanged until the new query encoder and new index can be switched together.

For each run, record build time, stored bytes, search memory, and end-to-end query latency. Measure both a warmed-up server and a cold start if your application frequently sleeps. Keep chunking, text normalization and the held-out queries identical so changes in results can be traced back to the embedding pipeline.

If you compare compressed indexes, record the compression settings alongside the quality scores. It is easy to accidentally compare a compact model with an expensive index against a larger model with a cheaper one, then credit all the difference to the encoder.

## Don't substitute the 840-parameter checkpoint

The [840-parameter ColBERT variant](https://huggingface.co/NeuML/colbert-atto-840-8d) returns multiple token vectors and uses late-interaction scoring. Its card documents txtai 9.0+ and the Sentence Transformers multi-vector interface. It does not replace a single dense vector per chunk.

If you already have a dense index, try late interaction as a separate reranking experiment over a fixed candidate list. Compare quality gains against additional latency, and keep that result separate from testing a full multi-vector index.

Choose the smallest system that meets your retrieval and latency targets. That may turn out to be the larger encoder with the narrower index.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI teams turn model releases and engineering tradeoffs into technical writing developers can use.

[Book a 30-minute call](https://cal.com/paras-madan/30min)
