Atto: Rethinking Compact Embedding Models
What if useful semantic retrieval didn't require millions of parameters?
Can a model with just 840 parameters perform meaningful retrieval? Can a 90K-parameter dense embedding model approach the performance of models hundreds of times larger?
These questions motivated Atto, a family of compact contextual models exploring parameter efficiency across dense, sparse, and late-interaction retrieval.
Atto spans an unusually broad range of model sizes, from an 840-parameter ColBERT configuration to 28-million-parameter dense embedding models. Across these configurations, it explores how architectural choices, representation strategies, and training methods can deliver useful retrieval capabilities under constrained parameter budgets.
But the motivation extends beyond building smaller embedding models.
What if organizations could train and deploy specialized embedding models for their own domains and datasets without requiring the infrastructure traditionally associated with much larger architectures?
This article explores Atto's benchmark results, architecture, training methodology, and potential for making domain-specific model development more accessible.
Benchmark Results: How Small Can Retrieval Models Be?
Atto's evaluation covers three BEIR retrieval datasets:
The following table shows a subset of BEIR scored with the txtai benchmarks script. Scores reported are ndcg@10.
Dense Embeddings
Dense Atto models produce fixed-dimensional representations suitable for conventional vector search.
| Model | Parameters | Dimensions | Vocab | ArguAna | NFCorpus | SciFact | Average |
|---|---|---|---|---|---|---|---|
| atto-36k-384d | 36K | 384 | 1K | 0.3249 | 0.2286 | 0.4920 | 0.3485 |
| atto-50k-768d | 50K | 768 | 1K | 0.3264 | 0.2430 | 0.5334 | 0.3676 |
| atto-90k-2048d | 90K | 2048 | 1K | 0.4021 | 0.2676 | 0.6074 | 0.4257 |
| bert-hash-femto | 243K | 128 | 30K | 0.1964 | 0.1402 | 0.2830 | 0.2065 |
| bert-hash-pico | 442K | 128 | 30K | 0.2853 | 0.2075 | 0.3912 | 0.2947 |
| bert-hash-nano | 953K | 128 | 30K | 0.3828 | 0.2562 | 0.5032 | 0.3807 |
| atto-1m-2048d | 1M | 2048 | 4K | 0.4189 | 0.2941 | 0.6229 | 0.4453 |
| atto-1m-4096d | 1M | 4096 | 4K | 0.4223 | 0.3044 | 0.6417 | 0.4561 |
| atto-2m-8192d | 2M | 8192 | 4K | 0.4343 | 0.3125 | 0.6653 | 0.4707 |
| atto-14m-384d | 14M | 384 | 30K | 0.4874 | 0.3329 | 0.6472 | 0.4892 |
| atto-17m-8192d | 17M | 8192 | 30K | 0.4963 | 0.3455 | 0.6909 | 0.5109 |
| all-MiniLM-L6-v2 | 23M | 384 | 30K | 0.4789 | 0.3089 | 0.6527 | 0.4802 |
| atto-25m-384d | 25M | 384 | 30K | 0.4879 | 0.3426 | 0.6600 | 0.4968 |
| atto-28m-8192d | 28M | 8192 | 30K | 0.5111 | 0.3461 | 0.6843 | 0.5138 |
| DenseOn | 110M | 768 | 50K | 0.5408 | 0.3915 | 0.7591 | 0.5638 |
The 90K model achieves an average of 0.4257, compared with 0.4802 for all-MiniLM-L6-v2, despite having approximately 256 times fewer parameters.
The 2M model reaches 0.4707, approaching MiniLM's average with substantially fewer parameters. At the larger end, the 17M and 28M models achieve averages of 0.5109 and 0.5138, respectively.
These results suggest that compact contextual processing networks can produce competitive retrieval representations, particularly when combined with wider output dimensions.
Sparse Embeddings
Atto's sparse models operate over a fixed-dimensional learned latent space rather than vocabulary-sized output representations.
| Model | Parameters | Dimensions | Vocab | ArguAna | NFCorpus | SciFact | Average |
|---|---|---|---|---|---|---|---|
| atto-17m-8192d-sparse | 17M | 8192 | 30K | 0.3981 | 0.3208 | 0.6600 | 0.4596 |
| atto-28m-8192d-sparse | 28M | 8192 | 30K | 0.3651 | 0.3292 | 0.6618 | 0.4520 |
| SlideOn | 174M | 768 | 50K | 0.4655 | 0.3863 | 0.7529 | 0.5349 |
The 17M sparse model achieves an average of 0.4596, retaining approximately 90% of the corresponding dense model's average score.
This provides an alternative approach to sparse semantic retrieval without requiring vocabulary-sized output representations.
ColBERT and Late Interaction
Atto's late-interaction family explores parameter efficiency at an even smaller scale.
| Model | Parameters | Dimensions | Vocab | ArguAna | NFCorpus | SciFact | Average |
|---|---|---|---|---|---|---|---|
| colbert-atto-840-8d | 840 | 8 | 250 | 0.2143 | 0.2585 | 0.4688 | 0.3139 |
| colbert-atto-9k-16d | 9K | 16 | 1K | 0.2459 | 0.2902 | 0.5604 | 0.3655 |
| colbert-atto-90k-32d | 90K | 32 | 4K | 0.3435 | 0.3263 | 0.6663 | 0.4454 |
| colbert-atto-950k-128d | 950K | 128 | 4K | 0.4149 | 0.3352 | 0.6946 | 0.4816 |
| colbert-atto-4m-384d | 4M | 384 | 30K | 0.4157 | 0.3349 | 0.6949 | 0.4818 |
| colbert-atto-14m-384d | 14M | 384 | 30K | 0.4248 | 0.3483 | 0.7057 | 0.4929 |
| colbert-atto-25m-384d | 25M | 384 | 30K | 0.4395 | 0.3461 | 0.7130 | 0.4995 |
| LateOn | 110M | 768 | 50K | 0.4979 | 0.3802 | 0.7609 | 0.5463 |
Two results stand out.
First, the 840-parameter model achieves an average of 0.3139. Second, the 950K model achieves virtually identical performance to the 4M model:
- 950K parameters: 0.4816.
- 4M parameters: 0.4818.
The 950K model uses a smaller vocabulary (4K) but a deeper architecture (4 layers), whereas the 4M model uses a 30K vocabulary and 2 layers.
Interpreting the Results
The benchmarks illustrate several operating points across the Atto family.
Smaller models prioritize parameter efficiency, while larger models generally offer stronger retrieval quality. Representation dimensionality, vocabulary size, pooling strategy, and retrieval architecture also influence performance.
The Atto Architecture
Atto uses a lightweight sequence-processing architecture built around two primary components:
- GatedConvAttentionMixer: Combines depthwise convolution with globally contextualized multi-head attention.
- PhaseGatedMoE: Combines local and dilated convolutional experts through parameter-free cross-scale modulation.
Together, these components provide local and global contextual processing without relying exclusively on conventional Transformer blocks.
Vocabulary Size and Parameter Efficiency
Atto explores vocabulary sizes ranging from 250 to 30K tokens. Vocabulary size directly affects model parameter count, as larger vocabularies require larger token embedding tables. Atto also incorporates hash embeddings as found with the BERT Hash series of models. With hash embeddings, a larger vocabulary can be projected to a smaller space to help further reduce the parameter count.
Smaller vocabularies reduce embedding parameters but can increase token fragmentation and sequence lengths. Larger vocabularies offer more efficient tokenization at the cost of additional parameters.
Atto explores this tradeoff as another dimension of compact model design.
GatedConvAttentionMixer
The sequence mixer combines local contextual processing with global information exchange.
It first applies depthwise convolution to the hidden representations, providing local context for each token.
The resulting features pass through fused projections that produce local gates, local values, and attention queries, keys, and values.
A directional gating mechanism combines two nonlinear activation streams to produce a bounded gating signal. This signal constructs local features that condition the attention projections.
The same locally derived conditioning signal is applied to the query, key, and value representations, allowing local contextual information to influence global attention.
The mixer then combines local convolutional features and the global attention output into a single residual update.
PhaseGatedMoE
The second component provides multi-scale feature processing.
It uses two depthwise convolutional experts:
- A local expert with a kernel size of 3.
- A dilated expert with a kernel size of 3 and dilation of 2.
The local expert captures nearby phrase-level information, while the dilated expert processes a wider contextual region.
Rather than using a conventional learned expert router, the wider-context expert produces a bounded modulation signal that adjusts the local expert's output.
The resulting features pass through normalization and asymmetric gating before being projected back into the hidden representation.
This creates cross-scale interactions without introducing a separate learned routing network.
Residual Processing
Each Atto layer contains one sequence mixer and one PhaseGatedMoE block.
Both operate within a Pre-LayerNorm residual architecture, with separate normalization stages for the mixer and expert components.
The combination of convolutional feature extraction, global attention, and multi-scale gated processing forms the foundation of Atto's compact contextual architecture.
Compact Contextual Processing, Expanded Embedding Space
One of Atto's defining architectural choices is the separation between its internal processing dimensions and its final embedding dimensions.
Rather than performing all contextual processing in a high-dimensional embedding space, Atto first processes token representations in a comparatively compact hidden space.
After contextual processing, a learned linear projection expands each token representation into the target embedding space.
This is particularly important for configurations producing 2048-, 4096-, and 8192-dimensional representations.
The architecture separates two tasks:
- Contextual understanding takes place in a compact internal representation.
- Learned dimensional expansion produces the wider representations used for retrieval.
This allows Atto to explore high-dimensional embeddings without requiring the entire contextual processing network to operate at the final embedding dimensionality.
Mean and Max Pooling
Atto supports different strategies for converting contextual token representations into document-level embeddings.
Most dense configurations use dimension-wise max pooling. Each embedding dimension independently retains its strongest activation across the input sequence.
Different dimensions can therefore preserve salient information originating from different token positions.
Atto also includes mean-pooled configurations. The 14M and 25M parameter models with 384-dimensional embeddings use mean pooling, providing a complementary approach to document representation.
The two strategies offer different tradeoffs between feature selection and aggregation.
The Connection to Late Interaction
There is an interesting conceptual connection between Atto's max pooling and late-interaction retrieval.
ColBERT preserves individual token representations and performs token-to-token matching during retrieval.
Atto's max-pooled dense representations instead perform dimension-wise selection during encoding.
These approaches are not equivalent, but both emphasize preserving salient token-level information rather than relying exclusively on averaging.
Atto consequently offers an interesting point in the design space between conventional pooled embeddings and explicit token-level late interaction.
Training Atto: From Language Modeling to Retrieval
Atto separates general language representation learning from retrieval-specific optimization.
The process begins with masked language modeling (MLM), followed by knowledge distillation to transfer retrieval capabilities into compact architectures.
- Base models: Masked language modeling on English Wikipedia.
- Dense: Distilled from DenseOn using MS MARCO 500K Triplets.
- Sparse: Distilled from DenseOn using MS MARCO 500K Triplets.
- ColBERT: Distilled using LightOn AI's MS MARCO BGE-Gemma dataset.
Base Model Pretraining
The foundation of the Atto family is masked language modeling on English Wikipedia.
During MLM training, portions of the input text are masked, and the model learns to reconstruct missing tokens from their surrounding context.
This provides the base models with general contextual language representations, including vocabulary, linguistic structure, and relationships between words.
Dense and Sparse Distillation
Both dense and sparse Atto models are distilled from DenseOn using the MS MARCO 500K Triplets dataset.
The dataset provides query, positive passage, and negative passage examples for retrieval supervision.
Knowledge distillation allows the smaller Atto models to learn retrieval-oriented representations from the larger teacher model.
Using the same teacher and dataset for both retrieval families provides a common foundation while allowing different representation strategies to be explored.
ColBERT Distillation
The ColBERT family follows a separate training pipeline using the LightOn AI MS MARCO BGE-Gemma dataset.
This provides retrieval supervision for training compact token-level late-interaction models.
Unlike dense retrieval, which produces a single vector for an entire query or document, ColBERT preserves token-level representations for more granular matching.
The 840-Parameter Model: How Small Can Retrieval Become?
One of the most unusual configurations in the Atto family is colbert-atto-840-8d.
With just 840 parameters, it achieves an average retrieval score of 0.3139 across ArguAna, NFCorpus, and SciFact.
For comparison:
| Model | Parameters | Average |
|---|---|---|
| BERT Hash Femto | 243K | 0.2065 |
| BERT Hash Pico | 442K | 0.2947 |
| Atto ColBERT | 840 | 0.3139 |
The result is striking because the Atto configuration outperforms both baselines despite having dramatically fewer parameters.
However, these models use different retrieval architectures, so this is primarily an illustration of parameter efficiency rather than a direct architectural comparison.
The 840-parameter model is a ColBERT-style late-interaction model, not a conventional dense embedding model. It retains token-level representations rather than compressing an entire document into one vector.
Consequently, parameter count alone does not describe the total retrieval-system cost. Document representation storage, indexing, and late-interaction scoring also matter.
The result should not be interpreted as suggesting that an 840-parameter model can replace larger general-purpose embedding models.
Instead, it illustrates how far parameter efficiency can be explored when architecture, representation dimensionality, and retrieval strategy are considered together.
Making Domain-Specific Embeddings Practical
One of the primary motivations behind Atto is making specialized embedding models more accessible.
Many applications do not need a massive general-purpose embedding model. They need a model that understands a particular collection of documents, terminology, vocabulary, and retrieval requirements.
Consider several examples:
- Healthcare: Medical literature, clinical documentation, and specialized terminology.
- Legal: Case law, contracts, legislation, and regulatory documents.
- Scientific research: Research papers and specialized technical terminology.
- Enterprise knowledge: Internal documentation, technical manuals, and organizational knowledge.
- Software engineering: Source code documentation, API references, and technical repositories.
In these environments, a model trained or adapted for a specific retrieval task could potentially provide better alignment with the target data than a general-purpose model.
Atto explores whether compact architectures can make this kind of specialization more practical.
Multiple Models for Multiple Domains
Instead of maintaining one large embedding model for every application, organizations could potentially maintain several smaller models, each optimized for a particular retrieval problem.
For example, an organization might deploy separate models for legal research, internal documentation, scientific publications, and software repositories.
This creates flexibility when different datasets have different vocabularies, terminology, and retrieval requirements.
It also creates opportunities to experiment with domain-specific training and fine-tuning without necessarily requiring the infrastructure associated with much larger architectures.
Model Availability and Usage
Atto models are available through the Hugging Face Hub.
Explore the model family:
The family offers several retrieval configurations:
- Dense: Single-vector semantic retrieval using conventional vector search.
- Sparse: Sparse latent-space retrieval using learned contextual features.
- ColBERT: Token-level late-interaction retrieval for granular query-document matching.
When selecting a checkpoint, consider parameter count, embedding dimensionality, vocabulary size, pooling strategy, retrieval quality, and indexing requirements.
Consult the individual model cards for configuration details, loading instructions, and usage examples.
Conclusion
Atto explores how far parameter efficiency can be pushed in contextual embedding models.
By combining convolution-conditioned attention, multi-scale gated processing, and learned dimensional expansion, it investigates a range of retrieval architectures spanning dense, sparse, and token-level representations.
The benchmark results demonstrate promising retrieval capabilities across a wide range of model sizes, from an 840-parameter ColBERT configuration to 28-million-parameter dense embeddings.
However, the broader motivation behind Atto is not simply minimizing parameter counts.
It is making specialized models more practical to develop and deploy.
Smaller architectures create opportunities for organizations to experiment with embedding models tailored to their own datasets, terminology, and retrieval requirements. Rather than relying exclusively on large general-purpose models, different applications could potentially use specialized models optimized for their particular domains.
There is also an opportunity to explore compact pretrained architectures for specialized decision models, extending Atto beyond semantic retrieval.
Ultimately, the interesting question is not just how small an embedding model can become, but how much useful capability can be delivered within a given computational budget.
Atto is not simply about smaller embedding models. It is about making specialized models practical.
Atto: compact contextual processing, expanded representations, and a more accessible path to specialized models.




