Atto: Rethinking Compact Embedding Models

Community Article
Published November 19, 2026

atto

What if useful semantic retrieval didn't require millions of parameters?

Can a model with just 840 parameters perform meaningful retrieval? Can a 90K-parameter dense embedding model approach the performance of models hundreds of times larger?

These questions motivated Atto, a family of compact contextual models exploring parameter efficiency across dense, sparse, and late-interaction retrieval.

Atto spans an unusually broad range of model sizes, from an 840-parameter ColBERT configuration to 28-million-parameter dense embedding models. Across these configurations, it explores how architectural choices, representation strategies, and training methods can deliver useful retrieval capabilities under constrained parameter budgets.

But the motivation extends beyond building smaller embedding models.

What if organizations could train and deploy specialized embedding models for their own domains and datasets without requiring the infrastructure traditionally associated with much larger architectures?

This article explores Atto's benchmark results, architecture, training methodology, and potential for making domain-specific model development more accessible.


Benchmark Results: How Small Can Retrieval Models Be?

Atto's evaluation covers three BEIR retrieval datasets:

The following table shows a subset of BEIR scored with the txtai benchmarks script. Scores reported are ndcg@10.


Dense Embeddings

Dense Atto models produce fixed-dimensional representations suitable for conventional vector search.

Model Parameters Dimensions Vocab ArguAna NFCorpus SciFact Average
atto-36k-384d 36K 384 1K 0.3249 0.2286 0.4920 0.3485
atto-50k-768d 50K 768 1K 0.3264 0.2430 0.5334 0.3676
atto-90k-2048d 90K 2048 1K 0.4021 0.2676 0.6074 0.4257
bert-hash-femto 243K 128 30K 0.1964 0.1402 0.2830 0.2065
bert-hash-pico 442K 128 30K 0.2853 0.2075 0.3912 0.2947
bert-hash-nano 953K 128 30K 0.3828 0.2562 0.5032 0.3807
atto-1m-2048d 1M 2048 4K 0.4189 0.2941 0.6229 0.4453
atto-1m-4096d 1M 4096 4K 0.4223 0.3044 0.6417 0.4561
atto-2m-8192d 2M 8192 4K 0.4343 0.3125 0.6653 0.4707
atto-14m-384d 14M 384 30K 0.4874 0.3329 0.6472 0.4892
atto-17m-8192d 17M 8192 30K 0.4963 0.3455 0.6909 0.5109
all-MiniLM-L6-v2 23M 384 30K 0.4789 0.3089 0.6527 0.4802
atto-25m-384d 25M 384 30K 0.4879 0.3426 0.6600 0.4968
atto-28m-8192d 28M 8192 30K 0.5111 0.3461 0.6843 0.5138
DenseOn 110M 768 50K 0.5408 0.3915 0.7591 0.5638

The 90K model achieves an average of 0.4257, compared with 0.4802 for all-MiniLM-L6-v2, despite having approximately 256 times fewer parameters.

The 2M model reaches 0.4707, approaching MiniLM's average with substantially fewer parameters. At the larger end, the 17M and 28M models achieve averages of 0.5109 and 0.5138, respectively.

These results suggest that compact contextual processing networks can produce competitive retrieval representations, particularly when combined with wider output dimensions.


Sparse Embeddings

Atto's sparse models operate over a fixed-dimensional learned latent space rather than vocabulary-sized output representations.

Model Parameters Dimensions Vocab ArguAna NFCorpus SciFact Average
atto-17m-8192d-sparse 17M 8192 30K 0.3981 0.3208 0.6600 0.4596
atto-28m-8192d-sparse 28M 8192 30K 0.3651 0.3292 0.6618 0.4520
SlideOn 174M 768 50K 0.4655 0.3863 0.7529 0.5349

The 17M sparse model achieves an average of 0.4596, retaining approximately 90% of the corresponding dense model's average score.

This provides an alternative approach to sparse semantic retrieval without requiring vocabulary-sized output representations.


ColBERT and Late Interaction

Atto's late-interaction family explores parameter efficiency at an even smaller scale.

Model Parameters Dimensions Vocab ArguAna NFCorpus SciFact Average
colbert-atto-840-8d 840 8 250 0.2143 0.2585 0.4688 0.3139
colbert-atto-9k-16d 9K 16 1K 0.2459 0.2902 0.5604 0.3655
colbert-atto-90k-32d 90K 32 4K 0.3435 0.3263 0.6663 0.4454
colbert-atto-950k-128d 950K 128 4K 0.4149 0.3352 0.6946 0.4816
colbert-atto-4m-384d 4M 384 30K 0.4157 0.3349 0.6949 0.4818
colbert-atto-14m-384d 14M 384 30K 0.4248 0.3483 0.7057 0.4929
colbert-atto-25m-384d 25M 384 30K 0.4395 0.3461 0.7130 0.4995
LateOn 110M 768 50K 0.4979 0.3802 0.7609 0.5463

Two results stand out.

First, the 840-parameter model achieves an average of 0.3139. Second, the 950K model achieves virtually identical performance to the 4M model:

  • 950K parameters: 0.4816.
  • 4M parameters: 0.4818.

The 950K model uses a smaller vocabulary (4K) but a deeper architecture (4 layers), whereas the 4M model uses a 30K vocabulary and 2 layers.


Interpreting the Results

The benchmarks illustrate several operating points across the Atto family.

Smaller models prioritize parameter efficiency, while larger models generally offer stronger retrieval quality. Representation dimensionality, vocabulary size, pooling strategy, and retrieval architecture also influence performance.


The Atto Architecture

architecture

Atto uses a lightweight sequence-processing architecture built around two primary components:

  • GatedConvAttentionMixer: Combines depthwise convolution with globally contextualized multi-head attention.
  • PhaseGatedMoE: Combines local and dilated convolutional experts through parameter-free cross-scale modulation.

Together, these components provide local and global contextual processing without relying exclusively on conventional Transformer blocks.


Vocabulary Size and Parameter Efficiency

Atto explores vocabulary sizes ranging from 250 to 30K tokens. Vocabulary size directly affects model parameter count, as larger vocabularies require larger token embedding tables. Atto also incorporates hash embeddings as found with the BERT Hash series of models. With hash embeddings, a larger vocabulary can be projected to a smaller space to help further reduce the parameter count.

Smaller vocabularies reduce embedding parameters but can increase token fragmentation and sequence lengths. Larger vocabularies offer more efficient tokenization at the cost of additional parameters.

Atto explores this tradeoff as another dimension of compact model design.


GatedConvAttentionMixer

The sequence mixer combines local contextual processing with global information exchange.

It first applies depthwise convolution to the hidden representations, providing local context for each token.

The resulting features pass through fused projections that produce local gates, local values, and attention queries, keys, and values.

A directional gating mechanism combines two nonlinear activation streams to produce a bounded gating signal. This signal constructs local features that condition the attention projections.

The same locally derived conditioning signal is applied to the query, key, and value representations, allowing local contextual information to influence global attention.

The mixer then combines local convolutional features and the global attention output into a single residual update.


PhaseGatedMoE

The second component provides multi-scale feature processing.

It uses two depthwise convolutional experts:

  • A local expert with a kernel size of 3.
  • A dilated expert with a kernel size of 3 and dilation of 2.

The local expert captures nearby phrase-level information, while the dilated expert processes a wider contextual region.

Rather than using a conventional learned expert router, the wider-context expert produces a bounded modulation signal that adjusts the local expert's output.

The resulting features pass through normalization and asymmetric gating before being projected back into the hidden representation.

This creates cross-scale interactions without introducing a separate learned routing network.


Residual Processing

Each Atto layer contains one sequence mixer and one PhaseGatedMoE block.

Both operate within a Pre-LayerNorm residual architecture, with separate normalization stages for the mixer and expert components.

The combination of convolutional feature extraction, global attention, and multi-scale gated processing forms the foundation of Atto's compact contextual architecture.


Compact Contextual Processing, Expanded Embedding Space

One of Atto's defining architectural choices is the separation between its internal processing dimensions and its final embedding dimensions.

Rather than performing all contextual processing in a high-dimensional embedding space, Atto first processes token representations in a comparatively compact hidden space.

After contextual processing, a learned linear projection expands each token representation into the target embedding space.

This is particularly important for configurations producing 2048-, 4096-, and 8192-dimensional representations.

The architecture separates two tasks:

  1. Contextual understanding takes place in a compact internal representation.
  2. Learned dimensional expansion produces the wider representations used for retrieval.

This allows Atto to explore high-dimensional embeddings without requiring the entire contextual processing network to operate at the final embedding dimensionality.


Mean and Max Pooling

Atto supports different strategies for converting contextual token representations into document-level embeddings.

Most dense configurations use dimension-wise max pooling. Each embedding dimension independently retains its strongest activation across the input sequence.

Different dimensions can therefore preserve salient information originating from different token positions.

Atto also includes mean-pooled configurations. The 14M and 25M parameter models with 384-dimensional embeddings use mean pooling, providing a complementary approach to document representation.

The two strategies offer different tradeoffs between feature selection and aggregation.


The Connection to Late Interaction

There is an interesting conceptual connection between Atto's max pooling and late-interaction retrieval.

ColBERT preserves individual token representations and performs token-to-token matching during retrieval.

Atto's max-pooled dense representations instead perform dimension-wise selection during encoding.

These approaches are not equivalent, but both emphasize preserving salient token-level information rather than relying exclusively on averaging.

Atto consequently offers an interesting point in the design space between conventional pooled embeddings and explicit token-level late interaction.


Training Atto: From Language Modeling to Retrieval

training

Atto separates general language representation learning from retrieval-specific optimization.

The process begins with masked language modeling (MLM), followed by knowledge distillation to transfer retrieval capabilities into compact architectures.

  • Base models: Masked language modeling on English Wikipedia.
  • Dense: Distilled from DenseOn using MS MARCO 500K Triplets.
  • Sparse: Distilled from DenseOn using MS MARCO 500K Triplets.
  • ColBERT: Distilled using LightOn AI's MS MARCO BGE-Gemma dataset.

Base Model Pretraining

The foundation of the Atto family is masked language modeling on English Wikipedia.

During MLM training, portions of the input text are masked, and the model learns to reconstruct missing tokens from their surrounding context.

This provides the base models with general contextual language representations, including vocabulary, linguistic structure, and relationships between words.


Dense and Sparse Distillation

Both dense and sparse Atto models are distilled from DenseOn using the MS MARCO 500K Triplets dataset.

The dataset provides query, positive passage, and negative passage examples for retrieval supervision.

Knowledge distillation allows the smaller Atto models to learn retrieval-oriented representations from the larger teacher model.

Using the same teacher and dataset for both retrieval families provides a common foundation while allowing different representation strategies to be explored.


ColBERT Distillation

The ColBERT family follows a separate training pipeline using the LightOn AI MS MARCO BGE-Gemma dataset.

This provides retrieval supervision for training compact token-level late-interaction models.

Unlike dense retrieval, which produces a single vector for an entire query or document, ColBERT preserves token-level representations for more granular matching.


The 840-Parameter Model: How Small Can Retrieval Become?

One of the most unusual configurations in the Atto family is colbert-atto-840-8d.

With just 840 parameters, it achieves an average retrieval score of 0.3139 across ArguAna, NFCorpus, and SciFact.

For comparison:

Model Parameters Average
BERT Hash Femto 243K 0.2065
BERT Hash Pico 442K 0.2947
Atto ColBERT 840 0.3139

The result is striking because the Atto configuration outperforms both baselines despite having dramatically fewer parameters.

However, these models use different retrieval architectures, so this is primarily an illustration of parameter efficiency rather than a direct architectural comparison.

The 840-parameter model is a ColBERT-style late-interaction model, not a conventional dense embedding model. It retains token-level representations rather than compressing an entire document into one vector.

Consequently, parameter count alone does not describe the total retrieval-system cost. Document representation storage, indexing, and late-interaction scoring also matter.

The result should not be interpreted as suggesting that an 840-parameter model can replace larger general-purpose embedding models.

Instead, it illustrates how far parameter efficiency can be explored when architecture, representation dimensionality, and retrieval strategy are considered together.


Making Domain-Specific Embeddings Practical

One of the primary motivations behind Atto is making specialized embedding models more accessible.

Many applications do not need a massive general-purpose embedding model. They need a model that understands a particular collection of documents, terminology, vocabulary, and retrieval requirements.

Consider several examples:

  • Healthcare: Medical literature, clinical documentation, and specialized terminology.
  • Legal: Case law, contracts, legislation, and regulatory documents.
  • Scientific research: Research papers and specialized technical terminology.
  • Enterprise knowledge: Internal documentation, technical manuals, and organizational knowledge.
  • Software engineering: Source code documentation, API references, and technical repositories.

In these environments, a model trained or adapted for a specific retrieval task could potentially provide better alignment with the target data than a general-purpose model.

Atto explores whether compact architectures can make this kind of specialization more practical.


Multiple Models for Multiple Domains

Instead of maintaining one large embedding model for every application, organizations could potentially maintain several smaller models, each optimized for a particular retrieval problem.

For example, an organization might deploy separate models for legal research, internal documentation, scientific publications, and software repositories.

This creates flexibility when different datasets have different vocabularies, terminology, and retrieval requirements.

It also creates opportunities to experiment with domain-specific training and fine-tuning without necessarily requiring the infrastructure associated with much larger architectures.


Model Availability and Usage

Atto models are available through the Hugging Face Hub.

Explore the model family:

The family offers several retrieval configurations:

  • Dense: Single-vector semantic retrieval using conventional vector search.
  • Sparse: Sparse latent-space retrieval using learned contextual features.
  • ColBERT: Token-level late-interaction retrieval for granular query-document matching.

When selecting a checkpoint, consider parameter count, embedding dimensionality, vocabulary size, pooling strategy, retrieval quality, and indexing requirements.

Consult the individual model cards for configuration details, loading instructions, and usage examples.


Conclusion

Atto explores how far parameter efficiency can be pushed in contextual embedding models.

By combining convolution-conditioned attention, multi-scale gated processing, and learned dimensional expansion, it investigates a range of retrieval architectures spanning dense, sparse, and token-level representations.

The benchmark results demonstrate promising retrieval capabilities across a wide range of model sizes, from an 840-parameter ColBERT configuration to 28-million-parameter dense embeddings.

However, the broader motivation behind Atto is not simply minimizing parameter counts.

It is making specialized models more practical to develop and deploy.

Smaller architectures create opportunities for organizations to experiment with embedding models tailored to their own datasets, terminology, and retrieval requirements. Rather than relying exclusively on large general-purpose models, different applications could potentially use specialized models optimized for their particular domains.

There is also an opportunity to explore compact pretrained architectures for specialized decision models, extending Atto beyond semantic retrieval.

Ultimately, the interesting question is not just how small an embedding model can become, but how much useful capability can be delivered within a given computational budget.

Atto is not simply about smaller embedding models. It is about making specialized models practical.


Atto: compact contextual processing, expanded representations, and a more accessible path to specialized models.

Community

Sign up or log in to comment