ColBERT Atto

atto

Atto is a family of compact contextual models designed for parameter-efficient language representation learning.

This is a model fined-tuned using PyLate

Usage (txtai)

This model can be used to build embeddings databases with txtai for semantic search and/or as a knowledge source for retrieval augmented generation (RAG).

Note: txtai 9.0+ is required for late interaction model support

import txtai

model = "neuml/colbert-atto-840-8d"

embeddings = txtai.Embeddings(
  path=model,
  content=True
)
embeddings.index(documents())

# Run a query
embeddings.search("query to run")

Late interaction models excel as reranker pipelines.

from txtai.pipeline import Reranker, Similarity

similarity = Similarity(path=model, lateencode=True)
ranker = Reranker(embeddings, similarity)
ranker("query to run")

Usage with Sentence Transformers

As of Sentence Transformers v6.0.0, this model loads directly as a multi-vector (ColBERT-style late interaction) retriever via the MultiVectorEncoder:

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder(model)

query = "What is the capital of France?"
documents = [
    "Paris is the capital and largest city of France.",
    "Berlin is the capital of Germany.",
]

query_embeddings = model.encode_query(query)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings[0].shape)

# MaxSim late-interaction scoring (higher is more relevant)
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)

Evaluation

The following table shows a subset of BEIR scored with the txtai benchmarks script. Scores reported are ndcg@10.

Model Parameters Dimensions Vocab ArguAna NFCorpus SciFact Average
colbert-atto-840-8d 840 8 250 0.2143 0.2585 0.4688 0.3139
colbert-atto-9k-16d 9K 16 1K 0.2459 0.2902 0.5604 0.3655
colbert-atto-90k-32d 90K 32 4K 0.3435 0.3263 0.6663 0.4454
colbert-atto-950k-128d 950K 128 4K 0.4149 0.3352 0.6946 0.4816
colbert-atto-4m-384d 4M 384 30K 0.4157 0.3349 0.6949 0.4818
colbert-atto-14m-384d 14M 384 30K 0.4248 0.3483 0.7057 0.4929
colbert-atto-25m-384d 25M 384 30K 0.4395 0.3461 0.7130 0.4995
LateOn 110M 768 50K 0.4979 0.3802 0.7609 0.5463

Training Parameters

Parameter Value
per_device_train_batch_size 32
num_train_epochs 1
learning_rate 0.01
warmup_steps 0.05
fp16 True
eval_strategy steps
dataloader_num_workers 20

Full Model Architecture

ColBERT(
  (0): Transformer({'max_seq_length': 449, 'do_lower_case': False, 'architecture': 'AttoModel'})
)

More Information

See the Introducing Atto article for benchmark results, architecture details, and training methodology.

Downloads last month
-
Safetensors
Model size
840 params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for NeuML/colbert-atto-840-8d

Finetuned
(1)
this model

Collections including NeuML/colbert-atto-840-8d

Article mentioning NeuML/colbert-atto-840-8d