Atto Models
Collection
Atto is a family of compact contextual models designed for parameter-efficient language representation learning. • 34 items • Updated • 2
How to use NeuML/atto-1m-4096d-embeddings with sentence-transformers:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("NeuML/atto-1m-4096d-embeddings", trust_remote_code=True)
sentences = [
"That is a happy person",
"That is a happy dog",
"That is a very happy person",
"Today is a sunny day"
]
embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [4, 4]How to use NeuML/atto-1m-4096d-embeddings with Transformers:
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("NeuML/atto-1m-4096d-embeddings", trust_remote_code=True, device_map="auto")Atto is a family of compact contextual models designed for parameter-efficient language representation learning.
This is a model fined-tuned using sentence-transformers.
This model can be used to build embeddings databases with txtai for semantic search and/or as a knowledge source for retrieval augmented generation (RAG).
import txtai
model = "neuml/atto-1m-4096d-embeddings"
embeddings = txtai.Embeddings(
path=model,
content=True,
vectors={"trust_remote_code": True}
)
embeddings.index(documents())
# Run a query
embeddings.search("query to run")
Alternatively, the model can be loaded with sentence-transformers.
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]
model = SentenceTransformer(model, trust_remote_code=True)
embeddings = model.encode(sentences)
print(embeddings)
The following table shows a subset of BEIR scored with the txtai benchmarks script. Scores reported are ndcg@10.
| Model | Parameters | Dimensions | Vocab | ArguAna | NFCorpus | SciFact | Average |
|---|---|---|---|---|---|---|---|
| atto-36k-384d | 36K | 384 | 1K | 0.3249 | 0.2286 | 0.4920 | 0.3485 |
| atto-50k-768d | 50K | 768 | 1K | 0.3264 | 0.2430 | 0.5334 | 0.3676 |
| atto-90k-2048d | 90K | 2048 | 1K | 0.4021 | 0.2676 | 0.6074 | 0.4257 |
| bert-hash-femto | 243K | 128 | 30K | 0.1964 | 0.1402 | 0.2830 | 0.2065 |
| bert-hash-pico | 442K | 128 | 30K | 0.2853 | 0.2075 | 0.3912 | 0.2947 |
| bert-hash-nano | 953K | 128 | 30K | 0.3828 | 0.2562 | 0.5032 | 0.3807 |
| atto-1m-2048d | 1M | 2048 | 4K | 0.4189 | 0.2941 | 0.6229 | 0.4453 |
| atto-1m-4096d | 1M | 4096 | 4K | 0.4223 | 0.3044 | 0.6417 | 0.4561 |
| atto-2m-8192d | 2M | 8192 | 4K | 0.4343 | 0.3125 | 0.6653 | 0.4707 |
| atto-14m-384d | 14M | 384 | 30K | 0.4874 | 0.3329 | 0.6472 | 0.4892 |
| atto-17m-8192d | 17M | 8192 | 30K | 0.4963 | 0.3455 | 0.6909 | 0.5109 |
| all-MiniLM-L6-v2 | 23M | 384 | 30K | 0.4789 | 0.3089 | 0.6527 | 0.4802 |
| atto-25m-384d | 25M | 384 | 30K | 0.4879 | 0.3426 | 0.6600 | 0.4968 |
| atto-28m-8192d | 28M | 8192 | 30K | 0.5111 | 0.3461 | 0.6843 | 0.5138 |
| DenseOn | 110M | 768 | 50K | 0.5408 | 0.3915 | 0.7591 | 0.5638 |
| Parameter | Value |
|---|---|
| per_device_train_batch_size | 128 |
| num_train_epochs | 25 |
| learning_rate | 0.003 |
| warmup_steps | 1000 |
| weight_decay | 0.01 |
| fp16 | True |
| per_device_eval_batch_size | 128 |
| dataloader_num_workers | 20 |
| prompts | {'query': 'query: ', 'document': 'document: '} |
SentenceTransformer(
(0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'AttoModel'})
(1): Pooling({'embedding_dimension': 4096, 'pooling_mode': 'max', 'include_prompt': True})
)
See the Introducing Atto article for benchmark results, architecture details, and training methodology.
Base model
NeuML/atto-1m-4096d