Atto Embeddings

atto

Atto is a family of compact contextual models designed for parameter-efficient language representation learning.

This is a model fined-tuned using sentence-transformers.

Usage (txtai)

This model can be used to build embeddings databases with txtai for semantic search and/or as a knowledge source for retrieval augmented generation (RAG).

import txtai

model = "neuml/atto-1m-4096d-embeddings"

embeddings = txtai.Embeddings(
  path=model,
  content=True,
  vectors={"trust_remote_code": True}
)
embeddings.index(documents())

# Run a query
embeddings.search("query to run")

Usage (Sentence-Transformers)

Alternatively, the model can be loaded with sentence-transformers.

from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer(model, trust_remote_code=True)
embeddings = model.encode(sentences)
print(embeddings)

Evaluation

The following table shows a subset of BEIR scored with the txtai benchmarks script. Scores reported are ndcg@10.

Model Parameters Dimensions Vocab ArguAna NFCorpus SciFact Average
atto-36k-384d 36K 384 1K 0.3249 0.2286 0.4920 0.3485
atto-50k-768d 50K 768 1K 0.3264 0.2430 0.5334 0.3676
atto-90k-2048d 90K 2048 1K 0.4021 0.2676 0.6074 0.4257
bert-hash-femto 243K 128 30K 0.1964 0.1402 0.2830 0.2065
bert-hash-pico 442K 128 30K 0.2853 0.2075 0.3912 0.2947
bert-hash-nano 953K 128 30K 0.3828 0.2562 0.5032 0.3807
atto-1m-2048d 1M 2048 4K 0.4189 0.2941 0.6229 0.4453
atto-1m-4096d 1M 4096 4K 0.4223 0.3044 0.6417 0.4561
atto-2m-8192d 2M 8192 4K 0.4343 0.3125 0.6653 0.4707
atto-14m-384d 14M 384 30K 0.4874 0.3329 0.6472 0.4892
atto-17m-8192d 17M 8192 30K 0.4963 0.3455 0.6909 0.5109
all-MiniLM-L6-v2 23M 384 30K 0.4789 0.3089 0.6527 0.4802
atto-25m-384d 25M 384 30K 0.4879 0.3426 0.6600 0.4968
atto-28m-8192d 28M 8192 30K 0.5111 0.3461 0.6843 0.5138
DenseOn 110M 768 50K 0.5408 0.3915 0.7591 0.5638

Training Parameters

Parameter Value
per_device_train_batch_size 128
num_train_epochs 25
learning_rate 0.003
warmup_steps 1000
weight_decay 0.01
fp16 True
per_device_eval_batch_size 128
dataloader_num_workers 20
prompts {'query': 'query: ', 'document': 'document: '}

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'AttoModel'})
  (1): Pooling({'embedding_dimension': 4096, 'pooling_mode': 'max', 'include_prompt': True})
)

More Information

See the Introducing Atto article for benchmark results, architecture details, and training methodology.

Downloads last month
-
Safetensors
Model size
1.59M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for NeuML/atto-1m-4096d-embeddings

Finetuned
(1)
this model

Collection including NeuML/atto-1m-4096d-embeddings

Article mentioning NeuML/atto-1m-4096d-embeddings