Atto Sparse Embeddings

atto

Atto is a family of compact contextual models designed for parameter-efficient language representation learning.

This is a model fined-tuned using sentence-transformers.

Usage (txtai)

This model can be used to build embeddings databases with txtai for semantic search and/or as a knowledge source for retrieval augmented generation (RAG).

Note: txtai 9.0+ is required for sparse vector scoring support

import txtai

model = "atto-28m-8192d-sparse-embeddings"

embeddings = txtai.Embeddings(
  sparse=model,
  content=True
)
embeddings.index(documents())

# Run a query
embeddings.search("query to run")

Usage (Sentence-Transformers)

Alternatively, the model can be loaded with sentence-transformers.

from sentence_transformers import SparseEncoder
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SparseEncoder(model)
embeddings = model.encode(sentences)
print(embeddings)

Evaluation

The following table shows a subset of BEIR scored with the txtai benchmarks script. Scores reported are ndcg@10.

Model Parameters Dimensions Vocab ArguAna NFCorpus SciFact Average
atto-17m-8192d-sparse 17M 8192 30K 0.3981 0.3208 0.6600 0.4596
atto-28m-8192d-sparse 28M 8192 30K 0.3651 0.3292 0.6618 0.4520
SlideOn 174M 768 50K 0.4655 0.3863 0.7529 0.5349

Training Parameters

Parameter Value
per_device_train_batch_size 64
num_train_epochs 25
learning_rate 0.0003
warmup_steps 1000
weight_decay 0.01
gradient_accumulation_steps 2
fp16 True
per_device_eval_batch_size 64
dataloader_num_workers 20
prompts {'query': 'query: ', 'document': 'document: '}

Full Model Architecture

SparseEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'AttoModel'})
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'embedding_dimension': 8192})
)

More Information

See the Introducing Atto article for benchmark results, architecture details, and training methodology.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for NeuML/atto-28m-8192d-sparse-embeddings

Finetuned
(2)
this model

Collection including NeuML/atto-28m-8192d-sparse-embeddings

Article mentioning NeuML/atto-28m-8192d-sparse-embeddings