Instructions to use NeuML/atto-28m-8192d-sparse-embeddings with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use NeuML/atto-28m-8192d-sparse-embeddings with sentence-transformers:
from sentence_transformers import SparseEncoder model = SparseEncoder("NeuML/atto-28m-8192d-sparse-embeddings", trust_remote_code=True) queries = ["Which planet is known as the Red Planet?"] documents = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) - Transformers
How to use NeuML/atto-28m-8192d-sparse-embeddings with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("NeuML/atto-28m-8192d-sparse-embeddings", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Atto Sparse Embeddings
Atto is a family of compact contextual models designed for parameter-efficient language representation learning.
This is a model fined-tuned using sentence-transformers.
Usage (txtai)
This model can be used to build embeddings databases with txtai for semantic search and/or as a knowledge source for retrieval augmented generation (RAG).
Note: txtai 9.0+ is required for sparse vector scoring support
import txtai
model = "atto-28m-8192d-sparse-embeddings"
embeddings = txtai.Embeddings(
sparse=model,
content=True
)
embeddings.index(documents())
# Run a query
embeddings.search("query to run")
Usage (Sentence-Transformers)
Alternatively, the model can be loaded with sentence-transformers.
from sentence_transformers import SparseEncoder
sentences = ["This is an example sentence", "Each sentence is converted"]
model = SparseEncoder(model)
embeddings = model.encode(sentences)
print(embeddings)
Evaluation
The following table shows a subset of BEIR scored with the txtai benchmarks script. Scores reported are ndcg@10.
| Model | Parameters | Dimensions | Vocab | ArguAna | NFCorpus | SciFact | Average |
|---|---|---|---|---|---|---|---|
| atto-17m-8192d-sparse | 17M | 8192 | 30K | 0.3981 | 0.3208 | 0.6600 | 0.4596 |
| atto-28m-8192d-sparse | 28M | 8192 | 30K | 0.3651 | 0.3292 | 0.6618 | 0.4520 |
| SlideOn | 174M | 768 | 50K | 0.4655 | 0.3863 | 0.7529 | 0.5349 |
Training Parameters
| Parameter | Value |
|---|---|
| per_device_train_batch_size | 64 |
| num_train_epochs | 25 |
| learning_rate | 0.0003 |
| warmup_steps | 1000 |
| weight_decay | 0.01 |
| gradient_accumulation_steps | 2 |
| fp16 | True |
| per_device_eval_batch_size | 64 |
| dataloader_num_workers | 20 |
| prompts | {'query': 'query: ', 'document': 'document: '} |
Full Model Architecture
SparseEncoder(
(0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'AttoModel'})
(1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'embedding_dimension': 8192})
)
More Information
See the Introducing Atto article for benchmark results, architecture details, and training methodology.
- Downloads last month
- -
Model tree for NeuML/atto-28m-8192d-sparse-embeddings
Base model
NeuML/atto-28m-8192d