Atto Models
Collection
Atto is a family of compact contextual models designed for parameter-efficient language representation learning. • 34 items • Updated • 2
Atto is a family of compact contextual models designed for parameter-efficient language representation learning.
These models can be loaded using Hugging Face Transformers. Since Atto uses a custom architecture, trust_remote_code=True is required.
from transformers import AutoModel
model = AutoModel.from_pretrained(
"neuml/atto-25m-384d",
trust_remote_code=True
)
| Parameter | Value |
|---|---|
| learning_rate | 0.001 |
| gradient_accumulation_steps | 1 |
| fp16 | True |
See the Introducing Atto article for benchmark results, architecture details, and training methodology.