File size: 2,087 Bytes
757b02a 9417b5d 757b02a 9417b5d 757b02a 9417b5d 757b02a 9417b5d 757b02a eb7ad60 64efea3 757b02a 9417b5d 757b02a 9417b5d 757b02a 9417b5d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 | ---
language:
- it
library_name: transformers
tags:
- bert
- encoder
- italian
- masked-language-modeling
- long-context
- alibi
- matformer
---
# AlBERTmini
**AlBERTmini** is a 95M-parameter Italian encoder model from the
**AlBERTurin** family.
It was trained from scratch on approximately 7B Italian tokens using
masked language modeling.
## Model Description
AlBERTmini is the smallest model in the AlBERTurin family of
encoder-only Transformer models for Italian.
The model incorporates several architectural improvements over the
original BERT architecture, including Pre-RMSNorm, SwiGLU activations,
ALiBi positional biases, and a mask-only pre-training objective.
AlBERTmini uses:
- 6 Transformer layers
- hidden size of 768
- 12 attention heads
- SwiGLU activations
- Pre-RMSNorm
- ALiBi positional biases
- 1,024-token training sequence length
- 20% mask-only MLM
- Muon optimizer
The model uses **gettone**, a 32,768-token BPE tokenizer optimized
for Italian and shared across the AlBERTurin model family.
The model was trained using
[Matformer](https://github.com/mrinaldi97/matformer).
## AlBERTurin Model Family
| Model | Parameters | Training Tokens |
| --- | ---: | ---: |
| **AlBERTmini** | 95M | 7B |
| [AlBERTina](https://huggingface.co/AlBERTurin/AlBERTina) | 140M | 14B |
| [AlBERTone101](https://huggingface.co/AlBERTurin/AlBERTone101) | 450M | ~101B |
## Installation
```bash
python -m pip install \
git+https://github.com/mrinaldi97/matformer.git@alberturin-v1
```
## Usage
```python
from transformers import AutoTokenizer, AutoModelForMaskedLM
model_id = "AlBERTurin/AlBERTmini"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(
model_id,
trust_remote_code=True,
)
```
## Citation
If you use AlBERTmini in your research, please cite:
> Matteo Rinaldi, Marco Madeddu, Calogero Jerik Scozzaro,
> Matteo Delsanto, Daniele Paolo Radicioni, and Viviana Patti.
> **AlBERTurin: A Fully Open Family of Italian Encoder Models with
> Modern Architectures.**
> CLiC-it 2026. |