language:
- it
library_name: transformers
tags:
- tokenizer
- italian
- bpe
- bert
- alberturin
gettone
gettone is a Byte-Pair Encoding (BPE) tokenizer developed specifically for Italian and used across the AlBERTurin family of encoder models.
It has a vocabulary size of 32,768 tokens and was designed to provide compact tokenization of Italian text while maintaining a moderate vocabulary size.
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("AlBERTurin/gettone")
text = "Questa è una frase di esempio in italiano."
tokens = tokenizer.tokenize(text)
encoded = tokenizer(text)
print(tokens)
print(encoded)
AlBERTurin Model Family
gettone is shared by all models in the AlBERTurin family:
| Model | Parameters | Training Tokens |
|---|---|---|
| AlBERTmini | 95M | 7B |
| AlBERTina | 140M | 14B |
| AlBERTone101 | 450M | ~101B |
Tokenizer Evaluation
The tokenizer was evaluated using fertility, defined as the average number of subword tokens produced per word. Lower values indicate more compact tokenization.
On the evaluation corpus used in the AlBERTurin paper, gettone achieved a fertility score of 1.429, the lowest among the evaluated Italian and multilingual tokenizers.
| Tokenizer | Type | Vocabulary Size | Fertility ↓ |
|---|---|---|---|
| gettone | BPE | 32,768 | 1.429 |
| Velvet-2B | BPE | 126,976 | 1.471 |
| UmBERTo | BPE | 32,005 | 1.493 |
| dbmdz | WordPiece | 31,102 | 1.497 |
| Minerva-7B | BPE | 51,200 | 1.560 |
| Minerva-350M | BPE | 32,768 | 1.644 |
| XLM-R | Unigram | 250,002 | 1.663 |
| mBERT | WordPiece | 119,547 | 1.719 |
| EuroBERT | BPE | 128,000 | 1.934 |
Citation
If you use gettone in your research, please cite:
Matteo Rinaldi, Marco Madeddu, Calogero Jerik Scozzaro, Matteo Delsanto, Daniele Paolo Radicioni, and Viviana Patti.
AlBERTurin: A Fully Open Family of Italian Encoder Models with Modern Architectures.
CLiC-it 2026.