File size: 2,184 Bytes
6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 6691fae 1404aa6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 | ---
language:
- it
library_name: transformers
tags:
- tokenizer
- italian
- bpe
- bert
- alberturin
---
# gettone
**gettone** is a Byte-Pair Encoding (BPE) tokenizer developed specifically
for Italian and used across the **AlBERTurin** family of encoder models.
It has a vocabulary size of **32,768 tokens** and was designed to provide
compact tokenization of Italian text while maintaining a moderate vocabulary
size.
## Usage
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("AlBERTurin/gettone")
text = "Questa è una frase di esempio in italiano."
tokens = tokenizer.tokenize(text)
encoded = tokenizer(text)
print(tokens)
print(encoded)
```
## AlBERTurin Model Family
gettone is shared by all models in the AlBERTurin family:
| Model | Parameters | Training Tokens |
| --- | ---: | ---: |
| [AlBERTmini](https://huggingface.co/AlBERTurin/AlBERTmini) | 95M | 7B |
| [AlBERTina](https://huggingface.co/AlBERTurin/AlBERTina) | 140M | 14B |
| [AlBERTone101](https://huggingface.co/AlBERTurin/AlBERTone101) | 450M | ~101B |
## Tokenizer Evaluation
The tokenizer was evaluated using **fertility**, defined as the average
number of subword tokens produced per word. Lower values indicate more
compact tokenization.
On the evaluation corpus used in the AlBERTurin paper, gettone achieved
a fertility score of **1.429**, the lowest among the evaluated Italian and
multilingual tokenizers.
| Tokenizer | Type | Vocabulary Size | Fertility ↓ |
| --- | --- | ---: | ---: |
| **gettone** | BPE | 32,768 | **1.429** |
| Velvet-2B | BPE | 126,976 | 1.471 |
| UmBERTo | BPE | 32,005 | 1.493 |
| dbmdz | WordPiece | 31,102 | 1.497 |
| Minerva-7B | BPE | 51,200 | 1.560 |
| Minerva-350M | BPE | 32,768 | 1.644 |
| XLM-R | Unigram | 250,002 | 1.663 |
| mBERT | WordPiece | 119,547 | 1.719 |
| EuroBERT | BPE | 128,000 | 1.934 |
## Citation
If you use gettone in your research, please cite:
> Matteo Rinaldi, Marco Madeddu, Calogero Jerik Scozzaro,
> Matteo Delsanto, Daniele Paolo Radicioni, and Viviana Patti.
> **AlBERTurin: A Fully Open Family of Italian Encoder Models with
> Modern Architectures.**
> CLiC-it 2026. |