| --- |
| language: |
| - it |
| library_name: transformers |
| tags: |
| - tokenizer |
| - italian |
| - bpe |
| - bert |
| - alberturin |
| --- |
| |
| # gettone |
|
|
| **gettone** is a Byte-Pair Encoding (BPE) tokenizer developed specifically |
| for Italian and used across the **AlBERTurin** family of encoder models. |
|
|
| It has a vocabulary size of **32,768 tokens** and was designed to provide |
| compact tokenization of Italian text while maintaining a moderate vocabulary |
| size. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("AlBERTurin/gettone") |
| |
| text = "Questa è una frase di esempio in italiano." |
| |
| tokens = tokenizer.tokenize(text) |
| encoded = tokenizer(text) |
| |
| print(tokens) |
| print(encoded) |
| ``` |
|
|
| ## AlBERTurin Model Family |
|
|
| gettone is shared by all models in the AlBERTurin family: |
|
|
| | Model | Parameters | Training Tokens | |
| | --- | ---: | ---: | |
| | [AlBERTmini](https://huggingface.co/AlBERTurin/AlBERTmini) | 95M | 7B | |
| | [AlBERTina](https://huggingface.co/AlBERTurin/AlBERTina) | 140M | 14B | |
| | [AlBERTone101](https://huggingface.co/AlBERTurin/AlBERTone101) | 450M | ~101B | |
|
|
| ## Tokenizer Evaluation |
|
|
| The tokenizer was evaluated using **fertility**, defined as the average |
| number of subword tokens produced per word. Lower values indicate more |
| compact tokenization. |
|
|
| On the evaluation corpus used in the AlBERTurin paper, gettone achieved |
| a fertility score of **1.429**, the lowest among the evaluated Italian and |
| multilingual tokenizers. |
|
|
| | Tokenizer | Type | Vocabulary Size | Fertility ↓ | |
| | --- | --- | ---: | ---: | |
| | **gettone** | BPE | 32,768 | **1.429** | |
| | Velvet-2B | BPE | 126,976 | 1.471 | |
| | UmBERTo | BPE | 32,005 | 1.493 | |
| | dbmdz | WordPiece | 31,102 | 1.497 | |
| | Minerva-7B | BPE | 51,200 | 1.560 | |
| | Minerva-350M | BPE | 32,768 | 1.644 | |
| | XLM-R | Unigram | 250,002 | 1.663 | |
| | mBERT | WordPiece | 119,547 | 1.719 | |
| | EuroBERT | BPE | 128,000 | 1.934 | |
|
|
|
|
| ## Citation |
|
|
| If you use gettone in your research, please cite: |
|
|
| > Matteo Rinaldi, Marco Madeddu, Calogero Jerik Scozzaro, |
| > Matteo Delsanto, Daniele Paolo Radicioni, and Viviana Patti. |
| > **AlBERTurin: A Fully Open Family of Italian Encoder Models with |
| > Modern Architectures.** |
| > CLiC-it 2026. |
