danjack0/dialettometro-product

Dialettometro: Italian dialect classifier (product config)

Sentence-level classifier for 11 Italian regional varieties, plus Standard Italian and an other class that rejects text in non-target languages (13 classes in total).

Code, data, training scripts and the full evaluation write-up: github.com/danjack0/dialettometro. Live demo (n-gram model only): dialettometro.onrender.com.

Classes

Emilian-Romagnol, Friulian, Ladin, Ligurian, Lombard, Neapolitan, Piedmontese, Sardinian, Sicilian, Tarantino, Venetian, Standard Italian, other.

What's in this repo

A 3-model stacked ensemble:

File	Model
model.safetensors, config.json, tokenizer files	XLM-RoBERTa base, fine-tuned for 13-class classification
product_ngram_boost.pt	From-scratch character + word n-gram network (PyTorch)
product_stacker.joblib	scikit-learn logistic-regression meta-model combining the two

product_stacker.joblib is a joblib (pickle) file, the standard format for scikit-learn models. Like any pickle, load it only from a source you trust.

How to use

The models are loaded by the project's own scripts, not by a plain transformers pipeline (the stacker needs both base models):

git clone https://github.com/danjack0/dialettometro.git
cd dialettometro
pip install -r requirements.txt

# full stacker (best model)
python scripts/app.py --hf-repo danjack0/dialettometro-product \
    --model product_stacker.joblib --ngram product_ngram_boost.pt \
    --xlmr danjack0/dialettometro-product --floor 0.60

# lightweight: n-gram only, no 1.1 GB transformer download
python scripts/app.py --hf-repo danjack0/dialettometro-product \
    --model product_ngram_boost.pt --floor 0.60

--floor is a confidence threshold: below it, the tool answers "not sure" instead of guessing.

Evaluation

Macro-F1 on a held-out, article-level split of the project's own Wikipedia-derived data (the product_v2 configuration, where five other languages are withheld from training entirely):

Model	Macro-F1
n-gram	0.7554
XLM-R	0.8188
Stacker	0.8811

Which models these numbers belong to. The files in this repo are the original product models, trained before a split leak in the other class was found and fixed. That leak invalidates the original models' held-out other score, not the models themselves. The numbers above come from the leak-free product_v2 retrain of the same architecture, which is the configuration to quote. Full details: README, "Fixed: the Product other class".

External benchmark. On the VarDial 2022 ITDI test set (casual register, eval only, never trained on), the separate 11-class ITDI-parity stacker (not published in this repo) scores 0.5707 weighted-F1, against 0.7726 for the strongest official baseline. See the GitHub README for the full comparison.

Limitations

• Register shift. Trained on encyclopedic Wikipedia text. Accuracy drops on casual or spoken-style text; Tarantino falls from 0.94-0.96 F1 in-domain to 0.21 on ITDI's casual text, scattering to Neapolitan and Sicilian.
• Ladin has high precision but low recall (0.24-0.47): small training set.
• Emilian-Romagnol is the smallest class (548 rows): its Wikipedia is mostly short stubs.
• The n-gram alone rejects unseen languages poorly; use the full stacker when the input may not be Italian at all.
• Sentence-level only; not intended for very short inputs (a word or two).

Training data

• Dialect and Standard Italian text: Wikipedia editions of each variety (CC-BY-SA 4.0).
• other class: Tatoeba sentences in 30 languages (CC-BY 2.0 FR).

Splits are by source article, with a row-count-aware split so every class gets close to its target test share. The VarDial 2022 ITDI data was used only for evaluation.

Citation

@misc{jackson2026dialettometro,
  title  = {Dialettometro: An Italian dialect classifier},
  author = {Jackson, Daniel},
  year   = {2026},
  url    = {https://github.com/danjack0/dialettometro}
}