OmarSY11/xlmr-intent-v2

Ferman intent classifier (XLM-R base, v2)

Classifies assistant commands into 34 intents in English, Arabic, Sorani Kurdish and Badini Kurdish. Fine-tuned from xlm-roberta-base. Part of the Ferman project: https://github.com/omarGH99/ferman-ai-assistant

Usage

from transformers import pipeline
clf = pipeline("text-classification", model="OmarSY11/xlmr-intent-v2", top_k=3)
print(clf("set an alarm for six thirty am"))
print(clf("ئەلارمێکی نوێ بۆ شەشی بەیانی"))      # Sorani

Training data

The v2 configuration of OmarSY11/ferman-nlu, train split. English and Arabic derive from MASSIVE; Sorani and Badini are machine-translated (see the dataset card for provenance and limits).

Training

Base model	xlm-roberta-base
Max length	64 tokens
Learning rate / batch size	2e-5 / 32
Epochs	up to 10, early stopping (patience 2) on validation macro-F1
Warm-up / weight decay	0.1 / 0.01
Loss	cross-entropy, label smoothing 0.1, inverse-frequency class weights (power 0.5)
Augmentation	random word deletion or adjacent swap, 3 extra copies per Sorani and Badini training row
Seed	42

The notebook is in the repo: ml/train_xlmr_intent_v2_runpod.ipynb.

Results

Held-out test split of the v2 data (4,648 rows).

Language	Test rows	Accuracy	Macro-F1
English	1,954	0.887	0.847
Arabic	1,954	0.823	0.761
Badini	370	0.719	0.682
Sorani	370	0.697	0.647
Overall	4,648	0.832	0.790

Kurdish test sets are small, so those figures carry wide uncertainty. The weakest intent is general_quirky, a catch-all class (F1 0.62 overall, 0.00 in Sorani). Per-intent results and the confusion matrix are in the repo under ml/results/.

Limitations

• Kurdish training and test data are machine-translated and informally reviewed by one person (native Badini, non-native Sorani).
• Sorani is the weakest language and is the one v2 did not improve over v1.
• 34 intents only. Out-of-domain input is rejected in the app by confidence and margin thresholds, not by the model itself.
• Not evaluated on real user speech or on other Kurdish varieties.

License and citation

Model weights: MIT, like the base model. Training data: CC BY 4.0, derived from MASSIVE (FitzGerald et al., 2022, arXiv:2204.08582), which must be credited. Please also cite https://github.com/omarGH99/ferman-ai-assistant.