setswana-sentiment-afroxlmr-large

AfroXLMR-large fine-tuned for 3-class sentiment classification of Setswana (Tswana) tweets: Positive / Negative / Neutral.

Trained on the DSFSI Setswana Sentiment corpus — 3,555 Setswana Twitter posts (2021–2022) annotated by three native speakers (arXiv:2605.27239).

Results (held-out test split, 346 tweets, seed 42)

Metric Value
Accuracy 0.532
Macro-F1 0.405

Sibling models fine-tuned on the same data for comparison — this is the weakest of the family on macro-F1:

Model Test accuracy Test macro-F1
Ido-shraga/setswana-sentiment-mbert 0.567 0.524
Ido-shraga/setswana-sentiment-afriberta-large 0.552 0.504
this model (afro-xlmr-large) 0.532 0.405
Ido-shraga/setswana-sentiment-afroxlmr-base 0.500 0.486

⚠️ Training instability — read before use

Two findings worth knowing:

  1. The base checkpoint is stored in fp16. Loading Davlan/afro-xlmr-large without torch_dtype=torch.float32 leaves the weights in fp16, gradients underflow to zero, and training deterministically collapses to the majority class (test accuracy exactly 0.431, macro-F1 0.201) at every learning rate, seed, and precision setting tried. Always load with an explicit fp32 upcast.
  2. Even with the upcast, this base is degenerate-run-prone on this 2,762-example dataset. An identical rerun of the winning configuration (same seed) once reproduced 0.532/0.405 and once collapsed. This is the known large-encoder instability on small datasets (Mosbach et al., arXiv:2006.04884). The DSFSI paper itself used afro-xlmr-base (53.6 macro-F1), not large, on this dataset. If you retrain, run multiple seeds and keep the best by validation macro-F1.

This model is the best of 7 runs; its weights were preserved from the winning run.

Training details

  • Base model: Davlan/afro-xlmr-large, loaded with torch_dtype=torch.float32 (fp16→fp32 upcast)
  • Data: dsfsi/setswana-sentiment text → consensus_label (train 2,762 / validation 346 / test 346)
  • 5 epochs, batch 16, lr 2e-5 (AdamW, linear decay, 10% warmup), max length 128, bf16 autocast, seed 42
  • Best checkpoint selected by validation macro-F1; test split evaluated once at the end
  • transformers 5.23.0 / torch 2.15.0

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="Ido-shraga/setswana-sentiment-afroxlmr-large")
clf("setswana text here")
# [{'label': 'Neutral', 'score': 0.91}]

Limitations

  • Trained and evaluated on Twitter (X) posts from 2021–2022; may not transfer to other registers.
  • Positive sentiment is under-represented in the corpus (~15%); the dominant confusion in the data is Negative vs Neutral, reflecting ironic phrasing common in Setswana commentary.
  • Single-seed result from an instability-prone configuration; treat the scores with a few points of uncertainty.

Data license

The training data is released under the NOODL license (dsfsi/setswana-sentiment).

Citation

@misc{abdulmumin2026temporalsimultaneitypredictsannotation,
      title={Temporal Simultaneity Predicts Annotation Quality in Sentiment Corpora},
      author={Idris Abdulmumin and Mokgadi Penelope Matloga and Tadesse Destaw Belay and Botshelo Kondowe and Letlhogonolo Mohleleng and Hareaipha Nkopo Letsoalo and Shamsuddeen Hassan Muhammad and Vukosi Marivate},
      year={2026},
      eprint={2605.27239},
      archivePrefix={arXiv},
      url={https://arxiv.org/abs/2605.27239},
}
Downloads last month
72
Safetensors
Model size
0.6B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Ido-shraga/setswana-sentiment-afroxlmr-large

Finetuned
(25)
this model

Dataset used to train Ido-shraga/setswana-sentiment-afroxlmr-large

Papers for Ido-shraga/setswana-sentiment-afroxlmr-large

Evaluation results