setswana-sentiment-afroxlmr-base

AfroXLMR-base fine-tuned for 3-class sentiment classification of Setswana (Tswana) tweets: Positive / Negative / Neutral, using the training recipe published by the dataset's authors.

Trained on the DSFSI Setswana Sentiment corpus — 3,555 Setswana Twitter posts (2021–2022) annotated by three native speakers (arXiv:2605.27239).

Results (held-out test split, 346 tweets, seed 42)

Metric Value
Accuracy 0.500
Macro-F1 0.486

Sibling models fine-tuned on the same data for comparison:

Model Test accuracy Test macro-F1
Ido-shraga/setswana-sentiment-mbert 0.567 0.524
Ido-shraga/setswana-sentiment-afriberta-large 0.552 0.504
Ido-shraga/setswana-sentiment-afroxlmr-large 0.532 0.405
this model (afro-xlmr-base) 0.500 0.486

vs the paper's number: the DSFSI paper reports 53.6±1.2 macro-F1 (mean over 5 seeds) for afro-xlmr-base on this dataset — our single-seed run lands ~5 points below. Unlike the paper's protocol we did not average or tune over seeds; treat the gap as a reproduction finding, not a contradiction.

Training details

  • Base model: Davlan/afro-xlmr-base
  • Data: dsfsi/setswana-sentiment text → consensus_label (train 2,762 / validation 346 / test 346)
  • Recipe from the DSFSI paper (Appendix E): lr 5e-5, batch 32, up to 20 epochs with early stopping (patience 3 on dev macro-F1), max length 128, AdamW, linear decay, 10% warmup, seed 42
  • Early stopping fired at epoch 10; best checkpoint by validation macro-F1; test split evaluated once at the end
  • transformers 5.23.0 / torch 2.15.0

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="Ido-shraga/setswana-sentiment-afroxlmr-base")
clf("setswana text here")
# [{'label': 'Negative', 'score': 0.87}]

Limitations

  • Trained and evaluated on Twitter (X) posts from 2021–2022; may not transfer to other registers.
  • Positive sentiment is under-represented in the corpus (~15%); the dominant confusion in the data is Negative vs Neutral, reflecting ironic phrasing common in Setswana commentary.
  • Single-seed result; the paper's own protocol shows ±1.2 macro-F1 seed variance on this task.

Data license

The training data is released under the NOODL license (dsfsi/setswana-sentiment).

Citation

@misc{abdulmumin2026temporalsimultaneitypredictsannotation,
      title={Temporal Simultaneity Predicts Annotation Quality in Sentiment Corpora},
      author={Idris Abdulmumin and Mokgadi Penelope Matloga and Tadesse Destaw Belay and Botshelo Kondowe and Letlhogonolo Mohleleng and Hareaipha Nkopo Letsoalo and Shamsuddeen Hassan Muhammad and Vukosi Marivate},
      year={2026},
      eprint={2605.27239},
      archivePrefix={arXiv},
      url={https://arxiv.org/abs/2605.27239},
}
Downloads last month
22
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Ido-shraga/setswana-sentiment-afroxlmr-base

Finetuned
(109)
this model

Dataset used to train Ido-shraga/setswana-sentiment-afroxlmr-base

Paper for Ido-shraga/setswana-sentiment-afroxlmr-base

Evaluation results