setswana-sentiment-afriberta-large

AfriBERTa-large fine-tuned for 3-class sentiment classification of Setswana (Tswana) tweets: Positive / Negative / Neutral.

Trained on the DSFSI Setswana Sentiment corpus — 3,555 Setswana Twitter posts (2021–2022) annotated by three native speakers (arXiv:2605.27239).

Results (held-out test split, 346 tweets, seed 42)

Metric Value
Accuracy 0.552
Macro-F1 0.504

Sibling models fine-tuned on the same data with the same recipe for comparison:

Model Test accuracy Test macro-F1
Ido-shraga/setswana-sentiment-mbert 0.567 0.524
this model (afriberta-large) 0.552 0.504
Ido-shraga/setswana-sentiment-afroxlmr-large 0.532 0.405
Ido-shraga/setswana-sentiment-afroxlmr-base 0.500 0.486

Note: the DSFSI paper's own AfriBERTa baseline reaches 49.0±2.1 macro-F1 (5-seed mean) — this single-seed model is in the same range.

Training details

  • Base model: castorini/afriberta_large
  • Data: dsfsi/setswana-sentiment text → consensus_label (train 2,762 / validation 346 / test 346)
  • 5 epochs, batch 16, lr 2e-5 (AdamW, linear decay, 10% warmup), max length 128, bf16, seed 42
  • Best checkpoint selected by validation macro-F1; test split evaluated once at the end
  • transformers 5.23.0 / torch 2.15.0

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="Ido-shraga/setswana-sentiment-afriberta-large")
clf("setswana text here")
# [{'label': 'Positive', 'score': 0.98}]

Limitations

  • Trained and evaluated on Twitter (X) posts from 2021–2022; may not transfer to other registers.
  • Positive sentiment is under-represented in the corpus (~15%); the dominant confusion in the data is Negative vs Neutral, reflecting ironic phrasing common in Setswana commentary.
  • Single-seed result; expect a few points of variance across seeds.

Data license

The training data is released under the NOODL license (dsfsi/setswana-sentiment).

Citation

@misc{abdulmumin2026temporalsimultaneitypredictsannotation,
      title={Temporal Simultaneity Predicts Annotation Quality in Sentiment Corpora},
      author={Idris Abdulmumin and Mokgadi Penelope Matloga and Tadesse Destaw Belay and Botshelo Kondowe and Letlhogonolo Mohleleng and Hareaipha Nkopo Letsoalo and Shamsuddeen Hassan Muhammad and Vukosi Marivate},
      year={2026},
      eprint={2605.27239},
      archivePrefix={arXiv},
      url={https://arxiv.org/abs/2605.27239},
}
Downloads last month
33
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Ido-shraga/setswana-sentiment-afriberta-large

Finetuned
(33)
this model

Dataset used to train Ido-shraga/setswana-sentiment-afriberta-large

Paper for Ido-shraga/setswana-sentiment-afriberta-large

Evaluation results