--- license: other language: [en, es, fr, ar, ur] tags: [text-classification, hate-speech, multilingual, ensemble] --- # Multi-corpus hate speech ensemble Binary classifier: 1 = abusive or hateful, 0 = neither. Trained on ten public corpora in English, Spanish, French, Arabic and Roman Urdu (Davidson, HateXplain, DynaHate, ETHOS, HatEval EN/ES, RUHSOLD, MLMA EN/FR/AR). The ensemble weights and threshold were chosen with HABC on validation data (minimax objective). ## Test results (5,874 texts, never used for any choice) accuracy 0.7673 | macro-F1 0.7386 | worst-corpus macro-F1 0.5312 | ROC-AUC 0.8264 | corpus | n | accuracy | macro-F1 | |---|---|---|---| | davidson2017 | 896 | 0.9118 | 0.8486 | | dynabench2021 | 900 | 0.6044 | 0.6042 | | ethos2022 | 150 | 0.6667 | 0.6563 | | hateval2019_en | 150 | 0.6867 | 0.6791 | | hateval2019_es | 75 | 0.6267 | 0.5879 | | hatexplain2021 | 899 | 0.6841 | 0.6714 | | mlma2019_ar | 492 | 0.7764 | 0.7180 | | mlma2019_en | 819 | 0.8449 | 0.5312 | | mlma2019_fr | 593 | 0.7707 | 0.6035 | | romanurdu2020 | 900 | 0.8333 | 0.8333 | ## Use ```python from huggingface_hub import snapshot_download import sys path = snapshot_download("Akash212/multicorpus-hate-speech-ensemble", token="") sys.path.insert(0, path) from predict import HateEnsemble model = HateEnsemble(path) model.predict(["text one", "text two"]) ``` ## Limitations Research model. Labels from different corpora follow different definitions, and hate and offensive language are merged. Scores drop on data unlike the training corpora. Use it to help human moderators, not to remove content automatically. Some training corpora allow research use only.