Bilingual hate / personal-toxicity detection for university course evaluations (EN + Egyptian AR)

Binary classifier: hate = personal abuse / insult / contempt toward an individual (instructor, TA); normal = everything else — including legitimate harsh-but-fair criticism, which must not be flagged (the project's cardinal rule).

This repo hosts every checkpoint of the campaign under models/<run_id>/. Use the champion below; everything else is history.

🏆 Production champion: `cln5B + cmarnols` (probability ensemble, 0.7 / 0.3)

component	subfolder	what it is
cln5B (w=0.7)	`models/20260612_subset_cln5B`	XLM-RoBERTa-large weight soup of 5 training seeds
cmarnols (w=0.3)	`models/20260612_133541_MARBERTv2_v38dsoupnols`	MARBERTv2 weight soup of 3 seeds (Egyptian-dialect diversity partner)

No language router — the same blend scores every input.

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

REPO = "Nexus-Analytics/multilingual-hatespeech-detection-model"
PARTS = [("models/20260612_subset_cln5B", 0.7),
         ("models/20260612_133541_MARBERTv2_v38dsoupnols", 0.3)]

def load(sub):
    tok = AutoTokenizer.from_pretrained(REPO, subfolder=sub)
    m = AutoModelForSequenceClassification.from_pretrained(REPO, subfolder=sub).eval()
    hate = next(i for i, v in m.config.id2label.items() if str(v).lower() == "hate")
    return tok, m, int(hate)

members = [(load(sub), w) for sub, w in PARTS]

def p_hate(text):
    p = 0.0
    for (tok, m, hi), w in members:
        enc = tok(text, return_tensors="pt", truncation=True, max_length=256)
        with torch.no_grad():
            p += w * m(**enc).logits.softmax(-1)[0, hi].item()
    return p   # hate if >= 0.5; route p in [0.35, 0.65] to human review

Headline metrics (frozen 738-case bilingual challenge sets + held-out test split)

metric	value
challenge misses (369 EN + 369 AR hard cases)	13 (3 EN + 10 AR)
test split overall	99.36–99.42%
Arabic false-positive rate (fair criticism wrongly flagged)	1.50%
English FP	0.00%
formatting-flip rate (1,000 real comments × 7,324 surface variants)	0.3%
sarcasm recall	54/55 per language
training-label debt	none (all members trained on verified-clean labels)
seed sensitivity	none (both halves are multi-seed weight soups)

Single-model alternatives (one checkpoint, one forward pass)

need	subfolder	trade-off
best fully-clean single	`models/20260612_subset_cln5A`	17 challenge misses, 0.5% flips
best single by score	`models/20260612_subset_subD`	13 misses, but 2 of 4 soup members saw a later-corrected label batch
English-only	`Nexus-Analytics/english-hatespeech-detection-model` → `models/20260602_151037_roberta`	never feed it Arabic (~50% there)

How it was built (short version)

~26k-row bilingual dataset; the key labeling decision: the Arabic survey's offensive class (harsh criticism) maps to normal, with only ~32 genuinely contemptuous rows excepted.
+1,050 verified hard-negatives teaching criticism→normal, sarcastic-praise→hate, hedged-contempt→hate.
Seed variance discovery: identical recipes swing ±10 challenge misses across random seeds → single runs are never compared. Each recipe trains 6 seeds (3 seeds × ±label smoothing); all 57 seed-subsets are searched in probability space, ranked by validation only, and the winners are built as parameter-averaged weight soups.
Architecture diversity (XLM-R + MARBERT) rescues knife-edge cases neither family fixes alone (19 → 13 misses).

Intended use & limitations

Built for Egyptian university course-evaluation comments (English, Egyptian Arabic, and their code-switching). Out of that domain, re-validate before trusting it.
Nearly all Arabic training data is synthetic (Gemini-generated, human-audited); the remaining known weakness is Arabic "write-off" hate (total dismissals without insult words): 9–11/13 recall on the hardest curated cases.
Recommended serving: abstain band p ∈ [0.35, 0.65] → human review; strip one trailing period before scoring (measured robustness guardrail).
This is a moderation aid, not a disciplinary instrument: scores are evidence for a human process, never an automatic verdict on a student or instructor.