halluscoring-camelbert-qa

CAMeLBERT (CAMeL-Lab/bert-base-arabic-camelbert-mix) fine-tuned on HalluScoring 2026 Task 1.1 using the starter kit's default QA framing ([CLS] question [SEP] model_answer [SEP] — gold_answer is not used). Internally this is run S01, our baseline reproduction.

This is the model that was officially scored on the hidden test set. It was submitted as the fully-compliant notebook (SUBMISSION_TEMPLATE.ipynb, unmodified [DO NOT MODIFY] cells) specifically because the official evaluation template's fixed InferenceDataset cell only tokenizes question + model_answer, which rules out the NLI framing our stronger models use. Its dev scores (AUC-ROC 0.9263, F1-Macro 0.8595) match the official CodaBench leaderboard's NAMAA row exactly. Official result: 3rd of 7 teams, Test AUC-ROC 0.7596.

It scores noticeably below our NLI-framed models on internal dev metrics (see halluscoring-arbert-nli and SYSTEM_WRITEUP.md for the full comparison) — it is kept as the officially-evaluated reference point, not as our best system.

How to Use

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "HassanB4/halluscoring-camelbert-qa"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

question = "..."
model_answer = "..."

inputs = tokenizer(question, model_answer, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
    prob_hallucinated = torch.softmax(logits, dim=-1)[0, 1].item()

print(f"hallucinated={int(prob_hallucinated > 0.5)}, score={prob_hallucinated:.4f}")

Training

Parameter	Value
Base model	CAMeL-Lab/bert-base-arabic-camelbert-mix
Input format	qa (question + model_answer)
Max sequence length	512
Batch size	16
Epochs	5
Learning rate	2e-5
Warmup ratio	0.1
Weight decay	0.01
Loss	cross-entropy
Seed	42

Evaluation

Split	AUC-ROC	F1-Macro	AUC-PR
Dev (official, n=1300)	0.9263	0.8595	0.9442
Dev (clean, unseen-question subset, n=800)	0.8713	—	—
Test (official leaderboard)	0.7596	0.6444	0.9199

Published baseline (CAMeLBERT, organizer-trained): 0.7093 Dev AUC-ROC / 0.6743 Test AUC-ROC.

Limitations

Uses the QA framing, which discards gold_answer — our own experiments show this costs ~5.6pp clean-dev AUC-ROC versus the NLI framing used by our other models (see halluscoring-camelbert-nli). Kept and submitted only because it is the framing the official submission template supports without modification.

Citation

@inproceedings{namaa2026halluscoring,
    title={{NAMAA at HalluScoring 2026: NLI-Framed BERT Classifiers and Ensembling for Model-Agnostic Arabic Hallucination Detection}},
    author={[AUTHOR NAMES TBD]},
    year={2026},
    booktitle={Proceedings of ArabicNLP 2026},
    note={HalluScoring 2026 Shared Task, Track 1}
}