Instructions to use Waqf-AI/arabic-berthash-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Waqf-AI/arabic-berthash-v2 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="Waqf-AI/arabic-berthash-v2", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForMaskedLM model = AutoModelForMaskedLM.from_pretrained("Waqf-AI/arabic-berthash-v2", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Download evaluation/protocol.md from Waqf-AI/arabic-berthash-v2: direct link, hf CLI and curl.
- Browser
- Download file 3.83 kB
-
https://huggingface.co/Waqf-AI/arabic-berthash-v2/resolve/main/evaluation/protocol.md
- Command line
-
hf download hf://Waqf-AI/arabic-berthash-v2/evaluation/protocol.md
-
curl -L -o protocol.md https://huggingface.co/Waqf-AI/arabic-berthash-v2/resolve/main/evaluation/protocol.md
Evaluation protocol
Luna-reviewed comparison
The table uses the same 100 examples per task and typed DSPy gpt-6-luna judgments. All scores are percentages.
| Model | MLM top-1 | MLM top-5 | NLI judge agreement | Retrieval R@1 | Retrieval R@5 |
|---|---|---|---|---|---|
| ArabicBertHash v2 | 65 | 93 | 37 | 31 | 39 |
| Arabic ModernBERT Unified | 70 | 100 | 68 | 48 | 61 |
| Arabic UltraBERT Small | 61 | 93 | 65 | 44 | 61 |
| Arabic UltraBERT Large | 66 | 96 | 74 | 44 | 57 |
| AraBERTv2 | 82 | 98 | 49 | 55 | 64 |
| AraModernBERT | 84 | 97 | 34 | 47 | 62 |
| Arabic MiniBERT | 31 | 79 | 35 | 46 | 63 |
How the tasks were measured
- MLM: one shared-tokenizer-eligible word is masked per question. Top-1 accepts the exact reference or a completion judged contextually acceptable; top-5 accepts at least one such completion. Uncertain judgments are excluded. Candidates are reviewed in their completed sentences with model identity and rank hidden. For this model, judge-acceptable-only scores were 62%/87%; adding exact-reference matches gives the reported 65%/93%.
- Classification: three-way natural language inference, using frozen CLS features and a linear probe trained for 400 updates on 4,096 pairs, with 512 development and 1,024 test pairs. The reviewed score measures agreement with Luna labels on 100 cases. It is not general classification accuracy.
- Retrieval: frozen mean-pooled features with a projection trained for 300 updates on filtered NLI-derived triplets. Evaluation uses 512 development queries, 512 test queries and a 1,024-sentence gallery; Luna reviews the retrieved top-five passages for 100 test queries. The released MLM weights do not include this evaluation projection.
Exact versus reviewed scores
On the paired 100-case subsets:
| Task | Dataset-reference score | Luna-reviewed score |
|---|---|---|
| MLM top-1 / top-5 | 12% / 22% | 65% / 93% |
| NLI | 33% accuracy | 37% judge agreement |
| Retrieval R@1 / R@5 | 33% / 45% | 31% / 39% |
Reviewed MLM includes plausible alternatives, while reviewed NLI and retrieval use the judge's labels/relevance judgments. These columns therefore measure different targets. In NLI, one judge label was uncertain and 24 confident labels disagreed with the dataset labels; automated review is not adjudicated ground truth.
On the full exact-reference diagnostic sets:
| Task | Test size | Score |
|---|---|---|
| Shared single-token whole-word MLM | 201 questions | Top-1 11.44%, top-5 23.88% |
| Frozen-feature NLI probe | 1,024 pairs | Accuracy 33.20% |
| Frozen-feature retrieval probe | 512 queries | R@1 27.15%, R@5 44.34% |
Full-set scores and 100-case subset scores have different denominators. Machine-readable results are available in evaluation/summary.json, with protocol details in evaluation/protocol.md.
Evaluation limitations
These are small, previously exposed research diagnostics with automated judgments. The judge was checked on small synthetic Arabic fixtures, not independently validated by human annotators. Most training comparisons use one seed, so small differences should not be treated as reliable general rankings.
Models share evaluation examples and probe-training budgets, but differ in pretraining data, compute and task exposure. Some comparison models already received NLI/retrieval training. Retrieval uses NLI-derived sentences rather than a broad Arabic passage-search benchmark. MLM acceptance measures contextual plausibility and may allow a different meaning from the reference.