Abdelkareem's picture
Focus model card on usage, architecture, training and evaluation; remove internal release-history commentary
a22d7ac verified
|
Raw History Blame Contribute Delete
3.83 kB

Evaluation protocol

Luna-reviewed comparison

The table uses the same 100 examples per task and typed DSPy gpt-6-luna judgments. All scores are percentages.

Model MLM top-1 MLM top-5 NLI judge agreement Retrieval R@1 Retrieval R@5
ArabicBertHash v2 65 93 37 31 39
Arabic ModernBERT Unified 70 100 68 48 61
Arabic UltraBERT Small 61 93 65 44 61
Arabic UltraBERT Large 66 96 74 44 57
AraBERTv2 82 98 49 55 64
AraModernBERT 84 97 34 47 62
Arabic MiniBERT 31 79 35 46 63

How the tasks were measured

  • MLM: one shared-tokenizer-eligible word is masked per question. Top-1 accepts the exact reference or a completion judged contextually acceptable; top-5 accepts at least one such completion. Uncertain judgments are excluded. Candidates are reviewed in their completed sentences with model identity and rank hidden. For this model, judge-acceptable-only scores were 62%/87%; adding exact-reference matches gives the reported 65%/93%.
  • Classification: three-way natural language inference, using frozen CLS features and a linear probe trained for 400 updates on 4,096 pairs, with 512 development and 1,024 test pairs. The reviewed score measures agreement with Luna labels on 100 cases. It is not general classification accuracy.
  • Retrieval: frozen mean-pooled features with a projection trained for 300 updates on filtered NLI-derived triplets. Evaluation uses 512 development queries, 512 test queries and a 1,024-sentence gallery; Luna reviews the retrieved top-five passages for 100 test queries. The released MLM weights do not include this evaluation projection.

Exact versus reviewed scores

On the paired 100-case subsets:

Task Dataset-reference score Luna-reviewed score
MLM top-1 / top-5 12% / 22% 65% / 93%
NLI 33% accuracy 37% judge agreement
Retrieval R@1 / R@5 33% / 45% 31% / 39%

Reviewed MLM includes plausible alternatives, while reviewed NLI and retrieval use the judge's labels/relevance judgments. These columns therefore measure different targets. In NLI, one judge label was uncertain and 24 confident labels disagreed with the dataset labels; automated review is not adjudicated ground truth.

On the full exact-reference diagnostic sets:

Task Test size Score
Shared single-token whole-word MLM 201 questions Top-1 11.44%, top-5 23.88%
Frozen-feature NLI probe 1,024 pairs Accuracy 33.20%
Frozen-feature retrieval probe 512 queries R@1 27.15%, R@5 44.34%

Full-set scores and 100-case subset scores have different denominators. Machine-readable results are available in evaluation/summary.json, with protocol details in evaluation/protocol.md.

Evaluation limitations

These are small, previously exposed research diagnostics with automated judgments. The judge was checked on small synthetic Arabic fixtures, not independently validated by human annotators. Most training comparisons use one seed, so small differences should not be treated as reliable general rankings.

Models share evaluation examples and probe-training budgets, but differ in pretraining data, compute and task exposure. Some comparison models already received NLI/retrieval training. Retrieval uses NLI-derived sentences rather than a broad Arabic passage-search benchmark. MLM acceptance measures contextual plausibility and may allow a different meaning from the reference.