Control and ablation runs for the Darija Arabizi embedding study

These are the EmbeddingGemma 2 (text part) training runs used as controls and ablations in The Arabizi Blind Spot of Text Embeddings for Moroccan Darija and a Training Fix. They are released so that the paper's comparisons can be checked and reused, not as recommended models. For actual use, take anaschahid/darija-arabizi-embeddinggemma2-text.

Each subfolder is a full checkpoint (same format and same prompts as the main model; see its card). Load one with AutoModel.from_pretrained("anaschahid/darija-arabizi-embedding-controls", subfolder="eg2_arabic_only").

What each run is

Folder Variant Medical Arabic Medical Arabizi Medical mix DODa Arz->En 50 typed Arabizi
eg2_base Base: EmbeddingGemma 2 (no fine-tuning) 20.4 2.9 17.1 8.8 6
eg2_full Full recipe, seed 1 (main model) 32.1 19.8 27.2 90.1 30
eg2_full_s2 Full recipe, seed 2 31.7 18.5 27.3 89.8 28
eg2_full_s3 Full recipe, seed 3 31.5 18.7 26.9 90.4 26
eg2_arabic_only Arabic letters only (control) 33.7 4.3 32.7 21.3 4
eg2_arabic_only_s2 Arabic letters only, seed 2 34.5 5.3 32.3 22.3 6
eg2_arabic_only_matched Arabic letters only, matched exposure (570 steps) 32.5 4.9 30.5 18.8 6
eg2_no_med_arabizi No medical Arabizi 33.5 5.2 31.1 91.4 8
eg2_no_med_arabizi_matched No medical Arabizi, matched exposure (1,140 steps) 32.8 5.5 31.3 89.3 10
eg2_single_style One fixed Arabizi spelling per word 32.1 19.0 27.4 89.8 28
eg2_no_mrl No Matryoshka loss 31.4 19.7 26.2 91.4 32
eg2_clean_generator Cleaned spelling generator 31.6 18.5 27.2 90.1 32
eg2_full_2ep Twice as many steps (3,400) 31.5 20.2 26.4 92.8 30

Recall@10 in %. Medical = 1,000 test questions against 5,000 answers (the Arabizi and mix versions are language-model text); "50 typed Arabizi" = questions typed by a native speaker. See the paper for the other test sets, intervals and discussion. Main findings these runs support:

  • Training only on Arabic letters leaves medical Arabizi unfixed (about 4 to 5%), even at matched exposure.
  • Everyday Arabizi (DODa) without medical Arabizi does not transfer to medical questions.
  • One fixed spelling per word works as well as several; cleaning the spelling generator did not help; doubling the steps did not help.
  • Dropping the Matryoshka loss leaves full-width scores unchanged but hurts short (128-number) vectors.

Licence and use

CC BY-NC 4.0 (inherited from DODa and MedQA-MA). Research use only; not medical advice.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for anaschahid/darija-arabizi-embedding-controls

Finetuned
(187)
this model

Collection including anaschahid/darija-arabizi-embedding-controls