README.md

---
base_model: google/gemma-4-E4B-it
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:google/gemma-4-E4B-it
- lora
- arabic
- diacritization
- tashkeel
language:
- ar
datasets:
- Misraj/Sadeed_Tashkeela
- Misraj/SadeedDiac-25
license: apache-2.0
---

# Gemma 4 E4B-it + LoRA for Arabic Diacritisation

LoRA adapters that restore Arabic diacritics (tashkeel) on top of
[`google/gemma-4-E4B-it`](https://huggingface.co/google/gemma-4-E4B-it). Three adapters are
included, trained on 10%, 30% and 50% of the corpus, so the effect of training
data volume can be read directly.

Part of **DiacriticS**, a contamination-controlled study of open-weights models on
this task — [project site](https://diacritics.vercel.app/) · [code](https://github.com/Psycodem/DiacriticS-Language-Model-for-Arabic-Diacritisation).

## Which subfolder to use

Each fraction is a subfolder of this repo. `50pct` is the strongest.

| Subfolder | Training rows | DER (CE) | DER (no CE) | WER (CE) | WER (no CE) |
|---|---:|---:|---:|---:|---:|
| `10pct` | 104,270 | **3.15** | 2.68 | 7.28 | 4.47 |
| `30pct` | 312,809 | **2.98** | 2.54 | 6.82 | 4.12 |
| `50pct` | 521,349 | **2.81** | 2.38 | 6.54 | 3.96 |

Scored on the full 1,200-paragraph **SadeedDiac-25** benchmark with
[`Evaluation_Functions_Corrected.py`](https://github.com/Psycodem/DiacriticS-Language-Model-for-Arabic-Diacritisation/blob/main/Evaluation_Functions_Corrected.py).
Percentages; lower is better. CE = sentence-final case ending (*i'rab*).

<details>
<summary>All three splits</summary>

DER / WER with case endings.

| Subfolder | Train sample (500) | Tashkeela test (2,485) | SadeedDiac-25 (1,200) |
|---|---|---|---|
| `10pct` | 1.23 / 3.24 | 12.96 / 16.2 | 3.15 / 7.28 |
| `30pct` | 1.81 / 3.54 | 12.73 / 15.73 | 2.98 / 6.82 |
| `50pct` | 1.76 / 3.19 | 12.57 / 15.41 | 2.81 / 6.54 |

The Tashkeela test split scores worse than the benchmark because it inherits the
residual annotation noise of the source corpus, not because the model does worse
on it. SadeedDiac-25 is expert-reviewed and balanced 50/50 between Modern
Standard and Classical Arabic.

</details>

## Usage

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

BASE     = "google/gemma-4-E4B-it"
ADAPTER  = "Psycodem/gemma-4-e4b-lora-diacritization"
FRACTION = "50pct"          # "10pct" | "30pct" | "50pct"

tok = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

model = AutoModelForCausalLM.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True)
model = PeftModel.from_pretrained(model, ADAPTER, subfolder=FRACTION)
model = model.merge_and_unload()
model.eval()

SYSTEM_PROMPT = (
    "أنت نظام متخصص في التشكيل الآلي للنصوص العربية. "
    "مهمتك إضافة الحركات (التشكيل) الصحيحة إلى النص العربي المُدخل دون تغيير الكلمات أو ترتيبها، "
    "مع مراعاة السياق النحوي والصرفي الكامل للجملة."
)

def diacritize(text):
    msgs = [{"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text}]
    try:
        prompt = tok.apply_chat_template(msgs, tokenize=False,
                                         add_generation_prompt=True, enable_thinking=False)
    except TypeError:
        prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
    enc = tok(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**enc, max_new_tokens=512, do_sample=False,
                             pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True).strip()

print(diacritize("ذهب الطالب إلى المدرسة"))
```

**The prompt must match the one above** — it is what the adapter was trained
against, and a different instruction degrades output in ways that look like a bad
model rather than a harness mistake. Decode greedily (`do_sample=False`); the
reported numbers assume it. If you batch, set `tok.padding_side = "left"`.

## Training

Trained on nested subsets of
[`Misraj/Sadeed_Tashkeela`](https://huggingface.co/datasets/Misraj/Sadeed_Tashkeela),
drawn with a fixed shuffle seed so the 10% subset is contained in the 30%, and
that in the 50%. The corpus has a measured 0.4% overlap with the benchmark, which
is what makes the benchmark number meaningful.

| | |
|---|---|
| Method | LoRA, base weights frozen in bf16 (not QLoRA) |
| Rank / alpha / dropout | 16 / 32 / 0.05 |
| Target modules | `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` |
| Adapted matrices | 258 |
| Effective batch | 96 (4 x 8 x 3 GPUs) |
| Learning rate | 2e-4, cosine |
| Warmup | 5% of total steps |
| Epochs | 1 |
| Max sequence length | 1024 |
| Seed | 42 |
| Hardware | 3× A100 80GB, DDP via `torchrun` |

Both base models in the study share this configuration exactly, so differences
between them reflect architecture and data volume rather than tuning. The warmup
ratio of every run was verified after the fact from its logged learning-rate
curve.

## Limitations

Sentence-final case endings (*i'rab*) and Classical Arabic remain the dominant
error sources, as they are for every system in the study. These adapters were
trained for one epoch on a subset of a single corpus and are not expected to
transfer to dialectal Arabic or to Quranic orthography with its additional
annotation marks.

## Citation

If you use these adapters, please cite the benchmark and corpus they build on:

```bibtex
@article{sadeed2025,
  title  = {Sadeed: Advancing Arabic Diacritization Through Small Language Model},
  author = {Aldallal, Z. and Chrouf, S. and Hennara, K. and Hamed, M. M. and
            Hreden, M. and AlModhayan, S.},
  journal = {arXiv preprint arXiv:2504.21635},
  year   = {2025}
}
```