Add model card

f89979a verified about 2 months ago

6.03 kB

---
base_model: google/gemma-4-E4B-it
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:google/gemma-4-E4B-it
- lora
- arabic
- diacritization
- tashkeel
language:
- ar
datasets:
- Misraj/Sadeed_Tashkeela
- Misraj/SadeedDiac-25
license: apache-2.0
---
# Gemma 4 E4B-it + LoRA for Arabic Diacritisation

LoRA adapters that restore Arabic diacritics (tashkeel) on top of
[`google/gemma-4-E4B-it`](https://huggingface.co/google/gemma-4-E4B-it). Three adapters are
included, trained on 10%, 30% and 50% of the corpus, so the effect of training
data volume can be read directly.

Part of **DiacriticS**, a contamination-controlled study of open-weights models on
this task — [project site](https://diacritics.vercel.app/) · [code](https://github.com/Psycodem/DiacriticS-Language-Model-for-Arabic-Diacritisation).

## Which subfolder to use

Each fraction is a subfolder of this repo. `50pct` is the strongest.

| Subfolder | Training rows | DER (CE) | DER (no CE) | WER (CE) | WER (no CE) |
|---|---:|---:|---:|---:|---:|
| `10pct` | 104,270 | **3.15** | 2.68 | 7.28 | 4.47 |
| `30pct` | 312,809 | **2.98** | 2.54 | 6.82 | 4.12 |
| `50pct` | 521,349 | **2.81** | 2.38 | 6.54 | 3.96 |

Scored on the full 1,200-paragraph **SadeedDiac-25** benchmark with
[`Evaluation_Functions_Corrected.py`](https://github.com/Psycodem/DiacriticS-Language-Model-for-Arabic-Diacritisation/blob/main/Evaluation_Functions_Corrected.py).
Percentages; lower is better. CE = sentence-final case ending (*i'rab*).

<details>
<summary>All three splits</summary>

DER / WER with case endings.

| Subfolder | Train sample (500) | Tashkeela test (2,485) | SadeedDiac-25 (1,200) |
|---|---|---|---|
| `10pct` | 1.23 / 3.24 | 12.96 / 16.2 | 3.15 / 7.28 |
| `30pct` | 1.81 / 3.54 | 12.73 / 15.73 | 2.98 / 6.82 |
| `50pct` | 1.76 / 3.19 | 12.57 / 15.41 | 2.81 / 6.54 |

The Tashkeela test split scores worse than the benchmark because it inherits the
residual annotation noise of the source corpus, not because the model does worse
on it. SadeedDiac-25 is expert-reviewed and balanced 50/50 between Modern
Standard and Classical Arabic.

</details>

## Usage

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
BASE     = "google/gemma-4-E4B-it"
ADAPTER  = "Psycodem/gemma-4-e4b-lora-diacritization"
FRACTION = "50pct"          # "10pct" | "30pct" | "50pct"
tok = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
if tok.pad_token is None:
    tok.pad_token = tok.eos_token
model = AutoModelForCausalLM.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True)
model = PeftModel.from_pretrained(model, ADAPTER, subfolder=FRACTION)
model = model.merge_and_unload()
model.eval()
SYSTEM_PROMPT = (
    "أنت نظام متخصص في التشكيل الآلي للنصوص العربية. "
    "مهمتك إضافة الحركات (التشكيل) الصحيحة إلى النص العربي المُدخل دون تغيير الكلمات أو ترتيبها، "
    "مع مراعاة السياق النحوي والصرفي الكامل للجملة."
)
def diacritize(text):
    msgs = [{"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text}]
    try:
        prompt = tok.apply_chat_template(msgs, tokenize=False,
                                         add_generation_prompt=True, enable_thinking=False)
    except TypeError:
        prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
    enc = tok(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**enc, max_new_tokens=512, do_sample=False,
                             pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True).strip()
print(diacritize("ذهب الطالب إلى المدرسة"))
```

**The prompt must match the one above** — it is what the adapter was trained
against, and a different instruction degrades output in ways that look like a bad
model rather than a harness mistake. Decode greedily (`do_sample=False`); the
reported numbers assume it. If you batch, set `tok.padding_side = "left"`.

## Training

Trained on nested subsets of
[`Misraj/Sadeed_Tashkeela`](https://huggingface.co/datasets/Misraj/Sadeed_Tashkeela),
drawn with a fixed shuffle seed so the 10% subset is contained in the 30%, and
that in the 50%. The corpus has a measured 0.4% overlap with the benchmark, which
is what makes the benchmark number meaningful.

| | |
|---|---|
| Method | LoRA, base weights frozen in bf16 (not QLoRA) |
| Rank / alpha / dropout | 16 / 32 / 0.05 |
| Target modules | `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` |
| Adapted matrices | 258 |
| Effective batch | 96 (4 x 8 x 3 GPUs) |
| Learning rate | 2e-4, cosine |
| Warmup | 5% of total steps |
| Epochs | 1 |
| Max sequence length | 1024 |
| Seed | 42 |
| Hardware | 3× A100 80GB, DDP via `torchrun` |

Both base models in the study share this configuration exactly, so differences
between them reflect architecture and data volume rather than tuning. The warmup
ratio of every run was verified after the fact from its logged learning-rate
curve.

## Limitations

Sentence-final case endings (*i'rab*) and Classical Arabic remain the dominant
error sources, as they are for every system in the study. These adapters were
trained for one epoch on a subset of a single corpus and are not expected to
transfer to dialectal Arabic or to Quranic orthography with its additional
annotation marks.

## Citation

If you use these adapters, please cite the benchmark and corpus they build on:

```bibtex
@article{sadeed2025,
  title  = {Sadeed: Advancing Arabic Diacritization Through Small Language Model},
  author = {Aldallal, Z. and Chrouf, S. and Hennara, K. and Hamed, M. M. and
            Hreden, M. and AlModhayan, S.},
  journal = {arXiv preprint arXiv:2504.21635},
  year   = {2025}
}
```

	---
	base_model: google/gemma-4-E4B-it
	library_name: peft
	pipeline_tag: text-generation
	tags:
	- base_model:adapter:google/gemma-4-E4B-it
	- lora
	- arabic
	- diacritization
	- tashkeel
	language:
	- ar
	datasets:
	- Misraj/Sadeed_Tashkeela
	- Misraj/SadeedDiac-25
	license: apache-2.0
	---

	# Gemma 4 E4B-it + LoRA for Arabic Diacritisation

	LoRA adapters that restore Arabic diacritics (tashkeel) on top of
	[`google/gemma-4-E4B-it`](https://huggingface.co/google/gemma-4-E4B-it). Three adapters are
	included, trained on 10%, 30% and 50% of the corpus, so the effect of training
	data volume can be read directly.

	Part of DiacriticS, a contamination-controlled study of open-weights models on
	this task — [project site](https://diacritics.vercel.app/) · [code](https://github.com/Psycodem/DiacriticS-Language-Model-for-Arabic-Diacritisation).

	## Which subfolder to use

	Each fraction is a subfolder of this repo. `50pct` is the strongest.

	\| Subfolder \| Training rows \| DER (CE) \| DER (no CE) \| WER (CE) \| WER (no CE) \|
	\|---\|---:\|---:\|---:\|---:\|---:\|
	\| `10pct` \| 104,270 \| 3.15 \| 2.68 \| 7.28 \| 4.47 \|
	\| `30pct` \| 312,809 \| 2.98 \| 2.54 \| 6.82 \| 4.12 \|
	\| `50pct` \| 521,349 \| 2.81 \| 2.38 \| 6.54 \| 3.96 \|

	Scored on the full 1,200-paragraph SadeedDiac-25 benchmark with
	[`Evaluation_Functions_Corrected.py`](https://github.com/Psycodem/DiacriticS-Language-Model-for-Arabic-Diacritisation/blob/main/Evaluation_Functions_Corrected.py).
	Percentages; lower is better. CE = sentence-final case ending (i'rab).

	<details>
	<summary>All three splits</summary>

	DER / WER with case endings.

	\| Subfolder \| Train sample (500) \| Tashkeela test (2,485) \| SadeedDiac-25 (1,200) \|
	\|---\|---\|---\|---\|
	\| `10pct` \| 1.23 / 3.24 \| 12.96 / 16.2 \| 3.15 / 7.28 \|
	\| `30pct` \| 1.81 / 3.54 \| 12.73 / 15.73 \| 2.98 / 6.82 \|
	\| `50pct` \| 1.76 / 3.19 \| 12.57 / 15.41 \| 2.81 / 6.54 \|

	The Tashkeela test split scores worse than the benchmark because it inherits the
	residual annotation noise of the source corpus, not because the model does worse
	on it. SadeedDiac-25 is expert-reviewed and balanced 50/50 between Modern
	Standard and Classical Arabic.

	</details>

	## Usage

	```python
	import torch
	from transformers import AutoTokenizer, AutoModelForCausalLM
	from peft import PeftModel

	BASE = "google/gemma-4-E4B-it"
	ADAPTER = "Psycodem/gemma-4-e4b-lora-diacritization"
	FRACTION = "50pct" # "10pct" \| "30pct" \| "50pct"

	tok = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
	if tok.pad_token is None:
	tok.pad_token = tok.eos_token

	model = AutoModelForCausalLM.from_pretrained(
	BASE, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True)
	model = PeftModel.from_pretrained(model, ADAPTER, subfolder=FRACTION)
	model = model.merge_and_unload()
	model.eval()

	SYSTEM_PROMPT = (
	"أنت نظام متخصص في التشكيل الآلي للنصوص العربية. "
	"مهمتك إضافة الحركات (التشكيل) الصحيحة إلى النص العربي المُدخل دون تغيير الكلمات أو ترتيبها، "
	"مع مراعاة السياق النحوي والصرفي الكامل للجملة."
	)

	def diacritize(text):
	msgs = [{"role": "system", "content": SYSTEM_PROMPT},
	{"role": "user", "content": text}]
	try:
	prompt = tok.apply_chat_template(msgs, tokenize=False,
	add_generation_prompt=True, enable_thinking=False)
	except TypeError:
	prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
	enc = tok(prompt, return_tensors="pt").to(model.device)
	with torch.no_grad():
	out = model.generate(**enc, max_new_tokens=512, do_sample=False,
	pad_token_id=tok.pad_token_id)
	return tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True).strip()

	print(diacritize("ذهب الطالب إلى المدرسة"))
	```

	The prompt must match the one above — it is what the adapter was trained
	against, and a different instruction degrades output in ways that look like a bad
	model rather than a harness mistake. Decode greedily (`do_sample=False`); the
	reported numbers assume it. If you batch, set `tok.padding_side = "left"`.

	## Training

	Trained on nested subsets of
	[`Misraj/Sadeed_Tashkeela`](https://huggingface.co/datasets/Misraj/Sadeed_Tashkeela),
	drawn with a fixed shuffle seed so the 10% subset is contained in the 30%, and
	that in the 50%. The corpus has a measured 0.4% overlap with the benchmark, which
	is what makes the benchmark number meaningful.

	\| \| \|
	\|---\|---\|
	\| Method \| LoRA, base weights frozen in bf16 (not QLoRA) \|
	\| Rank / alpha / dropout \| 16 / 32 / 0.05 \|
	\| Target modules \| `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj` \|
	\| Adapted matrices \| 258 \|
	\| Effective batch \| 96 (4 x 8 x 3 GPUs) \|
	\| Learning rate \| 2e-4, cosine \|
	\| Warmup \| 5% of total steps \|
	\| Epochs \| 1 \|
	\| Max sequence length \| 1024 \|
	\| Seed \| 42 \|
	\| Hardware \| 3× A100 80GB, DDP via `torchrun` \|

	Both base models in the study share this configuration exactly, so differences
	between them reflect architecture and data volume rather than tuning. The warmup
	ratio of every run was verified after the fact from its logged learning-rate
	curve.

	## Limitations

	Sentence-final case endings (i'rab) and Classical Arabic remain the dominant
	error sources, as they are for every system in the study. These adapters were
	trained for one epoch on a subset of a single corpus and are not expected to
	transfer to dialectal Arabic or to Quranic orthography with its additional
	annotation marks.

	## Citation

	If you use these adapters, please cite the benchmark and corpus they build on:

	```bibtex
	@article{sadeed2025,
	title = {Sadeed: Advancing Arabic Diacritization Through Small Language Model},
	author = {Aldallal, Z. and Chrouf, S. and Hennara, K. and Hamed, M. M. and
	Hreden, M. and AlModhayan, S.},
	journal = {arXiv preprint arXiv:2504.21635},
	year = {2025}
	}
	```