---
language:
- ar
library_name: transformers
pipeline_tag: fill-mask
base_model:
- UBC-NLP/MARBERT
tags:
- arabic
- maghrebi-arabic
- darija
- arabizi
- code-switching
- algeria
- morocco
- tunisia
- libya
- bert
- pytorch
- masked-language-modeling
---

<img src="https://i.imgur.com/cwgkvqe.png"
alt="MagharibiBERT" width="35%" align="right"/>

# MagharibiBERT

**MagharibiBERT** is a regionally adapted Transformer language model
for **Maghrebi Arabic**.

It is initialized from `UBC-NLP/MARBERT` and further adapted using
Masked Language Modeling (MLM) on a large multi-source corpus of
Maghrebi text.

The model targets the linguistic and orthographic diversity of
North African online communication, including Arabic-script dialect,
Arabizi, spelling variation, and Arabic-French/English code-switching.

MagharibiBERT is developed by **NumidiaNLP**.

---

## Model architecture

| Property | Value |
|---|---:|
| Architecture | BERT |
| Parent model | `UBC-NLP/MARBERT` |
| Objective | Masked Language Modeling |
| Parameters | 162,942,880 |
| Transformer layers | 12 |
| Hidden size | 768 |
| Attention heads | 12 |
| Vocabulary size | 100,000 |
| Maximum position embeddings | 512 |

---

## Regional continued pretraining

MagharibiBERT underwent additional regional continued pretraining on
a curated mixture of **17 corpora** covering Maghrebi Arabic and
North African social-media language.

The training mixture includes material associated with Algeria, Tunisia,
Morocco, Libya, and cross-regional Maghrebi sources, with explicit
coverage of Arabic script, Arabizi, and code-mixed text.

### Training configuration

- MLM sequence length: 128
- Optimizer steps: 120,000
- Effective batch size: 48
- Learning rate: 2e-5
- Warmup steps: 6,000
- Weight decay: 0.01
- Precision: BF16
- Random seed: 42

---

## How to use

### Fill-mask pipeline

```python
from transformers import pipeline

pipe = pipeline(
    "fill-mask",
    model="NumidiaNLP/MagharibiBERT"
)

results = pipe("واش راك [MASK]", top_k=5)

for result in results:
    print(result["token_str"], result["score"])
```

### Example output

Output produced by the released MagharibiBERT checkpoint:

```text
{"token": "دير", "score": 0.277943}
{"token": "تهدر", "score": 0.071477}
{"token": "تدير", "score": 0.062739}
{"token": "تقول", "score": 0.047855}
{"token": "تحكي", "score": 0.038360}
```

> Scores depend on the complete masked context. The example above is
> provided only to demonstrate model usage rather than as an evaluation result.

### Load the model directly

```python
from transformers import AutoTokenizer, AutoModelForMaskedLM

model_name = "NumidiaNLP/MagharibiBERT"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)
```

---

## Intended use

MagharibiBERT is intended primarily for research involving Maghrebi Arabic,
including:

- hate and offensive language detection,
- sentiment analysis,
- dialect identification,
- Arabizi processing,
- code-switched text processing,
- social-media text classification,
- representation learning for Maghrebi Arabic.

---

## Intrinsic evaluation

On the fixed held-out MLM test set used during development, the released
checkpoint obtained:

| Metric | MagharibiBERT |
|---|---:|
| MLM loss | 2.8099 |
| Perplexity | 16.6080 |
| Masked-token accuracy | 52.42% |

Downstream evaluation against MARBERT and other Arabic language models
is being conducted separately.

---

## Limitations

Maghrebi Arabic is highly heterogeneous across countries, regions,
writing systems, platforms, and communities. Performance may therefore
vary across domains and linguistic varieties.

The model should not be interpreted as representing all North African
languages or all Maghrebi speakers equally.

---

## Authors

- **Ahmed Zoubir Messalti**
- **Sadik Bessou**

**Institution:** University Ferhat Abbas Sétif 1, Algeria  
**Corresponding author:** ahmedzoubir.messalti@univ-setif.dz

---

## Citation

If you use MagharibiBERT in your research, please cite the accompanying paper:

> Ahmed Zoubir Messalti and Sadik Bessou.  
> **MagharibiBERT: A Regionally Adapted Language Model for Maghrebi Arabic Social Media.**  
> *International Journal of Computers and Their Applications*, 2026.  
> **Accepted for publication. Planned issue: Volume 33, No. 4, December 2026.**

Final issue assignment, page numbers, and the article link will be updated upon publication.

```bibtex
@article{messalti2026magharibibert,
  author  = {Messalti, Ahmed Zoubir and Bessou, Sadik},
  title   = {{MagharibiBERT}: A Regionally Adapted Language Model
             for Maghrebi Arabic Social Media},
  journal = {International Journal of Computers and Their Applications},
  year    = {2026},
  note    = {Accepted for publication.
             Planned issue: volume 33, number 4, December 2026}
}
```

MagharibiBERT is developed within the MARBERT model lineage.
Users should also cite the original MARBERT work:

> Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. (2021).  
> **ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic.**  
> *ACL-IJCNLP 2021*, pp. 7088–7105.  
> https://doi.org/10.18653/v1/2021.acl-long.551

---

## Organization

**NumidiaNLP — Language Intelligence for North Africa**