BENKHLOUF

BENKHLOUF (بن خلوف) is a 278.1M-parameter Matryoshka sentence embedding model for Algerian Darija (arq) across Arabic script, Latin Arabizi, and French code-switching, trained with Matryoshka Representation Learning (MRL) over Multiple Negatives Ranking Loss on 123,602 decontaminated sentence pairs.

On 1,000 held-out Algerian Darija validation pairs, BENKHLOUF achieves 0.853 mean positive cosine similarity at 768 dimensions and 0.897 at its 64-dimensional Matryoshka slice (a 12× reduction in vector storage), with 0.001 mean negative cosine (+0.852 contrastive margin) and 0.708 cross-script Arabizi-to-Arabic alignment.

Task Sentence Embeddings, Cross-Script Retrieval & Semantic Similarity
Architecture 12L × 768 Mean-Pooled Bi-Encoder with Matryoshka Slices (768 → 64d)
Parameters 278,075,904 (209.6M backbone + 68.4M expanded embeddings)
Input / Scripts Arabic script (arq-Arab), Latin Arabizi (arq-Latn), FR/EN code-switch (max 128 tokens)
License (Weights) Apache-2.0
Author(s) Abderahmane Ainouche, Algerian NLP Collective

Quickstart

Install dependencies:

pip install sentence-transformers torch

Standard 768d Embeddings (sentence-transformers)

from sentence_transformers import SentenceTransformer

REPO = "algerian-nlp/BENKHLOUF"
model = SentenceTransformer(REPO)

sentences = [
    "واش راك خويا لابس عليك؟",          # Arabic script
    "wesh rak khoya cava chwiya?",      # Latin Arabizi
    "السوق راه بعيد على الدار بزاف",     # Unrelated negative
]

embeddings = model.encode(sentences, normalize_embeddings=True)
similarity = embeddings @ embeddings.T

print(similarity.shape)
# (3, 3)
print(f"Arabic script <-> Arabizi similarity: {similarity[0, 1]:.3f}")
# Arabic script <-> Arabizi similarity: 0.424
print(f"Arabic script <-> Unrelated similarity: {similarity[0, 2]:.3f}")
# Arabic script <-> Unrelated similarity: 0.034

Matryoshka 64d Truncation (12× Smaller Index)

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("algerian-nlp/BENKHLOUF")
sentences = [
    "واش راك خويا لابس عليك؟",
    "wesh rak khoya cava chwiya?",
]

embeddings = model.encode(sentences)
embeddings_64d = embeddings[:, :64]
embeddings_64d = embeddings_64d / np.linalg.norm(embeddings_64d, axis=1, keepdims=True)

sim_64d = np.dot(embeddings_64d[0], embeddings_64d[1])
print(f"64d cosine similarity: {sim_64d:.3f}")
# 64d cosine similarity: 0.463

Direct transformers Usage

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

REPO = "algerian-nlp/BENKHLOUF"
tokenizer = AutoTokenizer.from_pretrained(REPO)
model = AutoModel.from_pretrained(REPO).eval()

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0]
    mask = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * mask, 1) / torch.clamp(mask.sum(1), min=1e-9)

texts = ["واش راك خويا", "wesh rak khoya"]
inputs = tokenizer(texts, padding=True, truncation=True, max_length=128, return_tensors="pt")

with torch.inference_mode():
    outputs = model(**inputs)
    embeddings = F.normalize(mean_pooling(outputs, inputs["attention_mask"]), p=2, dim=1)

print(embeddings.shape)
# torch.Size([2, 768])

Evaluation

Evaluated on 1,000 held-out Algerian Darija validation sentence pairs (2,000 sentences across distinct semantic clusters) after 3 epochs (2,898 global steps, batch size 64, single deterministic pass, seed 42).

Semantic Alignment & Contrastive Margin

Evaluation Dimension Dim Measured Value Target Property
Positive pair cosine 768 0.853 > 0.800 Paraphrase and parallel alignment
Matryoshka compressed slice 64 0.897 > 0.800 12× compressed vector search
Cross-script transfer 768 0.708 > 0.650 Arabizi ↔ Arabic script alignment
Negative pair cosine 768 0.001 < 0.050 Unrelated pair orthogonality
Contrastive margin 768 +0.852 > +0.750 Positive–negative separation
Embedding isotropy (mean cosine) 768 0.013 < 0.100 Isotropic space (no collapse)

Training & Generalization Trajectory

Checkpoint Step Train Loss Validation Loss Mean Cosine (Isotropy)
Initial 50 14.9100 — —
Epoch 1 966 0.0639 2.1770 0.0261
Epoch 2 1,932 0.0226 2.0640 0.0168
Epoch 3 (Final) 2,898 0.0125 2.0330 0.0160

Note on anchor suites: Collective suites such as DziriEval and MADAR evaluate multiple-choice QA and dialect classification rather than sentence retrieval / STS; standardized public Algerian Darija retrieval benchmarks remain an open community gap.


Intended Use & Scope

Recommended uses:

  • Dense retrieval, semantic search, and Retrieval-Augmented Generation (RAG) over Algerian Darija content.
  • Cross-script matching between Arabizi queries and Arabic-script documents without transliteration.
  • Semantic deduplication, FAQ matching, and customer-support routing.

Out-of-scope uses:

  • Text generation or machine translation.
  • Long-document encoding beyond 128 tokens.
  • Languages or dialects outside Algerian Darija, or automated decisions about individuals.

Training pairs include informal social-media vernacular. The model has not been evaluated for bias, toxicity, or factuality.


Architecture

Component Specification
Parameters 278,075,904 (209,635,584 backbone + 68,440,320 embeddings)
Backbone 12 layers, hidden size 768, 12 attention heads, GELU FFN (dim 3,072), initialized from XLM-RoBERTa base
Tokenizer 250,044 BPE tokens (expanded by 42 tokens over 250,002)
Pooling & Slices Mean pooling; Matryoshka dimensions 768, 512, 256, 128, 64
Context Length 128 tokens (learned absolute positions)
Objective MatryoshkaLoss wrapping MultipleNegativesRankingLoss
Release Precision float32 (trained with fp16 mixed precision)

Dialectal character expansion: 42 tokens were added to the BPE vocabulary to provide single-token entries for Algerian Darija characters missing from standard multilingual vocabularies, including ڨ (U+06A8, voiced velar stop /g/, occurring 1,112 times in training pairs), ڢ (U+06A2), and ڋ (U+068B), preventing <unk> fallback.


Training Data

Trained on 123,602 deduplicated, decontaminated Algerian Darija sentence pairs across 62,301 semantic clusters (with 1,000 held-out validation pairs). Paraphrases and translation variants are grouped into disjoint semantic clusters so mini-batches sample strictly across distinct clusters, preventing false negatives. Text normalization applies Unicode NFC and collapses redundant whitespace while preserving Arabizi phoneme digits (3, 7, 9), tatweel (ـ), diacritics, and French accented characters.

Source Domain / Type License Tier Train Pairs Share
Kamel Touati et al. Parallel English–Darija corpus Permissive (CC-BY-4.0) 113,804 92.07%
PADIC (Algerian component) Dialectal Arabic parallel corpus Academic research use 9,798 7.93%
Total Tri-script Algerian Darija pairs Mixed 123,602 100.00%

Training Data License Composition

The model weights are released under Apache-2.0. That grant applies to weights and code only and does not relicense the underlying training text.

Tier Pairs Share
Permissive (CC-BY-4.0) 96,830 78.34%
Academic research use 26,772 21.66%

Training Details

Parameter Value
Objective MatryoshkaLoss (768, 512, 256, 128, 64) + MultipleNegativesRankingLoss
Optimizer AdamW (lr=2e-5, weight_decay=0.01, betas=(0.9, 0.999), eps=1e-8)
Batch Size 64 pairs/step (128 sequences/step)
Schedule Linear warmup (289 steps, 10%) + cosine decay to 0
Steps / Epochs 2,898 steps (3 epochs, 370,806 pair exposures)
Regularization Dropout 0.10, gradient clipping 1.0
Precision fp16 mixed precision
Hardware & Duration 2× NVIDIA Tesla T4 (16 GB), 2.78 h (9,996.8 s)
Seed 42

Limitations

  • 128-token context cap. Optimized for sentences, search queries, comments, and short messages rather than long documents.
  • Colloquial Algerian domain focus. Trained specifically on Algerian Darija; formal MSA and non-Maghrebi Arabic varieties are better served by general Arabic encoders.
  • Single training run. All reported metrics come from one seed-42 run; cross-seed variance is unmeasured.
  • No safety evaluation. Bias, toxicity, and factuality have not been evaluated.

Repository Contents

File Size Description / SHA-256
model.safetensors 1.11 GB Float32 weights (199 tensors) · SHA-256 0d718427ea4437db4477f0e0dc77c3ba9bd6fd40c86018e86e26b6bf1c94f129
config.json 742 B Model architecture configuration
config_sentence_transformers.json 284 B SentenceTransformers metadata and similarity config
modules.json 229 B Pipeline definition (Transformer + Pooling)
1_Pooling/config.json 90 B Mean-pooling configuration
tokenizer.json 17.1 MB Fast BPE tokenizer (250,044 tokens) · SHA-256 d45fd5f5313b236c9edabcdd81258f6ffb9f788bf4639f2afd6bc4672294785c
sentencepiece.bpe.model 5.07 MB SentencePiece BPE model · SHA-256 cfc8146abe2a0488e9e2a0c56de7952f7c11ab059eca145a0a727afce0db2865
tokenizer_config.json 382 B Tokenizer settings
special_tokens_map.json 166 B Special token definitions
export_report.json 1.7 KB Verified file sizes, SHA-256 checksums, and parameter count

Optimizer states and intermediate checkpoints are excluded. The published weights reload via SentenceTransformer("algerian-nlp/BENKHLOUF") with zero missing or unexpected keys.


The name

Sidi Lakhdar Ben Khlouf (c. 1479–c. 1585), from the Dahra region near Mostaganem, is celebrated as a foundational master of Algerian Malhun poetry. Writing in vernacular Algerian Arabic rather than classical court verse, his poetry preserved both popular memory and everyday speech across the 16th century. Naming a cross-script Darija embedding model after him honors that vernacular tradition: capturing shared meaning across written scripts.

The naming is homage and implies no endorsement.


Citation

@misc{algerian_nlp_benkhlouf_2026,
  title  = {BENKHLOUF: Matryoshka Sentence Embeddings for Algerian Darija},
  author = {Ainouche, Abderahmane and Algerian NLP Collective},
  year   = {2026},
  url    = {https://huggingface.co/algerian-nlp/BENKHLOUF}
}

Upstream backbone:

@inproceedings{conneau-etal-2020-unsupervised,
  title     = {Unsupervised Cross-lingual Representation Learning at Scale},
  author    = {Conneau, Alexis and Khandelwal, Kartikay and Goyal, Naman and Chaudhary, Vishrav and Wenzek, Guillaume and Guzm{\'a}n, Francisco and Grave, Edouard and Ott, Myle and Zettlemoyer, Luke and Stoyanov, Veselin},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year      = {2020},
  url       = {https://aclanthology.org/2020.acl-main.747}
}

License

Weights and code are released under Apache-2.0. Refer to the Training Data License Composition section above regarding the underlying training text before redistributing derivatives.

Downloads last month
165
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including algerian-nlp/BENKHLOUF

Evaluation results

  • Positive Pair Cosine Similarity (768d, 3 epochs) on Algerian Darija STS Validation (1,000 pairs)
    self-reported
    0.853
  • Matryoshka 64d Cosine Similarity (64d, 3 epochs) on Algerian Darija STS Validation (1,000 pairs)
    self-reported
    0.897
  • Cross-Script Transfer Arabizi-to-Arabic (768d, 3 epochs) on Algerian Darija STS Validation (1,000 pairs)
    self-reported
    0.708
  • Negative Pair Orthogonality (768d, 3 epochs) on Algerian Darija STS Validation (1,000 pairs)
    self-reported
    0.001