Amrouche-2-1.3B

A 1.37B-parameter (1,370,638,336) bidirectional machine translation model for Kabyle (Taqbaylit, kab_Latn), English (eng_Latn), French (fra_Latn), and Modern Standard Arabic (arb_Arab), fine-tuned from facebook/nllb-200-distilled-1.3B on 544,965 verified parallel sentence pairs from the AƔBALU parallel corpus.

Amrouche-2-1.3B outperforms base NLLB-1.3B across all six translation directions on FLORES+ devtest, averaging 38.22 macro chrF++ (+2.92 gain) and 13.80 macro BLEU (+14.2% relative gain). On the held-out gold test split (868 sentence pairs), it reaches 52.31 macro chrF++ and 31.99 macro BLEU.

Unlike Amrouche-1.3B, which trimmed the embedding table to 52,209 tokens, Amrouche-2-1.3B preserves the full 256,206-token SentencePiece vocabulary. This eliminates <unk> substitutions on unseen vocabulary and maintains direct compatibility with M2M100ForConditionalGeneration, CTranslate2, and ONNX runtimes.

For resource-constrained environments, edge devices, or high-throughput batch translation, see the lightweight companion model Amrouche-2-600M (615M parameters, 1.23 GB BF16), as well as the official quantized CTranslate2 releases:

• Amrouche-2-1.3B-CT2-int8 (1.38 GB int8, 205.6 ms single-sentence CPU latency, 7.1× speedup over PyTorch with only -0.14 chrF++ quantization delta).
• Amrouche-2-600M-CT2-int8 (622.6 MB int8, 155.0 ms single-sentence CPU latency, 4.9× speedup over PyTorch with only -0.15 chrF++ quantization delta).

Results

All models are evaluated on FLORES+ devtest (1,012 sentences per direction) using sacrebleu (chrF++ with word order 2 and BLEU with 13a tokenization) against the orthographically corrected kab_Latn reference:

Direction	chrF++ (Base → Amrouche-2-1.3B)	BLEU (Base → Amrouche-2-1.3B)	Relative BLEU Gain
kab → eng	44.60 → 47.78 (+3.18)	23.06 → 25.81	+11.9%
kab → fra	42.00 → 44.64 (+2.64)	19.46 → 21.68	+11.4%
eng → kab	31.53 → 35.26 (+3.73)	7.74 → 9.90	+27.9%
kab → arb	33.78 → 34.98 (+1.20)	9.64 → 10.19	+5.7%
fra → kab	30.21 → 33.53 (+3.32)	6.33 → 7.65	+20.9%
arb → kab	29.69 → 33.13 (+3.44)	6.24 → 7.54	+20.8%
6-Direction Macro	35.30 → 38.22 (+2.92)	12.08 → 13.80	+14.2%

• Bidirectional Arabic Support: Expands beyond English and French to include full bidirectional translation with Modern Standard Arabic (arb_Arab), outperforming the base model by +3.44 chrF++ on arb → kab and +1.20 chrF++ on kab → arb.
• Target-Side Generation: Gains are concentrated when generating Kabyle (+3.73 chrF++ on eng → kab, +3.44 on arb → kab, and +3.32 on fra → kab), addressing the base model's morphology and agreement errors.
• Full Vocabulary Retention: Preserving the complete 256,206-token vocabulary lifts kab → eng to 47.78 chrF++ (+1.53 chrF++ above Amrouche-1.3B's 46.25) by eliminating <unk> substitutions on unseen named entities and loanwords.

Orthographic Consistency

Because the upstream FLORES+ kab_Latn split contains 16.2% homoglyph corruption (such as Greek ε U+03B5 in place of Latin ɛ U+025B), hypotheses and references are scored both raw and after passing through the versioned AƔBALU normalizer (1.3.0+rules1.0.0):

Evaluation Condition	Base NLLB-1.3B	Amrouche-2-1.3B
eng → kab Orthography Gap (chrF++)	+0.17	0.00
fra → kab Orthography Gap (chrF++)	+0.19	0.00
arb → kab Orthography Gap (chrF++)	+0.15	0.00

Normalizing hypotheses and references yields identical scores (an orthography gap of 0.00), confirming that Amrouche-2-1.3B natively outputs canonical Kabyle Latin characters (ɣ ɛ ḥ ḍ ṣ ṭ ẓ ṛ č ǧ ţ) without character drift.

Comparison: Amrouche-2-1.3B vs. Amrouche-2-600M

Amrouche-2 is provided in two complementary parameter tiers sharing the exact same verified bitext curriculum, tokenizer, and standalone inference architecture:

Attribute	Amrouche-2-1.3B	Amrouche-2-600M	Trade-off / Delta
Parameters	1,370,638,336 (1.37B)	615,073,792 (615M)	-55.1% parameters
Model Size (BF16)	2.74 GB	1.23 GB	-55.1% disk / VRAM
Model Size (CT2 int8)	1.38 GB	622.6 MB	-54.9% storage
FLORES+ Macro chrF++	38.22	34.88	+3.34 chrF++
Gold Test Macro chrF++	52.36	48.58	+3.78 chrF++
Layers (Enc / Dec)	24 / 24	12 / 12	Half depth
FFN Intermediate Dim	8,192	4,096	Half width
CPU Latency (batch=1, 8 threads)	206 ms (CT2) / 958 ms (PT)	155 ms (CT2) / 631 ms (PT)	+33% lower latency
CPU Throughput (batch=8, 8 threads)	12.93 sent/s (178.7 tok/s)	17.05 sent/s (233.6 tok/s)	+31% faster throughput

Intended Use

• Primary applications: Bidirectional sentence, paragraph, and document translation between Kabyle and English, French, or Modern Standard Arabic.
• Downstream integration: Bilingual corpus alignment, subtitle and UI localization, cross-lingual information retrieval, and pivot translation (X → eng → kab or kab → eng → X).

Not suitable for:

• Safety-critical medical, clinical, or legally binding decisions without qualified human review.
• Sibling Amazigh languages such as Tashelhit (shi), Central Atlas Tamazight (tzm), Tarifit (rif), Shawiya (shy), or Tamasheq (taq), which have distinct lexicons and morphosyntax.
• Unstandardized Arabizi or Neo-Tifinagh inputs without prior conversion through Boulifa-48M or Juba-27M.

Usage

Amrouche-2-1.3B can be used via custom generation presets (trust_remote_code=True), standard Hugging Face pipeline, or native M2M100ForConditionalGeneration:

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "agbalu/Amrouche-2-1.3B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
).eval()

# Batch translation (English -> Kabyle)
print(
    model.translate(
        ["The house is big.", "I speak Kabyle.", "Water is life."],
        src_lang="en",
        tgt_lang="kab",
        tokenizer=tokenizer,
    )
)
# ['Meqqer uxxam-nni.', 'Heddreɣ taqbaylit.', 'Aman d tudert.']

# Modern Standard Arabic -> Kabyle
print(
    model.translate(
        "هل بإمكانك أن تفتح لي الباب؟",
        src_lang="ar",
        tgt_lang="kab",
        tokenizer=tokenizer,
    )
)
# 'Tzemreḍ ad iyi-d-teldiḍ tawwurt?'

# Kabyle -> French
print(
    model.translate(
        "Aman d tudert, taqbaylit d tutlayt-nneɣ.",
        src_lang="kab",
        tgt_lang="fr",
        tokenizer=tokenizer,
    )
)
# "L'eau est la vie, le kabyle est notre langue."

Using the Hugging Face pipeline API:

from transformers import pipeline

translator = pipeline(
    "translation",
    model="agbalu/Amrouche-2-1.3B",
    trust_remote_code=True,
    src_lang="eng_Latn",
    tgt_lang="kab_Latn",
)
print(translator("When did you graduate?"))
# [{'translation_text': 'Melmi i d-tewwiḍ agerdas?'}]

Using native M2M100ForConditionalGeneration without remote code:

import torch
from transformers import AutoTokenizer, M2M100ForConditionalGeneration

model_id = "agbalu/Amrouche-2-1.3B"
tokenizer = AutoTokenizer.from_pretrained(model_id, src_lang="eng_Latn")
model = M2M100ForConditionalGeneration.from_pretrained(model_id, dtype=torch.bfloat16).eval()

inputs = tokenizer(["Water is life."], return_tensors="pt", padding=True)
with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        forced_bos_token_id=tokenizer.convert_tokens_to_ids("kab_Latn"),
        num_beams=4,
        max_new_tokens=128,
    )
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
# ['Aman d tudert.']

Standalone Inference Features (trust_remote_code=True)

When loaded with trust_remote_code=True, model.translate(...) activates the standalone AƔBALU inference engine:

• Arbitrary Document Length: Automatically chunks multi-paragraph texts along sentence boundaries and reassembles them with exact layout preservation, bypassing NLLB's 512-token context limit.
• In-Flight Normalisation & Typography Folding: Standardises Kabyle orthography and folds smart typography (« » “ ” —) that base NLLB maps to <unk>, preventing token fragmentation.
• Decoding Presets:
  • preset="fast": 2-beam low-latency decoding.
  • preset="balanced" (default): 4-beam search with confidence validation.
  • preset="quality": Multi-candidate generation with Minimum Bayes Risk (chrF++) and round-trip noisy-channel reranking.
• Failure Detection & Retry: Automatically detects repetition loops, untranslated copies, and off-script drift, retrying affected segments with contrastive guidance.
• Zero Remote Code Alternative: For standard production environments (vLLM, CTranslate2, ONNX), use native M2M100ForConditionalGeneration directly (trust_remote_code=False).

Architecture

Parameter	Specification
Parameters, trained	1,370,638,336
Parameters, published	1,370,638,336
Base Model	facebook/nllb-200-distilled-1.3B
Backbone	24 encoder layers, 24 decoder layers, 1,024 hidden dimension, 16 attention heads, 8,192 intermediate (ReLU)
Context Window	512 tokens per segment (1,024 maximum positional embeddings; unbounded document length via automatic segmentation)
Position Embeddings	Sinusoidal positional embeddings
Vocabulary	256,206 tokens (native NLLB SentencePiece BPE, tied input/output embeddings)
Precision	BF16

The model preserves the exact parameter structure and tied embedding matrix (model.shared.weight) of M2M100ForConditionalGeneration. No vocabulary trimming or adapter layers are used, allowing direct conversion to CTranslate2 and ONNX runtimes.

Training Recipe

Parameter	Specification
Corpus Size	544,965 parallel pairs → 1,089,930 directional examples (Stage 1: 544,965 pairs; Stage 2: 539,494 gold pairs)
Optimizer	Fused AdamW, $\beta=(0.9, 0.98)$, $\epsilon=10^{-8}$, weight decay 0.01, gradient norm clipping 1.0
Learning Rate	Stage 1: $1.0\times 10^{-4} \to 1.0\times 10^{-5}$; Stage 2: $2.0\times 10^{-5} \to 2.0\times 10^{-6}$ (cosine decay)
Effective Batch Size	32,768 target tokens per update (2,048 micro-batch tokens, max 16 sequences per micro-step)
Training Steps	1,560 total updates (Stage 1: 1,091 steps; Stage 2: 469 steps)
Precision	bfloat16 with gradient checkpointing

• Two-Stage Curriculum: Stage 1 adapted the base checkpoint on 544,965 parallel pairs using a 60/40 sampling mix of gold human translations and multi-sentence web alignments capped at 3 segments. Stage 2 reinitialized the optimizer at a $5\times$ lower learning rate and trained exclusively on 539,494 verified human gold pairs across all six translation directions.
• Exclusion of Mined Bitext: Approximately 90.1% of publicly available Kabyle parallel data is web-mined by NLLB with unverified precision and severe sibling-Berber confusion. All 440,750 unverified mined records were excluded to prevent distilling base-model errors.
• Full Parameter Fine-Tuning: Full parameter fine-tuning in bfloat16 with gradient checkpointing was used over parameter-efficient methods to ensure complete adaptation across all encoder and decoder layers.

Decontamination

All candidate rows were filtered through exact-string and 13-word n-gram overlap checks against FLORES+ (dev and devtest, raw and normalized kab_Latn, eng_Latn, fra_Latn, and arb_Arab), SIB-200, and local evaluation references. Overlapping records were quarantined prior to training. Multi-sentence records and shared translation hubs were grouped transitively via union-find ContentGroups, guaranteeing zero sentence or document overlap across training, validation, and test splits.

Limitations

• Dialect & Script Scope: Trained and evaluated on standard Kabyle Latin orthography (kab_Latn). Informal keyboard spellings, Arabizi numerals (3, 7, 9), and Neo-Tifinagh script require upstream normalization before translation.
• Domain Coverage: Because 76.6% of gold training pairs originate from Tatoeba community translations and 4.2% from biblical text, literary and conversational sentences are better covered than specialized legal, clinical, or industrial terminology.
• Arabic Supervision Share: Direct arb_Arab ↔ kab_Latn gold training data comprises 4,135 pairs per direction (0.77% of Stage 2 training examples). On technical Arabic prose, pivoting through English (arb_Arab → eng_Latn → kab_Latn) can yield higher lexical precision.
• Human Adequacy Evaluation: Reported metrics (chrF++, BLEU) are reference-based automated measurements; formal human adequacy and fluency evaluations have not yet been conducted.

Files

File	Size	Description
model.safetensors	2.74 GB	Published BF16 model weights (1,370,638,336 parameters)
tokenizer.json	32.2 MB	Fast NLLB-200 subword tokenizer (256,206 tokens)
sentencepiece.bpe.model	4.85 MB	Native SentencePiece BPE model (CTranslate2 compatible)
modeling_amrouche.py	40.5 KB	Standalone AmroucheForConditionalGeneration and MBR reranker
preprocessing_amrouche.py	20.4 KB	Kabyle character normalizer, typography folder, and segmenter
pipeline_amrouche.py	4.3 KB	Standalone AmroucheTranslationPipeline implementation
tokenizer_config.json	3.7 KB	Language-tag registry and tokenizer configuration
special_tokens_map.json	3.5 KB	Special token definitions (<s>, <pad>, </s>, <unk>)
config.json	1.1 KB	Architecture hyperparameters, auto_map, and custom_pipelines
generation_config.json	184 B	Default decoder generation parameters and token IDs
export.stats.json	2.0 KB	Checksum manifest and staging verification report

Reproduction

Fine-tuned from facebook/nllb-200-distilled-1.3B across 544,965 verified parallel pairs (1,089,930 directional examples) using fused AdamW with effective batch size 32,768 target tokens over a two-stage curriculum (1,560 total steps). Evaluated across all six translation directions on the decontaminated FLORES+ devtest benchmark and the 868-pair held-out test split using sacrebleu 2.6.0. All reported metrics reproduce deterministically from the published checkpoint.

The Name

Taos Amrouche (1913–1976) sang exclusively in Kabyle and wrote in French. She published Jacinthe noire in 1947, the first novel by a Kabyle woman; from 1936 she collected and performed the Kabyle songs her mother had preserved; and she co-founded the Académie berbère in 1966. One of her albums is Chants sauvés de l'oubli — songs saved from oblivion.

A life spent carrying Kabyle into another language without ever surrendering it is what a translation model is for. Her mother, Fadhma Aït Mansour Amrouche, who from 1930 began writing down the songs and tales inherited from her ancestors, gives her name to the speech recognition model — the two halves of one family's work, split the way the two models are.

The naming is homage and implies no endorsement by anyone.

Citation

@software{agbalu_amrouche_2_1_3b_2026,
  title  = {Amrouche-2-1.3B: Bidirectional Kabyle, English, French, and Arabic Machine Translation},
  author = {AƔBALU},
  year   = {2026},
  url    = {https://huggingface.co/agbalu/Amrouche-2-1.3B},
  note   = {1,370,638,336 parameters; 38.22 macro chrF++ across 6 FLORES+ devtest directions; 52.31 macro chrF++ on held-out gold test}
}

Licence

Apache-2.0 on weights and code. Base model weights derive from facebook/nllb-200-distilled-1.3B (CC-BY-NC-4.0).

The training text composition over the curated parallel training corpus (1,089,930 directional examples) is:

Redistribution	Share
Permissive	88.8%
Unclear	10.9%
Share-alike	0.3%
Non-commercial	0.0%

Part of AƔBALU, a Kabyle and Amazigh corpus and model collection.