Ha-Te-21/tigrinya-Gemma_4_E4B

Tigrinya Gemma 4 E4B

A Tigrinya (ትግርኛ) language model, adapted from google/gemma-4-E4B-it by two stages of continued pre-training followed by instruction tuning. Built to run offline on consumer hardware; a four-bit build for CPU inference is at Ha-Te-21/sft_v1_q4km.

What is in this repository

Path	Contents
stage1_final/	after continued pre-training, Stage 1
stage2_final/	after continued pre-training, Stage 2
SFT/final/	the model to use — after instruction tuning
stage1_ckpts/, stage2_ckpts/, SFT/checkpoint-*	intermediate checkpoints

Weights are bfloat16 safetensors. SFT/final is the model all reported results refer to.

Intended use

This model is at its best when the answer is in the text you give it: summarisation, extraction, and question answering over a supplied passage. It is not an oracle. Asked open questions without supporting text it produces fluent Tigrinya that is frequently wrong — asked to name the world's tallest mountain it named a country. See Limitations.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

mid = "Ha-Te-21/tigrinya-Gemma_4_E4B"
tok = AutoTokenizer.from_pretrained(mid, subfolder="SFT/final")
model = AutoModelForCausalLM.from_pretrained(mid, subfolder="SFT/final",
                                             device_map="auto")

msgs = [{"role": "user", "content": "ናይ ትግርኛ ሕቶኻ ኣብዚ ጽሓፍ።"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True,
                              return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=512, temperature=0.6,
                     top_p=0.9, repeat_penalty=1.15)
print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))

The chat template is required. Sent raw text without the Gemma turn format the model runs as a base model in continuation mode and performs far worse than it should. This was the single largest source of apparent quality loss during development.

Training

Corpus. 144.6 million tokens from 21 native Tigrinya sources (555.3 MB) and 5 machine-translated ones (1,299 MB). Cleaning addressed two problems specific to this setting: optical character recognition on Ge'ez script, which emits Unicode Private Use Area code points invisible to language identification, and Amharic contamination, which no script-level filter detects — a trained drift detector rejected 1,771 records. Translated text was round-trip verified with NLLB and kept only above a LaBSE cosine of 0.65. Dataset: Ha-Te-21/tigrinya-corpus.

Stage	Steps	Data	Learning rate	Eval loss
CPT Stage 1	2,195	native + translated, 50/50 by characters	2e-5 cosine, 3% warm-up	1.9272 → 1.6072
CPT Stage 2	1,114	native only	5e-6 cosine, no warm-up	1.7334 → 1.7154
Instruction tuning	3,502	56,024 records, 2 epochs	1e-5 linear, 5% warm-up	0.7795

The two continued-pre-training stages are scored on different validation splits and their losses are not comparable with each other.

Optimiser: Adafactor with scale_parameter=False, relative_step=False, warmup_init=False. These are not the library defaults; with the defaults Adafactor computes its own learning rate and silently ignores the one supplied. Both stages are full fine-tunes — the available parameter-efficient implementations do not support this architecture's linear layer class. Vision and audio towers frozen; per-layer embeddings trainable.

Instruction set: 57,198 records assembled across five groups (extractive QA, multiple choice, passage-grounded tasks, translation pairs, conversational), 56,024 after the held-out split. Translation pairs were capped at 10,000 of 98,000 available, so that they did not become two-thirds of the mixture.

Evaluation

865 held-out multiple-choice questions, ten options each, 10% random baseline. Splits were audited along three leakage routes, including the case where a passage used to generate an instruction item also appears in the pre-training corpus.

Arm	Generative	Unparseable	Likelihood
Base model	0.12%	852	24.51%
Base + CPT	0.00%	864	27.40%
Base + SFT	33.06%	0	35.61%
Full pipeline (this model)	37.23%	0	38.15%

95% Wilson interval on 37.23%: [34.07, 40.50]. Held-out perplexity (Stage 1): 4.989.

Instruction tuning contributes 11.10 points on the likelihood metric at p < 0.0001. Continued pre-training contributes 2.89 points at p = 0.0725, which does not reach significance; on the generative metric the same comparison is 4.16 points at p = 0.0046. Significance by exact McNemar test on discordant pairs.

Limitations

• Not a source of facts. Hallucination rate is unmeasured. Use it on supplied text, not as a knowledge base.
• The evaluation set is translated from MMLU-Pro. It measures knowledge expressed in Tigrinya, not knowledge of Tigrinya or of the communities that speak it.
• No comparison against other Tigrinya models. None publishes held-out results and no shared Tigrinya benchmark exists. The evaluation splits are released so that comparison becomes possible.
• Part of the training corpus is machine translated. The round-trip filter removes translations that fail to recover their source, but a fluent mistranslation passes.
• Orthographic variation is not normalised, so some vocabulary is spread across spelling variants.
• The mixture ratio was not tuned. The translation-pair cap is justified by proportion, not by experiment.

Licence

This model is a derivative of Gemma and its use and redistribution are governed by the Gemma Terms of Use. The code that produced it is MIT licensed.

Citation

@article{teweldemedhn2026tigrinya,
  title  = {An offline Tigrinya language model: corpus construction,
            staged adaptation, and deployment-aware evaluation},
  author = {Teweldemedhn Hagos, Atsbaha and Demewez, Guesh Dagnew and
            Teklu Fkadu, Desta and Gebreslassie Gebremariam, Abrhaley},
  year   = {2026},
  note   = {Mekelle University}
}

Code and measurement files: https://github.com/Ats-Tewe/tigrinya-offline-llm