gemma-science-translate (final run final2, step 401): English to 77 African languages

The final run checkpoint of the paper draft Small but Sabi: RL-final, then a short science-domain SFT stage on AfriScience-MT, interpolated with RL-final (A2W, alpha 0.7), then 401 steps of GRPO (final2, planned 700, wound down after step 350). Pinned revisions: branch final2/step401 and tag final2-step401 (commit 8bb1a029f4402d9529f7309ad67750fb41ee5905), branch latest (same weights, model.safetensors sha256 1bd2e310f146e233aac0c0be339329321f30823dfe05d95c39f561cfbd6589c5; latest has one extra commit that replaces the earlier README with this card). The branch main of this repository still holds the SFT-stage models under sft/ (A, A2W, A2W_alpha0.7, B_v1) and was not changed. Only the last step was kept; no mid-run checkpoint exists. A GGUF Q8_0 build for CPUs is in BeardedMonster/gemma-science-translate-GGUF.

Repository layout (read this first)

Revision	What it is
main	Front page (this card, LICENSE_DRAFT.md) and the SFT-stage models under sft/ (A, A2W, A2W_alpha0.7, B_v1); these are the supervised start models of the science stage, not the final model.
latest	Final run checkpoint (final2, step 401): load with revision="latest".
final2/step401 and tag final2-step401	Same final weights, pinned (commit 8bb1a029f4402d9529f7309ad67750fb41ee5905); the paper cites this commit.

Quick start: AutoModelForCausalLM.from_pretrained("BeardedMonster/gemma-science-translate", revision="latest"); prompt <bos><start_of_turn>user\nTranslate the text to {Language}: {english}<end_of_turn>\n<start_of_turn>model\n (two spaces after the colon); the answer is {iso} {translation}. CPU build: BeardedMonster/gemma-science-translate-GGUF.

Training data classes seen by this model (read before any licence decision)

This checkpoint is the one whose lineage matters for the licence.

• SFT data class: the SFT data of SFT-final (public parallel corpora regrouped; see the data audit in the project report weak_language_data_2026-10-22 section 5). Documented in that audit: repo-level scan classes commercial_ok / unknown (used and flagged) / non-commercial (skipped); 213 of 292 scanned repositories had no stated licence and were used with a flag; two owner-approved exceptions are recorded (a Twi parallel set declared CC-BY-NC-4.0 and XL-Sum rows inside the naija sets, CC-BY-NC-SA); about 58% of the replay tokens are Jehovah's-Witnesses-style text whose publisher copyright is not cleared. The owner states that a separate verification found no non-commercial data in the SFT mixture: TODO(owner: attach that verification so the two statements can be reconciled before any commercial licence is declared).
• AfriScience-MT training split (six languages: amh, hau, lug, nso, yor, zul; 5,792 training sentences per language, simplified science text; used in the science SFT stage and, as prompts, in the final RL run; the AfriScience dev split was used for checkpoint selection, so it is not independent; the test split was not used for any decision).
• Science-stage SFT mixture (about 320 steps at 3e-6): replay of the general human references, self-distilled anchor examples, the best translations saved by the RL archive (model output), and long-document distillation (documents translated sentence by sentence by the RL model, best-of-four chosen by the QE ensemble, joined; 13.5% of the mixture).
• RL prompts of the final run: 70% science (AfriScience-MT train side, PubMed, arXiv, LibreTexts) and 30% general pools (English Wikipedia, FineWeb-Edu).
• NLLB-200 3.3B pseudo-reference signal (CC-BY-NC-4.0 model, non-commercial): used as a REWARD signal only, for five languages (Kabuverdianu kea, Kabiye kbp, Nuer nus, Bambara bam, Kabyle kab; weight 0.23-0.35). No NLLB text is a training target of the weights (the pseudo-references enter only the scalar reward), but the policy was optimised against them. NLLB outputs were also tested as teacher data in an SFT mixture (C3) and not adopted; they are not in this lineage's SFT stage as far as the paper states (TODO(owner): confirm).
• Learned reward scorers: AfriCOMET-QE and SSA-COMET-MTL (no reference) as an ensemble; GlotLID.
• No TranslatePsy data.

If the owner needs a model without NLLB signal and without AfriScience-MT data, use RL-final (BeardedMonster/gemma-3-270m-translate-rl2c, branch latest) or SFT-final (AletheiaAING/gemma3-270m-translate), not this one.

Training stages and data

Supervised fine-tuning data (existing public parallel datasets, regrouped, not newly translated). 82% of the original pairs are single sentences; consecutive pairs of the same source and language pair were concatenated into windows in four length bins (b64/b256/b512/b1024 by AfroXLM-R tokens), packed into 4,096-token packs, one language per pack, block-diagonal attention. Final build: 11,442,172 windows, 538,480 training packs, 2.204 B tokens (1.318 B target tokens), 77 target languages, English-to-X only. Pairs were vetted with AfriCOMET-QE where the language is covered (bottom 15% per language and QE below 0.20 removed) and by rules otherwise; text is Unicode NFC. The corpus is dominated by religious text for low-resource languages (about 58% of replay tokens). A large part comes from repositories with unstated or non-commercial licences (paper Sections 3.4 and 11; 213 of 292 scanned repositories had no stated licence).

SFT in two stages, one set of weights. Stage A: AdamW, lr 5e-5, 3% warm-up, cosine decay, betas (0.9, 0.95), weight decay 0.01, clip 1.0, fp32 master weights with bf16 autocast, 131,072 tokens per step, languages sampled proportional to the square root of their pack count; stopped at about step 4,750. Stage B (fresh optimiser): Muon, lr 4e-5, UniMax-style language budget of 3 B tokens (every pack of every language used 1-2 times, 20 priority languages with 4x weight). Final checkpoint = step 27,638 (27,638 steps x 131,072 tokens = 3.62 B training tokens), macro validation loss 1.2654 over the 77 languages. Base: google/gemma-3-270m-it (18 layers, hidden size 640, 262,144-token vocabulary, about 270 M parameters, full fine-tuning, no adapters).

General GRPO of RL-final, phases 1, 2, 2b, 2c (step 1,433), the start of everything below (phase 2c restarted from the step-488 model of the diverged phase 2b; about 900,000 sampled completions, 40.5 h on one GPU). One update per sampled batch, groups of 16 completions at temperature 1, KL penalty (beta 0.05) to the reference, lr 1e-6 to 3e-6 (2e-6 in 2c) with a KL guard, length curriculum b64 -> b256 -> b512 -> b1024 -> b2048 -> mixture. Prompts: English windows of consecutive sentences from English Wikipedia and FineWeb-Edu, filtered; 77 language files, draw weights favouring weak languages. Reward (reference-free): AfriCOMET-QE only as the learned metric (for the 42 validated and, blended 50/50, the 12 unvalidated languages), consensus chrF++ among the valid completions of the group for the 23 uncovered languages (and blended for the 12), per-language piecewise-linear shaping calibrated on human references (calibration only; no target-side reference is a training target), hard-failure floor (-2) for wrong/missing tag, wrong language by GlotLID, copied source, loops, unfinished, plus length/lexical/diacritic penalties and sentence-level chunk scoring. See the paper, Sections 4.1-4.5 and Appendix C.

Science stage and interpolation. Short SFT on AfriScience-MT plus replay/anchor/archive/long-document data (see data classes), then weight interpolation theta = (1 - alpha) theta_RL + alpha theta_SFT-stage with alpha = 0.7 (model A2W), chosen by guardrails (best AfriScience dev chrF++ such that the general dev set, every weak-language group and the finished rate do not get worse).

Final run (final2). Start A2W (alpha 0.7), KL reference = start model, lr 1e-6 (20 warm-up steps), KL coefficient 0.1, groups of 16 with one greedy member (advantage capped at 3, scaled 0.3), KL guard, QE ensemble reward (AfriCOMET-QE + SSA-COMET-MTL, min rule when they disagree), anchors v2/v3 (copy, entity loss, padding, repetition), long-document reward v2, NLLB signal for five languages, language floor and static draw weights. 401 steps, 26.2 h of step time; planned restart at step 150 and two OOM kills with resume from step 250 (the trajectory is not a single clean run). Seed 1234, one run.

Evaluation summary (numbers of the paper draft; greedy decoding, up to 768 new tokens)

Final run vs	RL-final	start A2W
FLORES+ no_pair, 45 languages, chrF++	34.31 vs 34.08 (+0.23 [0.19, 0.27])	+0.15 [0.12, 0.17]
AfriCOMET on FLORES+	-0.0016 [-0.0022, -0.0011]	+0.0034
NTREX-128 / MAFAND-MT / SmolSent chrF++	+0.17 / +0.16 / +0.20
AfriScience-MT test, sentence chrF++ (6 languages)	+2.40 [2.25, 2.56]; sentence chrF 50.2 (metric of the published tables)	+0.17 [0.07, 0.27]
AfriScience-MT test, whole paper in one pass, chrF	46.05 vs 33.05 (+13.0)	+0.28 [-1.18, 1.73] (inside noise)
Papers finished (test)	98.2% vs 72.0%	98.2% vs 94.0%

On the general benchmarks the final run is level with RL-final ("the honest summary is level"); the gain over RL-final on science text comes from the science-stage SFT data, not from the 401 RL steps. Dev sets of the run (steps 0 -> 401): AfriScience dev sentence chrF++ 48.79 -> 48.92, general dev (19 languages) 41.97 -> 42.04, weak-language dev (13 languages) 24.90 -> 25.28, paper-level finished 0.927 -> 0.967. Evaluation reward 192 held-out prompts: -0.560 at step 401; b2048 prompts stayed near the failure floor (-1.92): RL with this reward did not teach 2,048-token windows. The papers it translates are shorter than the references (median length ratio 0.84; 45% below 0.8), i.e. it finishes and writes the language but omits as much as A2W. The 2,048-token long-context benchmark was not run for this checkpoint (it was run on the step-250 checkpoint: 87% of source sentences covered at about 1,024 tokens, 70% at about 2,048). Compared with published AfriScience-MT numbers (different pipelines, trained on the split): sentence chrF 50.2, level with Gemma2-9B LoRA (50.3), below fine-tuned NLLB (51.7-52.3) and GPT-5.4 10-shot (53.3). CPU (Q8_0 GGUF of this checkpoint): macro chrF++ 34.96 on 1,170 FLORES+ sentences (GPU bf16 34.98), 2.0 sentences/s on four server cores.

Prompt format (exact)

Use the chat template of the base model and ONE user turn of exactly this form (two spaces after the colon):

<bos><start_of_turn>user
Translate the text to {Language}:  {english}<end_of_turn>
<start_of_turn>model

The model answers {iso} {translation}<end_of_turn>: the three-letter tag of the target language, one space, then the translation. Example: prompt "Translate the text to Yoruba: Good morning, how are you today?" -> yor E kaaro, bawo ni o se wa loni? (illustrative). Remove the tag in post-processing. {Language} must be one of the names in the table below, written exactly as there. The model translates from English only; it is not a chat model.

Recommended decoding: greedy, max_new_tokens 256 for sentences (up to about 1,024 for paragraphs, 7,168 are used in long-document evaluation); make sure <end_of_turn> (id 106) is among the stop tokens.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
repo = "BeardedMonster/gemma-science-translate"   # revision: final2/step401
tok = AutoTokenizer.from_pretrained(repo, revision="final2/step401")
model = AutoModelForCausalLM.from_pretrained(repo, revision="final2/step401", torch_dtype=torch.bfloat16, device_map="auto")
msgs = [{"role": "user", "content": "Translate the text to Yoruba:  Good morning, how are you today?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**ids, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))   # "yor ..." -> drop the tag

Languages

85 rows in the language table; 77 are target languages of SFT and of the RL phases. The 8 marked below are table entries without SFT data (zero-shot; only Kanuri and North Ndebele are scored on a benchmark, the model does not write the tag for Kanuri). Reward groups (paper Section 3.2): 42 languages with a validated QE metric, 12 with an unvalidated metric (blended with a consensus signal), 23 not covered by AfriCOMET (consensus signal only). 19 table rows have no human-reference benchmark.

tag	name in the instruction	note
yor	Yoruba
hau	Hausa
ibo	Igbo
pcm	Pidgin
efi	Efik
twi	Twi
ewe	Ewe
fon	Fon
wol	Wolof
swh	Swahili
kin	Kinyarwanda
som	Somali
orm	Oromo
amh	Amharic
tir	Tigrinya
lug	Luganda
lin	Lingala
zul	Zulu
xho	Xhosa
afr	Afrikaans
sna	Shona
nya	Chichewa
sot	Sesotho
tsn	Setswana
mlg	Malagasy
ibb	Ibibio
urh	Urhobo
bem	Bemba
tum	Tumbuka
run	Kirundi
ven	Venda
tso	Tsonga
nso	Sepedi
ssw	Swati
nbl	South Ndebele
nde	North Ndebele	in the language table but NO SFT data (zero-shot, not RL-trained)
sag	Sango
tiv	Tiv
umb	Umbundu
kmb	Kimbundu
mos	Mossi
kik	Kikuyu
luo	Luo
kam	Kamba
bin	Edo
dag	Dagbani
oke	Okpe
shi	Tachelhit
dyu	Dyula
bam	Bambara
kbp	Kabiye
bbj	Ghomala
kon	Kongo
kea	Kabuverdianu
fat	Fante	in the language table but NO SFT data (zero-shot, not RL-trained)
knc	Kanuri	in the language table but NO SFT data (zero-shot, not RL-trained)
gaa	Ga	in the language table but NO SFT data (zero-shot, not RL-trained)
kab	Kabyle
nus	Nuer
swc	Congo Swahili
lua	Luba-Kasai
kqn	Kaonde
lun	Lunda
iso	Isoko
loz	Lozi
wal	Wolaytta
ach	Acholi
tsc	Tswa
vmw	Makhuwa
kua	Kuanyama
seh	Sena
nyn	Nyankore
her	Herero
yao	Yao
sid	Sidamo
ndc	Ndau
men	Mende	in the language table but NO SFT data (zero-shot, not RL-trained)
kri	Krio
luy	Luyia	in the language table but NO SFT data (zero-shot, not RL-trained)
ish	Esan
dua	Duala
bas	Basaa
wes	Cameroon Pidgin
tzm	Tamazight	in the language table but NO SFT data (zero-shot, not RL-trained)
ful	Fulah	in the language table but NO SFT data (zero-shot, not RL-trained)

Known limitations

• Omission and looping on long inputs. A 270M model drops sentences, repeats itself and sometimes copies English. Native whole-document translation works only for the science-stage checkpoints up to about 2,048 tokens and omits 13-30% of the source sentences (paper Section 6.10, measured on the step-250 checkpoint of the final run, not on this one where stated); unfinished/looping completions occur at the 768-1,024 token budget in some languages. Inputs above about 500 tokens were rare in the SFT data.
• Weak languages. The weak language groups score about 15-27 chrF++ on FLORES+ (group means); Ethiopic-script languages are far behind NLLB-200 (Amharic 19.6 vs 37.4, Tigrinya 13.2 vs 22.6 on a 1,170-sentence CPU test set). Languages written with many tone marks lose diacritics or invent words. The tokenizer fragments Yoruba/Tigrinya about 1.5-1.7 times more than Swahili, so these cost more time per sentence.
• Metric coverage. All quality statements are automatic metrics (chrF++, spBLEU, AfriCOMET, SSA-COMET), no human evaluation of the outputs. AfriCOMET-QE was the RL reward model, so AfriCOMET is not an independent judge; 23 languages have no validated learned metric; 19 table rows have no human-reference benchmark; the SFT text contains the English side of 95% of FLORES+ devtest sentences (we report no_pair and clean subsets).
• Register. The SFT text for low-resource languages is dominated by religious text (about 58% of replay tokens, 75-93% for most weak languages); the publisher copyright of the Jehovah's Witnesses text is not cleared.
• One run per configuration, no seed variance; the intervals in the paper cover sentence noise only.
• No safety or toxicity evaluation. Output may reproduce biases of the training data and of the base model.

Intended use and out-of-scope use

Intended: research on small multilingual translation models and on reward-hardened RL; drafts of English-to-African-language translation for low-stakes use with review by a speaker of the target language; offline/CPU experiments (see the GGUF repository). Out of scope: legal, medical, safety-critical or official translation without native-speaker review; translation into English or between African languages (not trained); languages outside the table; judging the quality of a language the metrics do not cover; use as a general chat model; generating data for a purpose not allowed by the licence terms that the owner will fix (see License).

Ethics

The model builds on data produced by African research communities (Masakhane, SALT, OLDI, MAFAND-MT, IbomMT, AfriScience-MT and others). No native-speaker reviewers took part in the evaluation. For weak languages the output is frequently wrong, and wrong translations in health, legal or safety contexts can cause harm; we publish per-language scores in the paper so that users see where it fails. Some SFT sources are non-commercial or of unstated licence.

License

DRAFT, not in force, not reviewed by counsel. Proposed terms: the Gemma Terms of Use and Prohibited Use Policy (https://ai.google.dev/gemma/terms) plus the SabiYarn Open-Output Addendum v0.1 in LICENSE_DRAFT.md: commercial use, modification and distribution are allowed; anyone who distributes data generated with the model, or uses such data to train or evaluate a model they distribute or serve, must publish that data publicly under CC-BY-4.0 or more permissive terms (with carve-outs for privacy law and third-party source licences). The repository is PRIVATE; nothing here grants any right until the owner publishes the licence.

Owner statement (2026-10-24): this checkpoint, although its RL used the NLLB-200 3.3B output (CC-BY-NC-4.0 model) as a reward-only signal for five languages, can support a commercial licence. This is the owner's position, not a legal conclusion: counsel should review it, together with the data-licence points in the training-data section above. The code is AGPL-3.0-or-later (file headers in the project repository).

Citation

@misc{sabiyarn2026,
  title={Small but Sabi: Reward-Hardened Reinforcement Learning for a 270M-Parameter Translator into 77 African Languages},
  author={TODO(owner: authors)},
  year={2026},
  note={Preprint draft}
}

Takedown and corrections

Requests to remove or correct any item are handled through the Community tab (Discussions) of this repository. Maintainers respond, remove the affected rows or files in the next commit and say so in the changelog of this card. Do not send personal data in a public discussion that you want removed; say that you want to share it privately and the maintainers will arrange a channel. Contributions of corrections follow the guide corrections/README.md of the dataset repository BeardedMonster/SabiYarn-MT-benchmark-translations.

Changelog

• 2026-10-24: model card written (agent D1) from the paper draft "Small but Sabi" (Sections 3, 4, 6, 12, 13, appendices). Weights unchanged.