License: free for commercial and non-commercial use, with attribution. If you use this model, its outputs, or anything derived from it in a product, service, paper or dataset, you must credit "AletheiaAING" (https://huggingface.co/AletheiaAING) and keep this notice. This model is derived from Google Gemma and is also subject to the Gemma Terms of Use.
csw: 301M code-switching and idiom model (SFT)
The supervised (SFT) 301M-parameter csw model (fp32, model.safetensors, 1.2 GB), public. See BeardedMonster/csw-sft and BeardedMonster/csw-rl (private) for the experiments, GGUF files and the RL model. The 1B successors are AletheiaAING/gemma-3-1b-csw-best and AletheiaAING/gemma-3-0.8b-csw.
What the model does
Six tasks over 37 African languages: translation from English, code-switching (mix a second language into the base language the way native speakers do) and idiom-rich informal rewriting.
| Task | What you give it | What you get | Prompt |
|---|---|---|---|
eng_m |
English text | translation into the base language | Translate the text to Yoruba: TEXT (language name) |
eng_hm |
English text | code-switched text | <hm> <yor>+<eng>: TEXT |
m_hm |
text in the base language | code-switched text | <hm> <yor>+<eng>: TEXT |
eng_ood |
English text | idiom-rich informal text | <ood> <yor>: TEXT |
m_ood |
text in the base language | idiom-rich informal text | <ood> <yor>: TEXT |
hm_ood |
code-switched text | idiom-rich text that keeps the switching | <ood> <yor>+<eng>: TEXT |
What do m_text, hm and ood mean? (in simple words)
m_text(eng_min the task table): a plain translation of English into the language you want, no mixing. Prompt:Translate the text to Yoruba: TEXT(use the language name).hm(human-style mixing): the text mixes two languages the way real people talk, for example Yoruba with English words in the middle of a sentence. Prompt:<hm> <yor>+<eng>: TEXT(language codes in angle brackets).ood(idiom-rich style): the text is rewritten in a lively, everyday way, full of sayings, proverbs and comparisons, the way a chatty native speaker talks. Prompt:<ood> <yor>: TEXT.
eng_ means the input is English, m_ means the input is already in the base language, and hm_ood means the input is already mixed and you want the idiom-rich rewrite. The usage code below runs m_text, hm and ood on one sentence.
Input format (read this before writing prompts).
- A single user turn, no system prompt:
<bos><start_of_turn>user+ prompt +<end_of_turn>+<start_of_turn>model(tokenizer.apply_chat_templateproduces the same). - Translation (
eng_m) is the only prompt that uses the language name:Translate the text to Yoruba: TEXT, with the names in the last column of the table below (for examplePidginforpcm,Cameroon Pidginforwes,Kirundiforrun). Writing the code (Translate the text to yor) is out of distribution. - All other tasks use the three-letter code inside angle brackets (
<hm> <yor>+<eng>: TEXT,<ood> <yor>: TEXT), never the name. - Two spaces after every colon. The tags
<yor>,<hm>,<ood>,<dev>are single vocabulary tokens. - For
<hm>prompts andhm_ooda switch language is required, taken from the table (other pairs were not trained). The model was never trained without a switch language on hm prompts: leaving it out gives an unswitched, idiom-like output. - Outputs:
eng_m->yor <translation>(language code, one space, text); other tasks -><yor><eng><dev>device one | device two</dev> text. If the structure is wrong the output is invalid; resample.
Supported languages and switch languages
The table lists every base language with the switch languages this model was trained with and, in the last column, the language name to use in the translation prompt.
| Language | Code used in prompts | Switch languages (trained pairs) | Example prompt (code-switching) | Translation prompt (name, not code) |
|---|---|---|---|---|
| Afrikaans | afr |
English eng, Zulu zul |
<hm> <afr>+<eng>: TEXT |
Translate the text to Afrikaans: TEXT |
| Amharic | amh |
English eng, Arabic ara |
<hm> <amh>+<eng>: TEXT |
Translate the text to Amharic: TEXT |
| Bambara | bam |
French fra, Wolof wol |
<hm> <bam>+<fra>: TEXT |
Translate the text to Bambara: TEXT |
| Cameroonian Pidgin | wes |
French fra, Ewondo ewo, Duala dua |
<hm> <wes>+<fra>: TEXT |
Translate the text to Cameroon Pidgin: TEXT |
| Chichewa | nya |
English eng, Tumbuka tum |
<hm> <nya>+<eng>: TEXT |
Translate the text to Chichewa: TEXT |
| Ewe | ewe |
French fra, English eng |
<hm> <ewe>+<fra>: TEXT |
Translate the text to Ewe: TEXT |
| Fon | fon |
French fra, Yoruba yor |
<hm> <fon>+<fra>: TEXT |
Translate the text to Fon: TEXT |
| Fulah | ful |
French fra, English eng |
<hm> <ful>+<fra>: TEXT |
Translate the text to Fulah: TEXT |
| Ghanaian Pidgin | gpe |
Twi twi, Ewe ewe, Ga gaa |
<hm> <gpe>+<twi>: TEXT |
Translate the text to Ghanaian Pidgin: TEXT |
| Hausa | hau |
English eng, Arabic ara |
<hm> <hau>+<eng>: TEXT |
Translate the text to Hausa: TEXT |
| Igbo | ibo |
English eng, Nigerian Pidgin pcm |
<hm> <ibo>+<eng>: TEXT |
Translate the text to Igbo: TEXT |
| Kikuyu | kik |
English eng, Swahili swh |
<hm> <kik>+<eng>: TEXT |
Translate the text to Kikuyu: TEXT |
| Kinyarwanda | kin |
French fra, English eng, Swahili swh |
<hm> <kin>+<fra>: TEXT |
Translate the text to Kinyarwanda: TEXT |
| Kirundi | run |
French fra, Swahili swh |
<hm> <run>+<fra>: TEXT |
Translate the text to Kirundi: TEXT |
| Kongo | kon |
French fra, Lingala lin |
<hm> <kon>+<fra>: TEXT |
Translate the text to Kongo: TEXT |
| Lingala | lin |
French fra, Kongo kon |
<hm> <lin>+<fra>: TEXT |
Translate the text to Lingala: TEXT |
| Luganda | lug |
English eng, Swahili swh |
<hm> <lug>+<eng>: TEXT |
Translate the text to Luganda: TEXT |
| Luo | luo |
English eng, Swahili swh |
<hm> <luo>+<eng>: TEXT |
Translate the text to Luo: TEXT |
| Malagasy | mlg |
French fra, English eng |
<hm> <mlg>+<fra>: TEXT |
Translate the text to Malagasy: TEXT |
| Nigerian Pidgin | pcm |
Yoruba yor, Igbo ibo, Hausa hau |
<hm> <pcm>+<yor>: TEXT |
Translate the text to Pidgin: TEXT |
| North Ndebele | nde |
English eng, Shona sna |
<hm> <nde>+<eng>: TEXT |
Translate the text to North Ndebele: TEXT |
| Oromo | orm |
Amharic amh, English eng |
<hm> <orm>+<amh>: TEXT |
Translate the text to Oromo: TEXT |
| Sepedi | nso |
English eng, Afrikaans afr |
<hm> <nso>+<eng>: TEXT |
Translate the text to Sepedi: TEXT |
| Sesotho | sot |
English eng, Afrikaans afr |
<hm> <sot>+<eng>: TEXT |
Translate the text to Sesotho: TEXT |
| Setswana | tsn |
English eng, Afrikaans afr |
<hm> <tsn>+<eng>: TEXT |
Translate the text to Setswana: TEXT |
| Shona | sna |
English eng, North Ndebele nde |
<hm> <sna>+<eng>: TEXT |
Translate the text to Shona: TEXT |
| Somali | som |
Arabic ara, English eng, Italian ita, Amharic amh |
<hm> <som>+<ara>: TEXT |
Translate the text to Somali: TEXT |
| Swahili | swh |
English eng, Arabic ara, Sheng sheng |
<hm> <swh>+<eng>: TEXT |
Translate the text to Swahili: TEXT |
| Swati | ssw |
English eng, Zulu zul |
<hm> <ssw>+<eng>: TEXT |
Translate the text to Swati: TEXT |
| Tigrinya | tir |
Amharic amh, English eng |
<hm> <tir>+<amh>: TEXT |
Translate the text to Tigrinya: TEXT |
| Tsonga | tso |
English eng, Zulu zul |
<hm> <tso>+<eng>: TEXT |
Translate the text to Tsonga: TEXT |
| Tumbuka | tum |
English eng, Chichewa nya |
<hm> <tum>+<eng>: TEXT |
Translate the text to Tumbuka: TEXT |
| Twi | twi |
English eng, Ghanaian Pidgin gpe |
<hm> <twi>+<eng>: TEXT |
Translate the text to Twi: TEXT |
| Wolof | wol |
French fra, Arabic ara |
<hm> <wol>+<fra>: TEXT |
Translate the text to Wolof: TEXT |
| Xhosa | xho |
English eng, Afrikaans afr |
<hm> <xho>+<eng>: TEXT |
Translate the text to Xhosa: TEXT |
| Yoruba | yor |
English eng, Nigerian Pidgin pcm |
<hm> <yor>+<eng>: TEXT |
Translate the text to Yoruba: TEXT |
| Zulu | zul |
English eng, Afrikaans afr |
<hm> <zul>+<eng>: TEXT |
Translate the text to Zulu: TEXT |
What is in this repository
model.safetensors (fp32), tokenizer, config.json, chat template: the SFT model.
How to use it
# pip install torch transformers (tested with transformers 5.x; also served with vLLM 0.32)
import random, re, sys, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
NAMES = {'yor': 'Yoruba', 'hau': 'Hausa', 'ibo': 'Igbo', 'pcm': 'Pidgin', 'twi': 'Twi', 'ewe': 'Ewe', 'fon': 'Fon', 'wol': 'Wolof', 'swh': 'Swahili', 'kin': 'Kinyarwanda', 'som': 'Somali', 'orm': 'Oromo', 'amh': 'Amharic', 'tir': 'Tigrinya', 'lug': 'Luganda', 'lin': 'Lingala', 'zul': 'Zulu', 'xho': 'Xhosa', 'afr': 'Afrikaans', 'sna': 'Shona', 'nya': 'Chichewa', 'sot': 'Sesotho', 'tsn': 'Setswana', 'mlg': 'Malagasy', 'tum': 'Tumbuka', 'run': 'Kirundi', 'tso': 'Tsonga', 'nso': 'Sepedi', 'ssw': 'Swati', 'nde': 'North Ndebele', 'kik': 'Kikuyu', 'luo': 'Luo', 'bam': 'Bambara', 'kon': 'Kongo', 'ful': 'Fulah', 'wes': 'Cameroon Pidgin', 'gpe': 'Ghanaian Pidgin'} # base language code -> name used in the TRANSLATION prompt (not the tags)
# switch languages the model was trained with, per base language (anything else is out of distribution)
SWITCH = {'yor': ['eng', 'pcm'], 'hau': ['eng', 'ara'], 'ibo': ['eng', 'pcm'], 'pcm': ['yor', 'ibo', 'hau'], 'twi': ['eng', 'gpe'], 'ewe': ['fra', 'eng'], 'fon': ['fra', 'yor'], 'kin': ['fra', 'eng', 'swh'], 'sna': ['eng', 'nde'], 'zul': ['eng', 'afr'], 'xho': ['eng', 'afr'], 'swh': ['eng', 'ara', 'sheng'], 'orm': ['amh', 'eng'], 'som': ['ara', 'eng', 'ita', 'amh'], 'wol': ['fra', 'ara'], 'tsn': ['eng', 'afr'], 'nya': ['eng', 'tum'], 'afr': ['eng', 'zul'], 'sot': ['eng', 'afr'], 'lin': ['fra', 'kon'], 'run': ['fra', 'swh'], 'bam': ['fra', 'wol'], 'tum': ['eng', 'nya'], 'tso': ['eng', 'zul'], 'nso': ['eng', 'afr'], 'lug': ['eng', 'swh'], 'kik': ['eng', 'swh'], 'luo': ['eng', 'swh'], 'ssw': ['eng', 'zul'], 'nde': ['eng', 'sna'], 'kon': ['fra', 'lin'], 'ful': ['fra', 'eng'], 'mlg': ['fra', 'eng'], 'amh': ['eng', 'ara'], 'tir': ['amh', 'eng'], 'gpe': ['twi', 'ewe', 'gaa'], 'wes': ['fra', 'ewo', 'dua']}
TASKS = ("eng_m", "eng_hm", "m_hm", "eng_ood", "m_ood", "hm_ood")
def build_prompt(task, base, text, switch=None):
"""text = English (eng_*), base-language text (m_*) or code-switched text (hm_ood). Two spaces after every colon."""
if task == "eng_m": # translation: the language NAME, not the code
return f"Translate the text to {NAMES[base]}: {text}"
if task in ("eng_hm", "m_hm"):
switch = switch or random.choice(SWITCH[base])
return f"<hm> <{base}>+<{switch}>: {text}"
if task in ("eng_ood", "m_ood"):
return f"<ood> <{base}>: {text}"
if task == "hm_ood":
if switch is None: raise ValueError("hm_ood needs the switch language of the input text")
return f"<ood> <{base}>+<{switch}>: {text}"
raise ValueError(task)
_LANG = re.compile(r"^((?:<(?!dev>)[a-z]{3,5}>)+)"); _DEV = re.compile(r"<dev>(.*?)</dev>", re.S)
def parse(task, completion):
"""eng_m: 'yor <text>'. Others: '<yor><eng><dev>device one | device two</dev> <text>'."""
c = completion.strip()
if task == "eng_m":
m = re.match(r"^([a-z]{3,5}) (.*)$", c, re.S)
return {"ok": bool(m), "tags": [m.group(1)] if m else [], "devices": [], "text": m.group(2).strip() if m else c}
t = _LANG.match(c)
if not t: return {"ok": False, "tags": [], "devices": [], "text": c}
rest = c[t.end():].lstrip(); d = _DEV.match(rest)
if not d: return {"ok": False, "tags": re.findall(r"<([a-z]{3,5})>", t.group(1)), "devices": [], "text": rest}
text = rest[d.end():].strip()
return {"ok": bool(text), "tags": re.findall(r"<([a-z]{3,5})>", t.group(1)), "devices": [x.strip() for x in d.group(1).split("|") if x.strip()], "text": text}
def load(path):
tok = AutoTokenizer.from_pretrained(path); model = AutoModelForCausalLM.from_pretrained(path, dtype=torch.bfloat16).eval()
return tok, model.to("cuda" if torch.cuda.is_available() else "cpu")
def generate(tok, model, prompt, sample=False, max_new=700):
ids = tok(f"<bos><start_of_turn>user\n{prompt}<end_of_turn>\n<start_of_turn>model\n", add_special_tokens=False, return_tensors="pt").input_ids.to(model.device)
stop = [tok.convert_tokens_to_ids("<end_of_turn>"), tok.eos_token_id]
kw = dict(do_sample=True, temperature=0.7, top_p=0.95) if sample else dict(do_sample=False) # greedy is what all reported numbers use
with torch.no_grad(): out = model.generate(ids, max_new_tokens=max_new, eos_token_id=stop, pad_token_id=tok.pad_token_id or 0, **kw)
return tok.decode(out[0][ids.shape[1]:], skip_special_tokens=False).replace("<end_of_turn>", "").replace("<eos>", "").replace("<pad>", "").strip()
tok, model = load("AletheiaAING/csw # or a local download of this repository")
en = "My brother told me that the meeting about the new school fees will start at ten tomorrow, so we must leave early."
print(parse("eng_m", generate(tok, model, build_prompt("eng_m", "yor", en)))["text"]) # translation
print(parse("eng_hm", generate(tok, model, build_prompt("eng_hm", "yor", en, "eng")))["text"]) # code-switching
print(parse("eng_ood", generate(tok, model, build_prompt("eng_ood", "yor", en)))["text"]) # idiom-rich rewrite
Greedy decoding is what all reported numbers use. For several candidates use temperature 0.7, top-p 0.95; choosing among 8 candidates with the evaluator did not raise chrF++, so do not expect much from sampling.
Evaluation
Full test set (6,230 items, 34 languages, greedy decoding, chrF++ of the final text, macro over languages; same decoder for all rows). "RL best" is the highest-reward checkpoint that passed the per-task chrF guard, "RL latest" the final RL policy.
| Task | SFT | RL best | RL latest |
|---|---|---|---|
| eng_hm | 50.05 | 48.78 | 49.38 |
| m_hm | 55.58 | 54.02 | 54.57 |
| eng_ood | 35.69 | 36.47 | 36.44 |
| m_ood | 40.49 | 41.78 | 41.75 |
| hm_ood | 37.96 | 38.77 | 38.82 |
RL raised the idiom tasks (+0.8 to +1.3 chrF++) and lowered the code-switching tasks (-0.7, -1.0); output-format validity on code-switching rose from 0.949 / 0.954 to 0.996 / 0.999. An independent Gemini 3.8 Flash judge on the full test set scored the RL model slightly lower than the SFT model (-0.088 on a 1-5 scale, every task), so for quality-sensitive use the SFT model is the safer choice.
How it was made
A Gemma-3-270M-based decoder first fine-tuned for English-to-African translation, with language and format tags added to the vocabulary and 6 identity-initialised layers interleaved (24 layers, 301M parameters), then fine-tuned on the project's generated code-switching and idiom corpus (832,421 training rows; the text was generated with a 31B instruction model, filtered by rules and by two LLM judges on about 22% of rows, with judge-corrected copies added). RL (GRPO) used a learned reference-free evaluator (mmBERT-base with 27 rating heads) plus deterministic rule penalties on a fresh prompt pool. Not used: SSA-MTE and AfriMTE (benchmark only), NLLB-produced rows, TranslatePsy. Details and ablations: see the project paper.
Limitations and risks
- Outputs are machine-generated and not reviewed by native speakers. Training data and references were produced and graded with language models; they can contain unnatural phrasing, invented idioms or proverbs, wrong tone marks and meaning drift. Do not use the outputs where correctness matters without human review.
- Quality varies by language and by pair. Low-resource languages are weaker, a few languages have very few test items, and Fulfulde and Bambara had almost no
m_hmprompts in the RL stage; per-language numbers are noisy. - Rewriting, not translation. It can drop or change content and add devices that are not in the source. Pidgin outputs keep much English because English words are normal Pidgin.
- The learned reward could be gamed. In one run the evaluator's score rose while quality fell (the policy drifted toward copying its input); the released RL model uses rules against that, but quality was measured only with chrF++ and the evaluator, not with human raters.
- No added safety training beyond what the base model had, and no systematic check for offensive or harmful output.
License and attribution
Commercial use is allowed, with attribution to AletheiaAING (see the notice at the top of this card). Because this model is derived from Google Gemma, the Gemma Terms of Use and prohibited use policy also apply. Training data and quality labels used the outputs of other models (a 31B instruction model for generation, Gemini 3.8 Flash as judge); check the terms of those services for your own use. A full data inventory is in the accompanying project report; if you need the exact sources and licences, ask the authors.
- Downloads last month
- 83