RiekoLM: Gemma 3 1B for Dholuo

RiekoLM ("Rieko") is a LoRA adapter for google/gemma-3-1b-pt. It makes the model a small assistant that speaks Dholuo (Luo of Kenya). It answers questions, translates between Dholuo, English and Swahili, and handles simple conversation in Dholuo.

Model Details

Model Description

The adapter was trained in two stages with QLoRA:

1. Continued pretraining (CPT) on Dholuo text, to improve the base model's knowledge of the language.
2. Supervised fine-tuning (SFT) on a rebalanced mix of instructions, chat and translation examples (SFT v2).

• Developed by: Benard Otieno (Bennerdo)
• Funded by: Benard Otieno
• Model type: Small causal language model (LoRA adapter)
• Language(s) (NLP): Dholuo (luo); English and Swahili for translation
• License: Gemma Terms of Use (inherited from the base model)
• Finetuned from model: google/gemma-3-1b-pt

Model Sources

• Repository: https://huggingface.co/bennerdo/rieko
• Base Model: google/gemma-3-1b-pt

Uses

Direct Use

• Answering questions and having short conversations in Dholuo.
• Translating between English and Dholuo, and between Swahili and Dholuo.
• Drafting and rewriting simple Dholuo text.
• Research on NLP for low-resource African languages.

Downstream Use

• A starting point for further fine-tuning on Dholuo tasks such as education, health information or customer support.
• A component in Dholuo chat or translation applications, with human review of the output.

Out-of-Scope Use

• Medical, legal or financial advice, or other high-stakes decisions.
• Treating the output as an authoritative reference for Dholuo grammar or spelling.
• Factual questions where an error causes harm. A 1B model often invents facts.
• Generating content that impersonates people, or deceptive or harmful content.

Bias, Risks, and Limitations

• Small model: at 1B parameters, the model has limited reasoning ability and makes factual errors and hallucinations.
• Synthetic data: part of the SFT data was generated by Gemini, and the identity and greeting examples were written from templates. A native Dholuo speaker (the author) reviewed this data and found almost all of it correct. A few unnatural phrasings or spelling errors may remain.
• Domain skew: the translation data is weighted towards religious (Bible) and OPUS text. Caps reduce this skew but do not remove it.
• Dialect: the training data favours Kenyan Nyanza Dholuo. The model is not tuned to other Luo varieties.
• Code-switching: the model sometimes mixes in English or Swahili.

Recommendations

Have a native speaker review the output before using it anywhere it matters. Don't rely on the model for facts without checking them.

How to Get Started with the Model

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base, adapter = "google/gemma-3-1b-pt", "bennerdo/rieko"
tok = AutoTokenizer.from_pretrained(adapter)
model = AutoModelForCausalLM.from_pretrained(base, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, adapter).eval()

msgs = [{"role": "user", "content": "An iluonga ni Akinyi, in iluongi ni ng'a?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=200, do_sample=True, temperature=0.3, top_p=0.9, top_k=40,
                     repetition_penalty=1.1, no_repeat_ngram_size=4,
                     eos_token_id=[tok.eos_token_id, tok.convert_tokens_to_ids("<end_of_turn>")])
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Training Details

Training Data

• Stage 1 (CPT): 338,912 decontaminated training rows (~5,404,734 words) and 2,000 validation rows. Sources include:
  • Parallel text extracts: OPUS parquet (134,108 rows kept), OPUS text (61,235), Bible (30,558), MAFAND (4,248 cpt + 113 text)
  • Monolingual corpora: Speech transcripts (84,269), ThinkKenya news/web (19,421), CLEAR synthetic text (5,000)
  • Cultural & grammatical texts: Wikipedia Luo (235), Lwanda Magere folklore (38), dictionary definitions (7)
  • English replay buffer: Teacher-generated English monolingual text (1,680 rows) to limit catastrophic forgetting
• Stage 2 (SFT):
  • SFT v1 (Trained & Evaluated): 166,650 rows (OPUS parallel 80k, Swahili–Luo 44.8k, Bible 30k, MAFAND 8.5k, Gemini instructions 925, dictionary 984, grammar 266, Lwanda Magere 104, English replay 1,118).
  • SFT v2 (Rebalanced mix): 52,398 rows with aggressive caps on translation sources (OPUS 20k, Bible 8k, Swahili–Luo 8k) and upsampled scarce conversational/cultural sources:
    • Gemini instructions: 925 unique pairs repeated ×5 (4,625 rows)
    • Identity & greeting templates: 68 unique pairs repeated ×5 (340 rows)
    • Multi-turn Gemini chat (gemini_chat_sft)
    • Grammar: 266 pairs repeated ×2 (532 rows)
    • Lwanda Magere: 104 pairs repeated ×3 (312 rows)
    • Dictionary: 984 rows
    • MAFAND-MT: 8,474 rows
    • English replay: 1,118 rows

Training Procedure

Preprocessing

Every source went through the same steps:

1. Decontamination: remove any row that matches a held-out evaluation item exactly or shares an 8-gram with one (checking against FLORES-200 dev/devtest, Belebele, MAFAND-test, and Swahili-Luo-test).
2. Global Deduplication on normalized MD5 hashes across all sources.
3. Alignment filter for translation pairs: drop pairs whose length ratio exceeds 2.5, and pairs where the source and target are identical.
4. Caps on high-volume translation sources to avoid religious-domain overfitting.
5. Upsampling of instruction, persona, grammar, and cultural folklore data.

Training Hyperparameters

• Hardware & Frameworks: Single NVIDIA RTX 6000 Blackwell GPU (48 GB VRAM); PyTorch, Transformers, PEFT 0.21.2, TRL (SFTTrainer), bitsandbytes 0.45+.
• Precision: bfloat16 mixed precision, 4-bit NormalFloat (NF4) base quantization with double quantization (QLoRA).
• LoRA Configuration:
  • Rank ($r$): 64
  • Alpha ($\alpha$): 128 (scaling factor = 2.0)
  • Dropout: 0.05
  • Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (all 18 layers)
  • Trainable parameters: 52,183,040 / 1,052,068,992 (4.96%)
• Stage 1 (CPT):
  • Objective: Causal language modeling with EOS token appended between packed documents
  • Max sequence length: 1,024 tokens
  • Batch size: 16 per device (effective batch size: 16, gradient accumulation: 1)
  • Peak learning rate: $1\times 10^{-4}$ with cosine decay schedule and 5% warmup
  • Optimizer: paged_adamw_8bit
  • Epochs: 2.0 (1,358 update steps)
  • Runtime: 1h 01m 16s (3,675.92 seconds) at 5.91 samples/s (0.37 steps/s)
  • Compute: $1.001\times 10^{17}$ FLOS
  • Metrics: Train loss: 3.0122 (started at 6.607); Validation loss: 2.6729; Validation Perplexity: 14.4819
• Stage 2 (SFT v1):
  • Starting weights: Stage 1 CPT final adapter (outputs/gemma_dholuo_cpt/final_adapter)
  • Objective: Instruction-tuning with TRL SFTTrainer and completion_only_loss=True (loss computed exclusively on model response tokens)
  • Chat template: Gemma format (<start_of_turn>user\n...<end_of_turn>\n<start_of_turn>model\n...<end_of_turn>)
  • Max sequence length: 1,024 tokens
  • Batch size: 16 per device (effective batch size: 16, gradient accumulation: 1)
  • Peak learning rate: $2\times 10^{-4}$ with cosine decay schedule and 5% warmup
  • Optimizer: paged_adamw_8bit
  • Epochs: 3.0 (31,248 update steps on 166,650 rows)
  • Runtime: 3h 28m 17s (12,497.10 seconds) at 40.01 samples/s (2.50 steps/s)
  • Compute: $3.547\times 10^{17}$ FLOS
  • Metrics: Train loss: 1.4136 (final step loss: 0.8868, train token accuracy: 78.29%); Validation loss on FLORES dev: 2.281 (eval token accuracy: 57.84%, eval entropy: 1.368)

Evaluation

Testing Data, Factors & Metrics

Testing Data

• FLORES-200 devtest, in both directions: English to Dholuo (eng_Latn → luo_Latn) and Dholuo to English.
• Belebele, Dholuo reading comprehension (luo_Latn).

All of this data was held out from training (see Preprocessing).

Metrics

• Translation: chrF++ and BLEU (sacrebleu). chrF++ is the main metric, because it handles Dholuo's rich morphology better than BLEU.
• Belebele: multiple-choice accuracy, scored by picking the answer option the model rates most likely.

Results

FLORES-200 devtest (n = 1,012). Belebele chance level is 0.25.

Model	eng→luo chrF++ / BLEU	luo→eng chrF++ / BLEU	Belebele acc
gemma-3-1b-pt (base)	10.27 / 0.46	14.55 / 1.82	0.2267
RiekoLM (SFT v1)	34.52 / 9.01	36.13 / 12.72	0.2278

Translation improves sharply: chrF++ rises by about 24 points from English to Dholuo and about 22 points from Dholuo to English. Belebele accuracy stays at chance level, so the adapter does not yet improve reading comprehension.

The two models are prompted differently. The base model has no chat template, so it gets plain completion prompts. The adapter gets the Gemma chat format. Keep this in mind when comparing the rows.

Environmental Impact

• Hardware Type: NVIDIA RTX 6000 (Blackwell architecture, 48 GB VRAM)
• Hours used: ~4.85 hours (CPT: 1.02h, SFT: 3.47h, Eval & Chat Synthesis: ~0.35h)
• Cloud Provider: molab (marimo reactive notebook platform)
• Compute Region: US Cloud Instance
• Total Compute FLOS: ~4.55e+17 FLOS (CPT: 1.00e+17 FLOS, SFT: 3.55e+17 FLOS)
• Carbon Emitted: Estimated ~0.55 kg CO2eq (based on ~300W average GPU load over 4.85 hours ≈ 1.455 kWh at 0.38 kg CO2/kWh grid emission factor)

Technical Specifications

Model Architecture and Objective

• Gemma 3 decoder-only transformer, 1B parameters, 18 layers, 256k vocabulary, with LoRA adapters (52.18M trainable parameters, 4.96% of total).
• Objective: Causal language modeling during CPT. During SFT, cross-entropy loss is computed exclusively on assistant response tokens (completion_only_loss=True).

Compute Infrastructure

• Hardware: 1× NVIDIA RTX 6000 Blackwell GPU (96 GB VRAM)
• Software: Linux, PyTorch 2.6+ (CUDA 12.8), Transformers >= 4.50, PEFT 0.21.2, TRL (SFTTrainer), bitsandbytes >= 0.45, sacrebleu

Citation

@misc{otieno2026riekolm,
  title  = {RiekoLM: A Small Dholuo Language Model},
  author = {Otieno, Benard},
  year   = {2026},
  url    = {https://huggingface.co/bennerdo/rieko},
  note   = {LoRA adapter for google/gemma-3-1b-pt}
}

Model Card Authors

Benard Otieno (Bennerdo)

Model Card Contact

• Hugging Face Profile: https://huggingface.co/bennerdo
• Model Repository: https://huggingface.co/bennerdo/rieko

Framework versions

• PEFT 0.21.2
• Transformers >= 4.50.0
• TRL
• PyTorch (CUDA 12.8)