ai4ug/pearl-chat-350m-base
Foundational monolingual Luganda base checkpoint trained with Stage 1 broad pretraining on 374,129 documents (39,681,882 BPE tokens) plus Stage 1.5 WSD Uganda knowledge annealing.
Model details
- Language: Luganda (
lug, 1947 Standard Orthography, strictly monolingual) - Preset:
pearl-v5-350m - Framework: JAX + Flax NNX (
safetensorsserialization,config.json) - Total parameters:
354,480,640(337,703,424transformer body parameters)
| Hyperparameter / Architectural Spec | Value |
|---|---|
d_model (hidden size) |
1024 |
n_layers (transformer blocks) |
28 |
n_heads / n_kv_heads (GQA) |
16 / 4 (head_dim=64) |
d_ff (SwiGLU FFN width) |
3072 |
| Attention schedule | Symmetric Hybrid 3:1 Sliding Window Attention (W=512, rope_theta=10000) interleaved with Global GQA every 4th layer (global_rope_theta=50000) |
max_seq_len |
2048 |
| Vocabulary and embeddings | 16,384 Luganda Byte-BPE (tie_word_embeddings=True) |
Training data and curriculum
- Stage 1 (Broad Foundational Pretraining): Trained across
374,129deduplicated monolingual Luganda documents (39,681,88216,384-vocab BPE tokens) spanninglg.wikipedia.org, UBOS district and subregion prose, FinePDFs Luganda books, filtered FineWeb-2 and Glot500 prose, Sunbird SALT train and Uganda prompts, Bukedde and Ugandan news, and literary prose, with0detected 13-gram leaks to held-out evaluation splits. - Stage 1.5 (WSD Uganda Knowledge Annealing): Token-balanced cosine decay (
3.0e-4to3.0e-6) continuing directly from best Stage 1 AdamW moments with4xupsampling on curated Ugandan136-district,15-subregion, encyclopedic, and85Engero z'Abaganda proverb documents plus30%Stage 1 token replay. - Stage 2 (Multi-Turn Grounded SFT): Length-bucketed supervised fine-tuning (
sqrt-balanced loss,1to4turns) on14,809monolingual Luganda conversations with best-checkpoint selection strictly on pretrain-disjoint validation NLL (clean_val_nll).
Pretraining and annealing validation
- Validation trajectory: Stage 1 best val_nll=3.483 (val_bpb=1.1254, ep=4); Stage 1.5 post-anneal val_nll=3.4723 (cloze=0.8344).
- Downstream instruction checkpoint: Evaluated end-to-end on the derived
ai4ug/pearl-chat-350m-itmodel card (eval_metrics_v5.json).
How to use (text completion and downstream fine-tuning)
from huggingface_hub import snapshot_download
from safetensors.flax import load_file
from tokenizers import Tokenizer
ckpt_dir = snapshot_download("ai4ug/pearl-chat-350m-base")
tokenizer = Tokenizer.from_file(f"{ckpt_dir}/tokenizer.json")
weights = load_file(f"{ckpt_dir}/model.safetensors")
Intended use and known limitations
- Intended use: Luganda conversational assistance (
ai4ug/pearl-chat-350m-it), encyclopedic Q&A on Ugandan geography, history, civics, agriculture, and proverbs, Luganda literacy/story continuation, and research into low-resource East African language modeling. - Scale and token budget: A
354M-parameter model trained on ~`40M` unique pretraining tokens cannot match frontier billion-parameter models on open-ended world trivia, symbolic mathematics, or ungrounded multi-hop reasoning. - Census and demographic temporal scope: All
136Ugandan district entries include 2014 UBOS Census counts and 2020 regional city distinctions;43of136districts incorporate provisional NPHC 2024 population totals, while the remaining93districts cite 2014 figures. - Residual corpus imperfections: Automated web/news filtering leaves roughly
0.06%(71documents) of residual non-Luganda lexical noise in pretraining web/news text, and approximately52%of single-turn SFT first turns follow definitional encyclopedia templates. - Domain safety: Not intended as a standalone medical, clinical, legal, or financial authority. Health discussions grounded in Uganda Clinical Guidelines must not substitute for a licensed healthcare worker.
Ethical considerations and safety
- Strictly monolingual (
lug) by design; translation and non-Luganda prompt pairs were intentionally excluded to avoid cross-lingual calques. - Epistemic abstention examples (
"Simanyi..."/ explicit uncertainty on unknowable future or unverified facts) are included in SFT to reduce fabricated statistics. - Checkpoints are uploaded to private repositories (
HUB_PRIVATE=1) by default prior to independent evaluation sign-off.
Citation
@misc{ai4ug2026pearlv5,
title={Pearl Chat 350m: A Monolingual Luganda Foundational and Conversational Language Model in Pure JAX},
author={AI for Uganda Initiative},
year={2026},
url={https://huggingface.co/ai4ug/pearl-chat-350m-base}
}
- Downloads last month
- 22