You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

ai4ug/pearl-chat-350m-base

Foundational monolingual Luganda base checkpoint trained with Stage 1 broad pretraining on 374,129 documents (39,681,882 BPE tokens) plus Stage 1.5 WSD Uganda knowledge annealing.

Model details

  • Language: Luganda (lug, 1947 Standard Orthography, strictly monolingual)
  • Preset: pearl-v5-350m
  • Framework: JAX + Flax NNX (safetensors serialization, config.json)
  • Total parameters: 354,480,640 (337,703,424 transformer body parameters)
Hyperparameter / Architectural Spec Value
d_model (hidden size) 1024
n_layers (transformer blocks) 28
n_heads / n_kv_heads (GQA) 16 / 4 (head_dim=64)
d_ff (SwiGLU FFN width) 3072
Attention schedule Symmetric Hybrid 3:1 Sliding Window Attention (W=512, rope_theta=10000) interleaved with Global GQA every 4th layer (global_rope_theta=50000)
max_seq_len 2048
Vocabulary and embeddings 16,384 Luganda Byte-BPE (tie_word_embeddings=True)

Training data and curriculum

  1. Stage 1 (Broad Foundational Pretraining): Trained across 374,129 deduplicated monolingual Luganda documents (39,681,882 16,384-vocab BPE tokens) spanning lg.wikipedia.org, UBOS district and subregion prose, FinePDFs Luganda books, filtered FineWeb-2 and Glot500 prose, Sunbird SALT train and Uganda prompts, Bukedde and Ugandan news, and literary prose, with 0 detected 13-gram leaks to held-out evaluation splits.
  2. Stage 1.5 (WSD Uganda Knowledge Annealing): Token-balanced cosine decay (3.0e-4 to 3.0e-6) continuing directly from best Stage 1 AdamW moments with 4x upsampling on curated Ugandan 136-district, 15-subregion, encyclopedic, and 85 Engero z'Abaganda proverb documents plus 30% Stage 1 token replay.
  3. Stage 2 (Multi-Turn Grounded SFT): Length-bucketed supervised fine-tuning (sqrt-balanced loss, 1 to 4 turns) on 14,809 monolingual Luganda conversations with best-checkpoint selection strictly on pretrain-disjoint validation NLL (clean_val_nll).

Pretraining and annealing validation

  • Validation trajectory: Stage 1 best val_nll=3.483 (val_bpb=1.1254, ep=4); Stage 1.5 post-anneal val_nll=3.4723 (cloze=0.8344).
  • Downstream instruction checkpoint: Evaluated end-to-end on the derived ai4ug/pearl-chat-350m-it model card (eval_metrics_v5.json).

How to use (text completion and downstream fine-tuning)

from huggingface_hub import snapshot_download
from safetensors.flax import load_file
from tokenizers import Tokenizer

ckpt_dir = snapshot_download("ai4ug/pearl-chat-350m-base")
tokenizer = Tokenizer.from_file(f"{ckpt_dir}/tokenizer.json")
weights = load_file(f"{ckpt_dir}/model.safetensors")

Intended use and known limitations

  • Intended use: Luganda conversational assistance (ai4ug/pearl-chat-350m-it), encyclopedic Q&A on Ugandan geography, history, civics, agriculture, and proverbs, Luganda literacy/story continuation, and research into low-resource East African language modeling.
  • Scale and token budget: A 354M-parameter model trained on ~`40M` unique pretraining tokens cannot match frontier billion-parameter models on open-ended world trivia, symbolic mathematics, or ungrounded multi-hop reasoning.
  • Census and demographic temporal scope: All 136 Ugandan district entries include 2014 UBOS Census counts and 2020 regional city distinctions; 43 of 136 districts incorporate provisional NPHC 2024 population totals, while the remaining 93 districts cite 2014 figures.
  • Residual corpus imperfections: Automated web/news filtering leaves roughly 0.06% (71 documents) of residual non-Luganda lexical noise in pretraining web/news text, and approximately 52% of single-turn SFT first turns follow definitional encyclopedia templates.
  • Domain safety: Not intended as a standalone medical, clinical, legal, or financial authority. Health discussions grounded in Uganda Clinical Guidelines must not substitute for a licensed healthcare worker.

Ethical considerations and safety

  • Strictly monolingual (lug) by design; translation and non-Luganda prompt pairs were intentionally excluded to avoid cross-lingual calques.
  • Epistemic abstention examples ("Simanyi..." / explicit uncertainty on unknowable future or unverified facts) are included in SFT to reduce fabricated statistics.
  • Checkpoints are uploaded to private repositories (HUB_PRIVATE=1) by default prior to independent evaluation sign-off.

Citation

@misc{ai4ug2026pearlv5,
  title={Pearl Chat 350m: A Monolingual Luganda Foundational and Conversational Language Model in Pure JAX},
  author={AI for Uganda Initiative},
  year={2026},
  url={https://huggingface.co/ai4ug/pearl-chat-350m-base}
}
Downloads last month
22
Safetensors
Model size
0.4B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ai4ug/pearl-chat-350m-base

Finetunes
1 model