wandaa-kin-gpt: a Kinyarwanda language model trained from scratch

A small GPT trained from scratch on Kinyarwanda text (not a fine-tune of another model). It is a research project of VAF Ubwenge Tech, built step by step: start small, measure, fix the data, repeat. It continues text. It is not a chatbot yet.

Where we are now (honest status)

version	what changed	held-out score	EVAL bits/char
model.pt	first run, step 5000, textbook corpus	not comparable	2.382
checkpoint.pt	epoch 299	not comparable	1.981
v2	resumed from checkpoint.pt, fixed validation split	val 2.757	1.768
v3	corpus v3: 6.6M tokens (news, web, textbook), one apostrophe style	val 3.745 (new val set)	1.379
v4	v3 recipe without the textbook source	news/web val 3.941	1.423

Notes: val sets differ between versions, so compare columns only within the same row group. EVAL bits/char is measured on 10 fixed everyday Kinyarwanda sentences that are not in the training data. Lower is better.

Best model today: v3 (v3/best.pt). v4 is slightly better on news-style text and worse on textbook text.

Known limits: 9.7M parameters, 128-token context, 8000-token vocabulary, about 6.6M training tokens. Output is fluent in grammar and style but drifts in meaning. It does not answer questions.

Files

• v2/, v3/, v4/ : weights (best.pt) and, for v3, model.py (standalone loader) and README
• model.pt, checkpoint.pt : the original from-scratch runs
• tokenizer: map-boy/wandaa-kin-tokenizer (GPT-2 style BPE, vocab 8000)
• corpus: map-boy/wandaa-kin-traindata

Architecture

token + position embeddings (256-dim, 128 positions), 6 pre-norm transformer blocks, 8 heads of 32 dims, feed-forward 256-1024-256 with GELU, final LayerNorm, linear head to 8000 tokens.

Usage

from huggingface_hub import hf_hub_download as d
from transformers import GPT2Tokenizer
import torch, importlib.util
R = "map-boy/wandaa-kin-gpt"
spec = importlib.util.spec_from_file_location("model", d(R, "v3/model.py")); mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)
tk = GPT2Tokenizer(d("map-boy/wandaa-kin-tokenizer", "vocab.json"), d("map-boy/wandaa-kin-tokenizer", "merges.txt"))
m = mod.GPT(); m.load_state_dict(torch.load(d(R, "v3/best.pt"), map_location="cpu")["model"]); m.eval()
print(mod.generate(m, tk, "Kigali ni umujyi"))

Roadmap (what comes next, in order)

1. More data: find and clean free public Kinyarwanda text; target 50M+ tokens. Data is the main limit right now.
2. New tokenizer: train a 16k-32k BPE on the full corpus.
3. Scale the model: about 30-50M parameters, 256-token context, same fixed validation set and best-checkpoint saving.
4. Make it useful: fine-tune on question-answer and dialogue data (wandaa-kin-qa, kinyarwanda-dialogue-dataset) so it can answer.
5. Demo: publish in the Wandaa Demo Space with honest labels.

Every version keeps a fixed held-out set so progress is measured, not guessed.