project README

README.md

@@ -0,0 +1,50 @@
+---
+language: [rw]
+tags: [kinyarwanda, gpt, from-scratch, pytorch, text-generation]
+---
+# wandaa-kin-gpt: a Kinyarwanda language model trained from scratch
+
+A small GPT trained from scratch on Kinyarwanda text (not a fine-tune of another model). It is a research project of VAF Ubwenge Tech, built step by step: start small, measure, fix the data, repeat. It continues text. It is not a chatbot yet.
+
+## Where we are now (honest status)
+| version | what changed | held-out score | EVAL bits/char |
+|---|---|---|---|
+| model.pt | first run, step 5000, textbook corpus | not comparable | 2.382 |
+| checkpoint.pt | epoch 299 | not comparable | 1.981 |
+| v2 | resumed from checkpoint.pt, fixed validation split | val 2.757 | 1.768 |
+| v3 | corpus v3: 6.6M tokens (news, web, textbook), one apostrophe style | val 3.745 (new val set) | **1.379** |
+| v4 | v3 recipe without the textbook source | news/web val 3.941 | 1.423 |
+
+Notes: val sets differ between versions, so compare columns only within the same row group. EVAL bits/char is measured on 10 fixed everyday Kinyarwanda sentences that are not in the training data. Lower is better.
+
+Best model today: **v3** (v3/best.pt). v4 is slightly better on news-style text and worse on textbook text.
+
+Known limits: 9.7M parameters, 128-token context, 8000-token vocabulary, about 6.6M training tokens. Output is fluent in grammar and style but drifts in meaning. It does not answer questions.
+
+## Files
+- v2/, v3/, v4/ : weights (best.pt) and, for v3, model.py (standalone loader) and README
+- model.pt, checkpoint.pt : the original from-scratch runs
+- tokenizer: map-boy/wandaa-kin-tokenizer (GPT-2 style BPE, vocab 8000)
+- corpus: map-boy/wandaa-kin-traindata
+
+## Architecture
+token + position embeddings (256-dim, 128 positions), 6 pre-norm transformer blocks, 8 heads of 32 dims, feed-forward 256-1024-256 with GELU, final LayerNorm, linear head to 8000 tokens.
+
+## Usage
+    from huggingface_hub import hf_hub_download as d
+    from transformers import GPT2Tokenizer
+    import torch, importlib.util
+    R = "map-boy/wandaa-kin-gpt"
+    spec = importlib.util.spec_from_file_location("model", d(R, "v3/model.py")); mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)
+    tk = GPT2Tokenizer(d("map-boy/wandaa-kin-tokenizer", "vocab.json"), d("map-boy/wandaa-kin-tokenizer", "merges.txt"))
+    m = mod.GPT(); m.load_state_dict(torch.load(d(R, "v3/best.pt"), map_location="cpu")["model"]); m.eval()
+    print(mod.generate(m, tk, "Kigali ni umujyi"))
+
+## Roadmap (what comes next, in order)
+1. More data: find and clean free public Kinyarwanda text; target 50M+ tokens. Data is the main limit right now.
+2. New tokenizer: train a 16k-32k BPE on the full corpus.
+3. Scale the model: about 30-50M parameters, 256-token context, same fixed validation set and best-checkpoint saving.
+4. Make it useful: fine-tune on question-answer and dialogue data (wandaa-kin-qa, kinyarwanda-dialogue-dataset) so it can answer.
+5. Demo: publish in the Wandaa Demo Space with honest labels.
+
+Every version keeps a fixed held-out set so progress is measured, not guessed.