Buckets:
Dholuo blind-spot evaluation
Q1: The blind spot
So as a dholuo speaker I have noticed that Large language Models mostly trained in the west have no understanding of such native languages
Method
- Model: google/gemma-3-4b-it (WHY YOU CHOSE IT)
- 10 prompt pairs (Dholuo + matched English) across four tiers: comprehension, generation, practical, codeswitch
- One sample per prompt, temperature 0.7, seed 0, max 150 new tokens
- Hand-scored 0/1/2 on: lang_ok, understanding, fidelity, usefulness (DEFINE EACH SCORE HERE)
- Separate translation probe with my own reference Dholuo (probe_scored.csv)
Findings
| tier | lang_ok | understanding | fidelity | usefulness |
|---|---|---|---|---|
| comprehension | 0.12 | 0.14 | 0.14 | 0.14 |
| generation | 0 | 0 | nan | nan |
Mean tokens per prompt: Dholuo 11.1 vs English 8.5
So every prompt I sent using the dholuo language to the LLM led to it confusing the language with Igbo
Q3: Path forward
The failure I most want to fix is that the model could not understand Dholuo. In 8 of 9 scored rows, understanding was 0, and the model typically named a different language (Igbo, Kalenjin) and then gave a confident, invented explanation, which can confuse and mislead real users. To address this, I would collect farming question-and-answer data written by Luo teachers, who write the language correctly, and fine-tune a small model on it. [YOUR LINE: why this would stop the bluffing, or what extra piece it needs.] Tokenization is a minor point: Dholuo prompts averaged 11.1 tokens against 8.5 for English (about 1.3x), so the gap in my data was small, but I would test an extended tokenizer later. To check that the fix worked, I would ask some simple questions in the languge just to see whether. This is speculative and untested.
Limitations Single small model, one sample per prompt, one scorer (me), one Dholuo variety (NAME IT).
Files
- data/prompts.csv: evaluation prompts
- data/results_full.csv: raw model outputs
- data/results_scored.csv: my hand scores and notes
- data/probe_scored.csv: translation probe against my reference Dholuo
- outputs/probe_outputs.txt: full raw probe answers
- notebook/dholuo_eval.ipynb: code used
- Total size
- 30.4 kB
- Files
- 6
- Last updated
- Oct 22
- Pre-warmed CDN
- US EU US EU