Buckets:
| # Dholuo blind-spot evaluation | |
| ## Q1: The blind spot | |
| So as a dholuo speaker I have noticed that Large language Models mostly trained in the west have no understanding of such native languages | |
| ## Method | |
| - Model: google/gemma-3-4b-it (WHY YOU CHOSE IT) | |
| - 10 prompt pairs (Dholuo + matched English) across four tiers: comprehension, generation, practical, codeswitch | |
| - One sample per prompt, temperature 0.7, seed 0, max 150 new tokens | |
| - Hand-scored 0/1/2 on: lang_ok, understanding, fidelity, usefulness (DEFINE EACH SCORE HERE) | |
| - Separate translation probe with my own reference Dholuo (probe_scored.csv) | |
| ## Findings | |
| | tier | lang_ok | understanding | fidelity | usefulness | | |
| |:--------------|----------:|----------------:|-----------:|-------------:| | |
| | comprehension | 0.12 | 0.14 | 0.14 | 0.14 | | |
| | generation | 0 | 0 | nan | nan | | |
| Mean tokens per prompt: Dholuo 11.1 vs English 8.5 | |
| So every prompt I sent using the dholuo language to the LLM led to it confusing the language with Igbo | |
| ## Q3: Path forward | |
| The failure I most want to fix is that the model could not understand Dholuo. In 8 of 9 scored rows, understanding was 0, and the model typically named a different language (Igbo, Kalenjin) and then gave a confident, invented explanation, which can confuse and mislead real users. | |
| To address this, I would collect farming question-and-answer data written by Luo teachers, who write the language correctly, and fine-tune a small model on it. [YOUR LINE: why this would stop the bluffing, or what extra piece it needs.] | |
| Tokenization is a minor point: Dholuo prompts averaged 11.1 tokens against 8.5 for English (about 1.3x), so the gap in my data was small, but I would test an extended tokenizer later. | |
| To check that the fix worked, I would ask some simple questions in the languge just to see whether. This is speculative and untested. | |
| ## Limitations Single small model, one sample per prompt, one scorer (me), one Dholuo variety (NAME IT). | |
| ## Files | |
| - data/prompts.csv: evaluation prompts | |
| - data/results_full.csv: raw model outputs | |
| - data/results_scored.csv: my hand scores and notes | |
| - data/probe_scored.csv: translation probe against my reference Dholuo | |
| - outputs/probe_outputs.txt: full raw probe answers | |
| - notebook/dholuo_eval.ipynb: code used | |
Xet Storage Details
- Size:
- 2.35 kB
- Xet hash:
- 2c27afe0f3b3d8f0949cd5141497c2844a6da10b13b2c487e56bcbfa6972c3e5
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.