samlovesML's picture
|
download
raw
2.35 kB
# Dholuo blind-spot evaluation
## Q1: The blind spot
So as a dholuo speaker I have noticed that Large language Models mostly trained in the west have no understanding of such native languages
## Method
- Model: google/gemma-3-4b-it (WHY YOU CHOSE IT)
- 10 prompt pairs (Dholuo + matched English) across four tiers: comprehension, generation, practical, codeswitch
- One sample per prompt, temperature 0.7, seed 0, max 150 new tokens
- Hand-scored 0/1/2 on: lang_ok, understanding, fidelity, usefulness (DEFINE EACH SCORE HERE)
- Separate translation probe with my own reference Dholuo (probe_scored.csv)
## Findings
| tier | lang_ok | understanding | fidelity | usefulness |
|:--------------|----------:|----------------:|-----------:|-------------:|
| comprehension | 0.12 | 0.14 | 0.14 | 0.14 |
| generation | 0 | 0 | nan | nan |
Mean tokens per prompt: Dholuo 11.1 vs English 8.5
So every prompt I sent using the dholuo language to the LLM led to it confusing the language with Igbo
## Q3: Path forward
The failure I most want to fix is that the model could not understand Dholuo. In 8 of 9 scored rows, understanding was 0, and the model typically named a different language (Igbo, Kalenjin) and then gave a confident, invented explanation, which can confuse and mislead real users.
To address this, I would collect farming question-and-answer data written by Luo teachers, who write the language correctly, and fine-tune a small model on it. [YOUR LINE: why this would stop the bluffing, or what extra piece it needs.]
Tokenization is a minor point: Dholuo prompts averaged 11.1 tokens against 8.5 for English (about 1.3x), so the gap in my data was small, but I would test an extended tokenizer later.
To check that the fix worked, I would ask some simple questions in the languge just to see whether. This is speculative and untested.
## Limitations Single small model, one sample per prompt, one scorer (me), one Dholuo variety (NAME IT).
## Files
- data/prompts.csv: evaluation prompts
- data/results_full.csv: raw model outputs
- data/results_scored.csv: my hand scores and notes
- data/probe_scored.csv: translation probe against my reference Dholuo
- outputs/probe_outputs.txt: full raw probe answers
- notebook/dholuo_eval.ipynb: code used

Xet Storage Details

Size:
2.35 kB
·
Xet hash:
2c27afe0f3b3d8f0949cd5141497c2844a6da10b13b2c487e56bcbfa6972c3e5

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.