samlovesML's picture
|
download
raw
2.35 kB

Dholuo blind-spot evaluation

Q1: The blind spot

So as a dholuo speaker I have noticed that Large language Models mostly trained in the west have no understanding of such native languages

Method

  • Model: google/gemma-3-4b-it (WHY YOU CHOSE IT)
  • 10 prompt pairs (Dholuo + matched English) across four tiers: comprehension, generation, practical, codeswitch
  • One sample per prompt, temperature 0.7, seed 0, max 150 new tokens
  • Hand-scored 0/1/2 on: lang_ok, understanding, fidelity, usefulness (DEFINE EACH SCORE HERE)
  • Separate translation probe with my own reference Dholuo (probe_scored.csv)

Findings

tier lang_ok understanding fidelity usefulness
comprehension 0.12 0.14 0.14 0.14
generation 0 0 nan nan

Mean tokens per prompt: Dholuo 11.1 vs English 8.5

So every prompt I sent using the dholuo language to the LLM led to it confusing the language with Igbo

Q3: Path forward

The failure I most want to fix is that the model could not understand Dholuo. In 8 of 9 scored rows, understanding was 0, and the model typically named a different language (Igbo, Kalenjin) and then gave a confident, invented explanation, which can confuse and mislead real users. To address this, I would collect farming question-and-answer data written by Luo teachers, who write the language correctly, and fine-tune a small model on it. [YOUR LINE: why this would stop the bluffing, or what extra piece it needs.] Tokenization is a minor point: Dholuo prompts averaged 11.1 tokens against 8.5 for English (about 1.3x), so the gap in my data was small, but I would test an extended tokenizer later. To check that the fix worked, I would ask some simple questions in the languge just to see whether. This is speculative and untested.

Limitations Single small model, one sample per prompt, one scorer (me), one Dholuo variety (NAME IT).

Files

  • data/prompts.csv: evaluation prompts
  • data/results_full.csv: raw model outputs
  • data/results_scored.csv: my hand scores and notes
  • data/probe_scored.csv: translation probe against my reference Dholuo
  • outputs/probe_outputs.txt: full raw probe answers
  • notebook/dholuo_eval.ipynb: code used

Xet Storage Details

Size:
2.35 kB
·
Xet hash:
2c27afe0f3b3d8f0949cd5141497c2844a6da10b13b2c487e56bcbfa6972c3e5

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.