30.4 kB
6 files
Updated 28 days ago
Name
Size
data
outputs
README.md2.35 kB
xet
README.md

Dholuo blind-spot evaluation

Q1: The blind spot

So as a dholuo speaker I have noticed that Large language Models mostly trained in the west have no understanding of such native languages

Method

  • Model: google/gemma-3-4b-it (WHY YOU CHOSE IT)
  • 10 prompt pairs (Dholuo + matched English) across four tiers: comprehension, generation, practical, codeswitch
  • One sample per prompt, temperature 0.7, seed 0, max 150 new tokens
  • Hand-scored 0/1/2 on: lang_ok, understanding, fidelity, usefulness (DEFINE EACH SCORE HERE)
  • Separate translation probe with my own reference Dholuo (probe_scored.csv)

Findings

tier lang_ok understanding fidelity usefulness
comprehension 0.12 0.14 0.14 0.14
generation 0 0 nan nan

Mean tokens per prompt: Dholuo 11.1 vs English 8.5

So every prompt I sent using the dholuo language to the LLM led to it confusing the language with Igbo

Q3: Path forward

The failure I most want to fix is that the model could not understand Dholuo. In 8 of 9 scored rows, understanding was 0, and the model typically named a different language (Igbo, Kalenjin) and then gave a confident, invented explanation, which can confuse and mislead real users. To address this, I would collect farming question-and-answer data written by Luo teachers, who write the language correctly, and fine-tune a small model on it. [YOUR LINE: why this would stop the bluffing, or what extra piece it needs.] Tokenization is a minor point: Dholuo prompts averaged 11.1 tokens against 8.5 for English (about 1.3x), so the gap in my data was small, but I would test an extended tokenizer later. To check that the fix worked, I would ask some simple questions in the languge just to see whether. This is speculative and untested.

Limitations Single small model, one sample per prompt, one scorer (me), one Dholuo variety (NAME IT).

Files

  • data/prompts.csv: evaluation prompts
  • data/results_full.csv: raw model outputs
  • data/results_scored.csv: my hand scores and notes
  • data/probe_scored.csv: translation probe against my reference Dholuo
  • outputs/probe_outputs.txt: full raw probe answers
  • notebook/dholuo_eval.ipynb: code used
Total size
30.4 kB
Files
6
Last updated
Oct 22
Pre-warmed CDN
US EU US EU

Contributors