ORÚKỌ-mini: Does a small frontier LLM keep an African name intact?

A small-scale, text-only probe of the identity-preservation gap described in the ORÚKỌ benchmark proposal, run on Qwen/Qwen3.5-4B.

Author: Flora Oladipupo · October 2026

---

TL;DR

Qwen3.5-4B claims 201 languages. When a correctly written Yorùbá, Igbo or Hausa name sits in the prompt, it mostly copies it, but asked to write that name in capital letters it drops the marks for most of them (Ɗanjuma → DANJUMA, Chiọma → CHIOMA). When it has to supply the marks itself, it fails almost completely: it cannot restore a stripped Yorùbá or Hausa name, cannot read Yorùbá tone marks even when the rule is in the prompt, gives the same answer to words that differ only by tone, and rates the stripped spelling as more likely than the correct one for every African name tested. The same model restores François and Nguyễn Thị Hương perfectly. Standard benchmarks miss this, because they either strip diacritics before scoring or never test names at all.

The practical consequence: in any pipeline where the name arrives without its marks (a form, an ASR transcript, a database that already folded it) or must be transformed (uppercased, normalised for TTS), the model will not recover the person's real name, and nothing in a typical eval will flag it.

Results (model: Qwen/Qwen3.5-4B, greedy, non-thinking, 363 generations)

1. Copying works while the name sits in the prompt, until the letter case changes. In the repeat, welcome-message, JSON and translation contexts the model kept names exactly 100% (Yorùbá), 100% (Igbo) and 100% (Hausa) of the time. Asked for capital letters, it kept only 30%, 0% and 33% intact, against 78% for controls (Adéṣọlá → ADEṢOLÁ, Ọláyínká → ỌLAYINKA, Fọláṣadé → FOLÁṢADÉ). All copy-task loss comes from this one context.
2. It cannot produce the marks itself. Restoring a stripped name: Yorùbá 0% exact (mark F1 0.453), Igbo 12%, Hausa 0%, against 100% for controls such as François and Nguyễn Thị Hương.
3. It cannot read tone marks, even with the rule in the prompt. Exact Yorùbá tone sequence: 5%; per-vowel accuracy 35%, close to the 33% expected from guessing among three tones.
4. It does not know what the names mean. Meaning correct for the marked name: Yorùbá 5%, Igbo 0%, Hausa 50% (only 3 Hausa names, all day-of-birth names).
5. Marks do not change its answer. Minimal-pair accuracy: Yorùbá 25%, Igbo 0%, Hausa 25%. In 4 of 5 sets it gave one answer to words that differ only by marks (e.g. igbá, ìgbà, igba, ìgbá were all "Time"). Asked to list what a bare form could mean, it recalled 12% / 0% / 25% of the senses and spelled 17% / 100% / 0% of the marked words correctly (Yorùbá / Igbo / Hausa).
6. The stripped spelling is its default. It scores the stripped name as more likely than the correct one in 100% (Yorùbá), 100% (Igbo) and 100% (Hausa) of cases, against 50% for controls. Correct Yorùbá names also cost 2.13× the tokens of their stripped forms (Hausa 1.89×, controls 1.23×).
7. Diacritic density did not predict copy failures (Spearman ρ = 0.09, p = 0.63); the failures cluster in one context rather than in heavily marked names.

Examples from the run (first non-exact case per language):

Numbers above are generated by the notebook; read results/generations.jsonl to audit every output.

---

1. The blind spot (Question 1)

Identity erasure of African personal names. In Yorùbá and Igbo, tone and diacritics are part of the word; in Hausa, ɓ, ɗ and ƙ are separate letters, not decorations. Ọkọ is a husband, ọkọ̀ a vehicle, ọkọ́ a hoe and oko a farm; Igbo àkwà, ákwà, ákwá and àkwá are bed, cloth, cry and egg. Strip the marks and the word, and any name built on it, becomes a different word or no word at all. Ɗanjuma folded to ASCII becomes Danjuma, and with a naive encode("ascii", "ignore") it becomes anjuma, a name that starts mid-word.

Why I care. I am a Nigerian AI engineer. I have watched names like mine, and those of nearly everyone I grew up with, come back from forms, airline systems, banking apps, voice assistants and now chatbots as a flattened approximation. I have experienced it over and over again as an African with an African name seen people mispronounce my name without remorse even when corrected so since I have the will power to correct this in AI system since I have limited control over correcting humans on such mistake hence why I am utilizing my skillset as an AI Engineer to correct this. Each instance looks like a small courtesy failure. Together they mean the systems that increasingly mediate identity (KYC, health records, government services, voice agents) are quietly recording people under names that are not theirs.

Why benchmarks miss it.

• Normalisation hides it. WER/CER pipelines, BLEU tokenisers and many pre-training filters lowercase and strip diacritics before scoring or deduplicating. The error is removed before anyone measures it.
• Names are treated as copy tokens. Benchmarks assume a model either knows a language or not, and treat names as opaque strings to pass through.
• Language coverage is mistaken for identity fidelity. Model cards list Yorùbá, Igbo and Hausa as "supported" based on QA or translation scores. A model can translate a Yorùbá sentence acceptably and still drop every tone mark in the speaker's name.
• It looks like success. The output is fluent, grammatical and in the right language. Only a speaker of the language notices the name is wrong, and those speakers are rarely the ones writing evals.

This probe takes the full ORÚKỌ framing (orthographic, phonemic, tonal, prosodic and semantic fidelity in speech systems) and asks the text-only, LLM-sized version of the question: before any audio is produced, does the language model at the front of the pipeline even keep the name?

---

2. How it was evaluated (Question 2)

Model choice: Qwen/Qwen3.5-4B

• Current frontier family. Released February 2026 as the small member of the Qwen3.5 family; its post-training recipe, tokenizer (≈248k vocabulary) and multilingual data are shared with the much larger flagship models, so its failure modes are a reasonable proxy for theirs.
• Explicitly multilingual. The model card claims support for 201 languages and dialects. That makes it the right test: if the gap appears here, it is not a "language not supported" story.
• Practical. ≈4B language-model parameters (≈5B with the vision encoder, inside the 0.6–6B limit), Apache-2.0, ungated, and loadable on a free Colab GPU.
• Run in non-thinking (instruct) mode with greedy decoding, so every number is reproducible.

Dataset (data/)

Controls include names with no marks at all (Chukwuebuka, Nkechi, Ladi) to catch over-correction, and tonal high-resource scripts (Vietnamese, pinyin) so that "tones are hard" cannot explain the result on its own. Igbo and Hausa are scored on dotted vowels and hooked letters only, since their standard orthographies do not write tone in names.

Correctness is bearer-defined. Rows marked check should be confirmed by a native speaker (ideally a bearer of the name) before the dataset is cited.

Tasks (the Identity Preservation Framework, adapted to text)

The analysis also relates outcomes to linguistic features (diacritic density, length, hooked letters), as proposed in ORÚKỌ.

Reproducing

Open oruko_eval.ipynb in Google Colab (GPU runtime; L4 or A100 preferred, T4 works), add an HF_TOKEN secret with write access, and run all cells. The run takes roughly 15–30 minutes, writes everything under results/ and figures/, fills in the results section of this README, and syncs the folder to the bucket.

---

3. A path forward (Question 3)

These are proposals, not tested results.

1. Stop deleting the signal in data curation.

• Audit pre-training and SFT pipelines for normalisation that strips combining marks or folds special letters (NFKD + ASCII fold, unidecode, aggressive dedup keys). Deduplicate on NFC text, never on folded text.
• Up-weight properly marked Yorùbá, Igbo and Hausa text (edited news, literature, Bible translations, curated parallel corpora) relative to the large volume of unmarked social-media text, and tag diacritic completeness as a quality signal.
• Build a bearer-verified name lexicon: name, orthography, tone pattern, meaning, regional variants, consented by native speakers. This is small, cheap, and the highest-value data for this gap.

2. Fine-tune for preservation, not just knowledge.

• Case-mapping data: the clearest failure in this run is uppercasing. Text where marked names appear in capitals (ID cards, boarding passes, headlines, legal documents) is rare in training data, so the model never learned that ọ uppercases to Ọ and ɗ to Ɗ. A few thousand synthetic examples should close most of this.
• Copy-under-transformation SFT: synthetic tasks that embed marked names in forms, JSON, translation, case changes and summaries, with targets that keep every mark. This attacks the most common real-world failure directly.
• Contrastive minimal pairs: train the model to answer ọkọ / ọkọ̀ / ọkọ́ / oko differently, and to say "this is ambiguous without tone marks" when given a bare form instead of guessing.

3. Small architectural and inference-time changes.

• Tokenizer: add merges for tone-marked syllables and NFC-precomposed under-dot vowels so a marked name is not 2–3x more fragmented than its stripped form. Fragmentation makes the stripped form the cheaper path at every decoding step.
• Protected spans: when a user supplies a name, treat it as an immutable span (a copy/pointer mechanism or constrained decoding) that must be reproduced byte-for-byte, the way some systems already protect URLs and code identifiers.
• Output guardrail: a cheap post-hoc check that any name present in the input reappears with identical identity features, and flags or repairs it if not.

4. Make it a standard metric. Report identity preservation on model cards next to multilingual QA scores, and never compute WER/CER/BLEU on diacritic-stripped text for these languages. The broader ORÚKỌ benchmark extends the same measurements to TTS and ASR, where the stripped name finally becomes a mispronounced one.

---

Bucket contents

README.md                 this file (answers to Q1 and Q3, method and results for Q2)
oruko_eval.ipynb          Colab notebook that runs the whole evaluation
oruko_core.py             dataset, prompts and scoring (pure Python, unit-testable)
data/                     names.csv, minimal_pairs.csv, prompts.jsonl
results/                  generations.jsonl, per-task score CSVs, summary.json, tokenizer and log-prob tables
figures/                  charts used in this README
model/                    server-side copy of Qwen/Qwen3.5-4B weights (exact model evaluated)

Limitations

• Small dataset (43 names, 16 words); results are indicative, not a leaderboard. Phase 1 covers only three Nigerian languages.
• Meaning is scored with keyword groups, which can miss valid paraphrases; the raw generations are released for manual review.
• The likelihood probe compares summed log-probabilities, so shorter (stripped) spellings get some advantage from length alone. The controls are scored the same way and prefer the stripped form far less often, which suggests length is not the whole story, but a per-token comparison would make this cleaner.
• Copy-task loss is concentrated in a single context (uppercase). More transformation contexts (title case, slug/URL, transliteration into other scripts, summarisation) would show how general that is.
• Greedy decoding with one prompt per template; sampling and prompt variation would give confidence intervals.
• Text only. The full ORÚKỌ framework measures pronunciation, tone contours and prosody in audio.

References

1. Olatunji et al. AfriNames: Most ASR models "butcher" African Names. arXiv:2306.00253, 2023.
2. Intron Innovation. Afri-names: Read Speech Dataset of Numbers and African Named Entities. Hugging Face, 2025. CC BY-NC-SA 4.0.
3. Qwen Team. Qwen3.5: Towards Native Multimodal Agents. 2026.

Dataset and code released under CC BY 4.0 (data) and Apache-2.0 (code). Model weights remain under their original Apache-2.0 license.