File size: 5,332 Bytes
76e7518 090b818 76e7518 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 | ---
license: apache-2.0
library_name: transformers
pipeline_tag: text-classification
tags:
- language-identification
- african-languages
---
# Sunflower Language ID v2
Language identification across 64 African languages, fine-tuned from
`google/t5-efficient-tiny` with a **replacement SentencePiece vocabulary** trained
on the target corpus.
## Why v2
v1 used the stock T5 English-C4 vocabulary, which has no byte fallback and could
not represent large parts of this language set — Amharic was 47% `<unk>`, Yoruba
20%. That damage falls hardest on short input, where there is no redundancy to
absorb a lost token. v2 ships a 32k byte-fallback vocabulary trained on balanced
text from all 64 languages, measured at **0.00% `<unk>`**.
## Accuracy by input length
Held-out split, 64 languages, ~16,000 sentences per bucket.
| Input length | v1 | v2 |
|---|---|---|
| 1 word | 0.201 | **0.405** |
| 2 words | 0.477 | **0.597** |
| 3 words | 0.614 | **0.701** |
| 5 words | 0.749 | **0.805** |
| 8 words | 0.832 | **0.864** |
| full sentence | 0.907 | **0.928** |
## Usage — two things that must match training
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
REPO = "yigagilbert/sunflower_language_classification_v2"
# 1. use_fast=False. The fast conversion drops SentencePiece byte_fallback and
# reintroduces <unk> (0.47% on Amharic), which is the bug this model fixes.
tok = AutoTokenizer.from_pretrained(REPO, use_fast=False)
mdl = AutoModelForSequenceClassification.from_pretrained(REPO).eval()
# 2. Normalise exactly as training did: lowercase + collapse whitespace.
def normalize_text(s):
return " ".join(s.lower().split())
def predict(text):
enc = tok(normalize_text(text), return_tensors="pt", truncation=True, max_length=64)
with torch.no_grad():
probs = mdl(**enc).logits[0].softmax(-1)
top = probs.argmax().item()
return mdl.config.id2label[top], probs[top].item()
print(predict("Webale nyo okutuyamba")) # ('lug', 0.62)
```
Skipping either step silently degrades short-text accuracy — passing raw-case
text to a model trained on lowercase is what caused the original regression.
## Supported languages
ISO 639-3 codes. "Short" is accuracy on 1-3 word input; chance is ~0.016.
| Code | Language | Short | | Code | Language | Short |
|---|---|---|---|---|---|---|
| `ach` | Acholi | 0.605 | | `luc` | Aringa | 0.479 |
| `adh` | Adhola (Jopadhola) | 0.292 | | `lug` | Luganda | 0.692 |
| `afr` | Afrikaans | 0.866 | | `luo` | Dholuo | 0.705 |
| `aka` | Akan | 0.505 | | `luy` | Luyia | 0.228 |
| `alz` | Alur | 0.453 | | `mhi` | Ma'di | 0.268 |
| `amh` | Amharic | 0.940 | | `mlg` | Malagasy | 0.906 |
| `bam` | Bambara | 0.476 | | `myx` | Masaaba | 0.727 |
| `bem` | Bemba | 0.746 | | `nbl` | Southern Ndebele | 0.579 |
| `bfa` | Bari | 0.468 | | `nuj` | Nyole | 0.338 |
| `cgg` | Chiga (Rukiga) | 0.193 | | `nya` | Nyanja (Chichewa) | 0.810 |
| `dag` | Dagbani | 0.485 | | `nyn` | Nyankole | 0.582 |
| `din` | Dinka | 0.696 | | `nyo` | Nyoro (Runyoro) | 0.203 |
| `eng` | English | 0.698 | | `orm` | Oromo | 0.838 |
| `ewe` | Ewe | 0.809 | | `pcm` | Nigerian Pidgin | 0.564 |
| `fra` | French | 0.655 | | `pok` | Pokoot | 0.047 |
| `ful` | Fulah | 0.511 | | `rub` | Gungu | 0.439 |
| `gwr` | Gwere | 0.256 | | `ruc` | Ruuli | 0.114 |
| `hau` | Hausa | 0.709 | | `run` | Rundi (Kirundi) | 0.627 |
| `ibo` | Igbo | 0.819 | | `rwm` | Amba | 0.031 |
| `kab` | Kabyle | 0.674 | | `sna` | Shona | 0.813 |
| `kau` | Kanuri | 0.321 | | `som` | Somali | 0.864 |
| `kdi` | Kumam | 0.376 | | `sot` | Southern Sotho | 0.556 |
| `kdj` | Karamojong | 0.432 | | `swa` | Swahili | 0.814 |
| `keo` | Kakwa | 0.421 | | `teo` | Teso (Ateso) | 0.793 |
| `kik` | Kikuyu | 0.796 | | `tlj` | Talinga-Bwisi | 0.417 |
| `kin` | Kinyarwanda | 0.615 | | `tsn` | Tswana | 0.747 |
| `koo` | Konzo | 0.702 | | `ttj` | Tooro (Rutooro) | 0.523 |
| `kpz` | Kupsabiny | 0.667 | | `wol` | Wolof | 0.669 |
| `laj` | Lango | 0.252 | | `xho` | Xhosa | 0.635 |
| `lgg` | Lugbara | 0.593 | | `xog` | Soga (Lusoga) | 0.497 |
| `lin` | Lingala | 0.845 | | `yor` | Yoruba | 0.885 |
| `lsm` | Saamia | 0.401 | | `zul` | Zulu | 0.640 |
## Known limitations
Several pairs are genuinely close to inseparable on short input, and the
confusions are symmetric because the languages overlap in reality:
| Pair | Rate | Note |
|---|---|---|
| cgg to nyn | 34.2% | Chiga/Nyankole, often treated as one language |
| nyo to ttj | 30.4% | Nyoro/Tooro, likewise |
| run to kin | 19.7% | Kirundi/Kinyarwanda, mutually intelligible |
| nbl to zul | 18.6% | Nguni cluster |
| pcm to eng | 14.0% | Nigerian Pidgin vs English |
For a single word, expect ~0.41 accuracy across 64 classes (chance is
~0.016). If your deployment only serves a known subset of languages, mask the
logits to that subset — it recovers a large amount of accuracy for free.
## Training
- Base: `google/t5-efficient-tiny`, embeddings re-initialised for the new vocab
- 60,000 steps, batch 64, lr 1e-3 cosine, bf16, best checkpoint by
short-text (1-3 word) accuracy
- Augmentation: log-uniform random crops biased toward short spans, light
character noise
- Logit adjustment (tau=1.0) with the class prior clipped to 50:1
- Languages with under 1,400 unique examples excluded as unlearnable
|