README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: text-classification
tags:
  - language-identification
  - african-languages
---

# Sunflower Language ID v2

Language identification across 64 African languages, fine-tuned from
`google/t5-efficient-tiny` with a **replacement SentencePiece vocabulary** trained
on the target corpus.

## Why v2

v1 used the stock T5 English-C4 vocabulary, which has no byte fallback and could
not represent large parts of this language set — Amharic was 47% `<unk>`, Yoruba
20%. That damage falls hardest on short input, where there is no redundancy to
absorb a lost token. v2 ships a 32k byte-fallback vocabulary trained on balanced
text from all 64 languages, measured at **0.00% `<unk>`**.

## Accuracy by input length

Held-out split, 64 languages, ~16,000 sentences per bucket.

| Input length | v1 | v2 |
|---|---|---|
| 1 word | 0.201 | **0.405** |
| 2 words | 0.477 | **0.597** |
| 3 words | 0.614 | **0.701** |
| 5 words | 0.749 | **0.805** |
| 8 words | 0.832 | **0.864** |
| full sentence | 0.907 | **0.928** |

## Usage — two things that must match training

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

REPO = "yigagilbert/sunflower_language_classification_v2"

# 1. use_fast=False. The fast conversion drops SentencePiece byte_fallback and
#    reintroduces <unk> (0.47% on Amharic), which is the bug this model fixes.
tok = AutoTokenizer.from_pretrained(REPO, use_fast=False)
mdl = AutoModelForSequenceClassification.from_pretrained(REPO).eval()

# 2. Normalise exactly as training did: lowercase + collapse whitespace.
def normalize_text(s):
    return " ".join(s.lower().split())

def predict(text):
    enc = tok(normalize_text(text), return_tensors="pt", truncation=True, max_length=64)
    with torch.no_grad():
        probs = mdl(**enc).logits[0].softmax(-1)
    top = probs.argmax().item()
    return mdl.config.id2label[top], probs[top].item()

print(predict("Webale nyo okutuyamba"))   # ('lug', 0.62)
```

Skipping either step silently degrades short-text accuracy — passing raw-case
text to a model trained on lowercase is what caused the original regression.

## Supported languages

ISO 639-3 codes. "Short" is accuracy on 1-3 word input; chance is ~0.016.

| Code | Language | Short | | Code | Language | Short |
|---|---|---|---|---|---|---|
| `ach` | Acholi | 0.605 | | `luc` | Aringa | 0.479 |
| `adh` | Adhola (Jopadhola) | 0.292 | | `lug` | Luganda | 0.692 |
| `afr` | Afrikaans | 0.866 | | `luo` | Dholuo | 0.705 |
| `aka` | Akan | 0.505 | | `luy` | Luyia | 0.228 |
| `alz` | Alur | 0.453 | | `mhi` | Ma'di | 0.268 |
| `amh` | Amharic | 0.940 | | `mlg` | Malagasy | 0.906 |
| `bam` | Bambara | 0.476 | | `myx` | Masaaba | 0.727 |
| `bem` | Bemba | 0.746 | | `nbl` | Southern Ndebele | 0.579 |
| `bfa` | Bari | 0.468 | | `nuj` | Nyole | 0.338 |
| `cgg` | Chiga (Rukiga) | 0.193 | | `nya` | Nyanja (Chichewa) | 0.810 |
| `dag` | Dagbani | 0.485 | | `nyn` | Nyankole | 0.582 |
| `din` | Dinka | 0.696 | | `nyo` | Nyoro (Runyoro) | 0.203 |
| `eng` | English | 0.698 | | `orm` | Oromo | 0.838 |
| `ewe` | Ewe | 0.809 | | `pcm` | Nigerian Pidgin | 0.564 |
| `fra` | French | 0.655 | | `pok` | Pokoot | 0.047 |
| `ful` | Fulah | 0.511 | | `rub` | Gungu | 0.439 |
| `gwr` | Gwere | 0.256 | | `ruc` | Ruuli | 0.114 |
| `hau` | Hausa | 0.709 | | `run` | Rundi (Kirundi) | 0.627 |
| `ibo` | Igbo | 0.819 | | `rwm` | Amba | 0.031 |
| `kab` | Kabyle | 0.674 | | `sna` | Shona | 0.813 |
| `kau` | Kanuri | 0.321 | | `som` | Somali | 0.864 |
| `kdi` | Kumam | 0.376 | | `sot` | Southern Sotho | 0.556 |
| `kdj` | Karamojong | 0.432 | | `swa` | Swahili | 0.814 |
| `keo` | Kakwa | 0.421 | | `teo` | Teso (Ateso) | 0.793 |
| `kik` | Kikuyu | 0.796 | | `tlj` | Talinga-Bwisi | 0.417 |
| `kin` | Kinyarwanda | 0.615 | | `tsn` | Tswana | 0.747 |
| `koo` | Konzo | 0.702 | | `ttj` | Tooro (Rutooro) | 0.523 |
| `kpz` | Kupsabiny | 0.667 | | `wol` | Wolof | 0.669 |
| `laj` | Lango | 0.252 | | `xho` | Xhosa | 0.635 |
| `lgg` | Lugbara | 0.593 | | `xog` | Soga (Lusoga) | 0.497 |
| `lin` | Lingala | 0.845 | | `yor` | Yoruba | 0.885 |
| `lsm` | Saamia | 0.401 | | `zul` | Zulu | 0.640 |

## Known limitations

Several pairs are genuinely close to inseparable on short input, and the
confusions are symmetric because the languages overlap in reality:

| Pair | Rate | Note |
|---|---|---|
| cgg to nyn | 34.2% | Chiga/Nyankole, often treated as one language |
| nyo to ttj | 30.4% | Nyoro/Tooro, likewise |
| run to kin | 19.7% | Kirundi/Kinyarwanda, mutually intelligible |
| nbl to zul | 18.6% | Nguni cluster |
| pcm to eng | 14.0% | Nigerian Pidgin vs English |

For a single word, expect ~0.41 accuracy across 64 classes (chance is
~0.016). If your deployment only serves a known subset of languages, mask the
logits to that subset — it recovers a large amount of accuracy for free.

## Training

- Base: `google/t5-efficient-tiny`, embeddings re-initialised for the new vocab
- 60,000 steps, batch 64, lr 1e-3 cosine, bf16, best checkpoint by
  short-text (1-3 word) accuracy
- Augmentation: log-uniform random crops biased toward short spans, light
  character noise
- Logit adjustment (tau=1.0) with the class prior clipped to 50:1
- Languages with under 1,400 unique examples excluded as unlearnable