wikalid / README.md
jpaulpoliquit's picture
model card: drop name/history lines
860c56c verified
|
Raw History Blame Contribute Delete
15.6 kB
---
license: other
license_name: wikalid-preview
license_link: https://huggingface.co/sapinsapin/wikalid/blob/main/MODEL_LICENSE.md
language:
- tl
- ceb
- hil
- war
- ilo
- bcl
- pam
- pag
- cbk
- krj
- tsg
- en
- es
- id
- ms
- multilingual
library_name: wikalid
pipeline_tag: text-classification
tags:
- language-identification
- lid
- philippines
- filipino
- tagalog
- taglish
- code-switching
- cebuano
- hiligaynon
- low-resource
- cpu
---
# WikaLID
**Language identification for Philippine languages, with code-switching measurement and calibrated abstention.**
## Overview
General-purpose language identifiers such as GlotLID, OpenLID and fastText `lid.176` cover hundreds of languages, but
they struggle with Philippine web and news text. They mix up the Visayan languages, fail on ALLCAPS headlines and
tabloid style, can't separate Chavacano from Spanish, and treat Tagalog–English code-switching (Taglish) as noise.
WikaLID is a small, CPU-only model built for that setting:
* **17 labels**: 11 Philippine languages, 4 contact languages (English, Spanish, Indonesian, Malay), "other language"
and "not a language".
* **Token-level tagging.** Every word gets a language, so the output includes the language mixture and the English
share. Taglish is reported as a measurement, not a separate class.
* **Calibrated uncertainty.** It returns a temperature-scaled confidence and a conformal label set, plus a gate that
sorts each text into *accept*, *review* or *junk*.
* **Small and deterministic.** Two 26 MB int8 taggers, numpy-only inference, no GPU, no PyTorch, no fastText.
It was built to label a Filipino text corpus for language-model pretraining. The gate exists so that uncertain rows
can be held back instead of being silently mislabelled.
## Supported labels
| code | language | notes |
|---|---|---|
| `tgl` | Tagalog / Filipino | Includes Taglish. Code-switching is reported in `cs_level` / `cs_share` |
| `ceb` | Cebuano | |
| `hil` | Hiligaynon | Most frequent confusions are with `ceb` and `krj` |
| `war` | Waray | |
| `ilo` | Ilocano | |
| `bcl` | Bikol | Central Bikol; Rinconada is also labelled `bcl` |
| `pam` | Kapampangan | |
| `pag` | Pangasinan | Strong on formal text; rare in news, so treat news predictions with care |
| `cbk` | Chavacano | Separated from Spanish |
| `krj` | Kinaray-a | **Experimental** |
| `tsg` | Tausug | **Experimental**: no human-labelled free-text test set |
| `eng` | English | |
| `spa` | Spanish | |
| `ind` | Indonesian | Often confused with `msa` on very short strings |
| `msa` | Malay | |
| `oth` | any other language | Includes Aklanon / Surigaonon detected by marker rules (`oth_sub` = `akl` / `sgd`) |
| `nol` | not a language | Numbered lists, name rosters, score tables (≈ ISO `zxx`) |
## Install
WikaLID is distributed as a wheel with the weights bundled (about 52 MB). It needs Python ≥ 3.10 and numpy.
```bash
pip install wikalid-1.0.1-py3-none-any.whl # add pyarrow for parquet batch jobs
wikalid verify --expect-version 1.0.1 # checks the version and the sha256 of every weight file
```
## Quickstart
### Python
```python
import wikalid
lid = wikalid.load() # bundled model; deterministic
lid.analyze("Nag-submit na ako ng report pero hindi pa daw na-approve.")
# {'label': 'tgl', 'conf': 0.9968, 'label_set': ['tgl'], 'abstain': False,
# 'mixture': {'tgl': 0.9998}, 'cs_level': 'none', 'cs_share': 2.3e-06, ...}
lid.analyze("Grabe yung traffic kanina sa EDSA, super late na ako sa meeting with the client.")
# {'label': 'tgl', 'conf': 0.9344, 'label_set': ['tgl'], 'mixture': {'tgl': 0.7477, 'eng': 0.252},
# 'cs_level': 'heavy', 'cs_share': 0.252, ...}
wikalid.classify("Indi ko mahibal-an kon ngaa wala sila nagkadto.") # analysis + gate decision
# {..., 'label': 'hil', 'conf': 0.9993, 'label_final': 'hil', 'oth_sub': None,
# 'gate': 'accept_ph_other', 'gate_reason': 'accepted', 'gate_p_correct': 0.997}
```
### Command line
```bash
$ wikalid predict "Wala ko kahibalo nganong wala sila miadto."
{"text": "Wala ko kahibalo nganong wala sila miadto.", "label": "ceb", "conf": 0.9895, "abstain": false,
"label_set": ["ceb"], "mixture": {"ceb": 1.0}, "cs_level": "none", ...}
$ echo "Ano ang pangalan mo?" | wikalid predict --gate
{"text": "Ano ang pangalan mo?", "label": "tgl", "conf": 0.9967, ..., "gate": "review", "gate_reason": "meta_low", "gate_p_correct": 0.504}
$ wikalid sidecar docs.parquet labels.parquet --workers 4 # input columns: document_id, source_name, text
```
The outputs above are abridged; values are rounded from real runs of 1.0.1.
`wikalid sidecar` writes one row per document with the model columns (`ourlid_label`, `ourlid_conf`,
`ourlid_label_set`, `ourlid_mixture`, `ourlid_cs_share`, …) and the gate columns (`ourlid_label_final`, `ourlid_oth_sub`,
`gate_v2`, `gate_v2_reason`, `gate_v2_p_correct`). `wikalid score` and `wikalid gate` run the two steps separately. `wikalid info` prints the version, labels and
weight hashes.
### Output fields
| field | meaning |
|---|---|
| `label` | Majority language of the text (standard LID convention) |
| `conf` | Temperature-scaled probability of `label` |
| `label_set`, `abstain` | Split-conformal prediction set at α = 0.10; `abstain` is true when the set is not exactly one label |
| `mixture` | Share of word tokens per language (languages ≥ 1%) |
| `cs_level`, `cs_share` | English share for non-English texts (`none` < 0.03 ≤ `light` < 0.25 ≤ `heavy`) |
| `label_final`, `oth_sub` | `label`, or `oth` + `akl`/`sgd` when the Aklanon/Surigaonon marker rule fires |
| `gate`, `gate_reason`, `gate_p_correct` | Gate decision (below) |
## The gate: accept, review, junk
A label is only useful if you know when to trust it. The gate combines named rules with a small gradient-boosted
"is this label correct?" model. That model was fit only on development data, so its scores are not tuned to the test
sets reported below.
| gate | meaning | example (real outputs) |
|---|---|---|
| `accept` | confident Tagalog | "Nag-submit na ako ng report…" → `tgl`, p_correct 0.995 |
| `accept_ph_other` | confident other Philippine language, or a marker-rule hit | "Indi ko mahibal-an…" → `hil`; an Aklanon sentence → `oth`/`akl`, reason `unsupported_ph_akl` |
| `accept_eng` | confident English | |
| `review` | the model abstained, the text is very short, a sibling language is close, or confidence is below that label's threshold | heavy Taglish → `tgl` but reason `low_conf_tgl`; "Ano ang pangalan mo?" → reason `meta_low` |
| `junk` | not language | a numbered name roster → `nol`, reason `junk_nol` |
The gate is deliberately strict. The English threshold is very high, so many clean English sentences land in `review`.
That suits a corpus where English is opt-in, but you may want your own thresholds for other uses.
Rows the gate accepts were 97.2–100% correct on every sealed evaluation set, and at least 98.5% correct on every set except BasahaCorpus (table below). On about 240k Filipino
news articles, it accepted 88.6% and sent 11.4% to review.
## Evaluation
### Method
* Each evaluation set is split 50/50 by a hash of a group key, so related items stay on the same side. Every modelling
and threshold decision was made on the *dev* half. The numbers below are from the *sealed* half, which was never
used for any decision.
* All baselines were run on exactly the same items with their published models.
* "Corrected gold": some gold labels in the Wikipedia-derived sets are known to be wrong (English sentences inside
regional-language wikis, Spanish inside Chavacano). Where at least two of GlotLID, OpenLID and lid.176 agree on
English or Spanish with probability ≥ 0.6, the gold is corrected. This rule relies on the baselines, so it can only
favour them.
* Three sets are public and ship with the source repository: Wikipedia held-out, PH-LID-Hard v0 and the Tatoeba test
split. The others are private, because they contain copyrighted news text or come from restricted sources.
### Results (sealed halves; accuracy on corrected gold)
| set (n) | description | WikaLID | GlotLID v3 | OpenLID-v3 | fastText lid.176 |
|---|---|---|---|---|---|
| Crawl realism (335) | Real Filipino news spans: Taglish, headlines, lists, bylines | **0.949** | 0.875 | 0.785 | 0.734 |
| News gold spans (327) | Hand-labelled news spans | **0.939** | 0.899 | 0.881 | 0.783 |
| News gold docs (148) | Hand-labelled whole articles | **0.986** | 0.959 | 0.919 | 0.818 |
| PLD read speech (1,653) | Prompts from the Philippine Languages Database, 10 languages | **0.917** | 0.834 | 0.417 | 0.341 |
| Wikipedia held-out (2,563) | Public | **0.925** | 0.907 | 0.710 | 0.664 |
| PH-LID-Hard v0 (6,379) | Public; synthetic noise: caps, SMS spelling, typos, Taglish, lists | **0.901** | 0.829 | 0.641 | 0.599 |
| BasahaCorpus (794) | Children's stories (Kinaray-a, Hiligaynon, …) | **0.739** | 0.583 | 0.026 | 0.031 |
| Tatoeba test (968) | Public; short, clean sentences | 0.928 | **0.955** | 0.758 | 0.805 |
| **Mean** | | **0.911** | 0.855 | 0.642 | 0.597 |
Paired bootstrap against GlotLID (2,000 resamples, 95% CI) on the sealed halves:
* WikaLID is better on seven sets: crawl +7.5 pts [+4.5, +11.0], gold spans +4.0 [+0.9, +7.3], gold docs +2.7
[+0.7, +5.4], PLD +8.3 [+6.7, +9.9], Wikipedia +1.8 [+1.0, +2.7], PH-LID-Hard +7.2 [+6.3, +8.0], Basaha +15.6
[+12.3, +18.9].
* **It is worse on Tatoeba: −2.7 pts [−4.6, −0.9].** On short, clean, textbook-style sentences, GlotLID's breadth wins.
### Human-labelled external sets (all items)
| set | WikaLID | GlotLID v3 | OpenLID-v3 | lid.176 |
|---|---|---|---|---|
| FLORES-200 via SIB-200, 9 languages (9,036) | **0.989** | 0.986 | 0.985 | 0.774 |
| Same sentences in UPPER / Title / lower case (5,388) | **0.999** | 0.975 | 0.575 | 0.522 |
| Central Bikol story sentences: Bikol recall (900) | **0.961** | 0.894 | 0.000 | 0.003 |
| 2–5 word fragments of FLORES sentences (4,320) | **0.832** | 0.771 | 0.773 | 0.590 |
| … of which 2 words (1,080) | **0.704** | 0.624 | 0.621 | 0.514 |
| Short Tatoeba sentences (543) | 0.856 | **0.866** | 0.632 | 0.685 |
| Aklanon / Surigaonon sentences → `oth` (110) | **0.345** | 0.300 | 0.082 | 0.055 |
### Gate (sealed halves)
| set | share accepted | accuracy of accepted rows |
|---|---|---|
| Crawl realism | 0.510 | 1.000 |
| News gold spans | 0.514 | 1.000 |
| News gold docs | 0.784 | 0.991 |
| PLD read speech | 0.658 | 0.985 |
| Wikipedia held-out | 0.566 | 0.989 |
| PH-LID-Hard v0 | 0.510 | 0.994 |
| BasahaCorpus | 0.358 | 0.972 |
| Tatoeba test | 0.395 | 0.995 |
These sets are deliberately hard and short, so acceptance rates are much lower than on ordinary news.
### Speed
Measured on a shared 8-core CPU, one BLAS thread per worker: about 7,000 documents per minute per core for 100-word
documents, and about 2,200 per minute per core for full news articles (~400 tokens). Throughput scales roughly
linearly with workers.
## Limitations and bias
* **Languages without a class.** Aklanon, Surigaonon and many smaller Philippine languages have no open training text.
Hand-written marker rules move only about a third of Aklanon/Surigaonon sentences to `oth`. The rest are labelled
`hil` or `ceb`. Do not use WikaLID to count speakers of unsupported languages.
* **The Visayan family.** Hiligaynon, Kinaray-a and Cebuano remain the main source of errors. `krj` and `tsg` are
experimental.
* **English-heavy Taglish.** Tabloid and showbiz text with ALLCAPS leads and English quotes can be labelled `eng`.
Usually the label set comes back empty and the gate sends the text to `review`. When WikaLID abstains on `eng`, the top
label is right only about 22% of the time on our gold data.
* **Short strings.** Two-word inputs are 70% accurate, and a third of the remaining errors are Indonesian vs Malay.
* **Rare labels on news.** In a manual sample of news predictions, every `krj` (3/3) and `pag` (2/2) prediction was
wrong: lists, Aklanon or Hiligaynon text. Route these to review.
* **Calibration across domains.** The conformal sets target 90% coverage at α = 0.10, but that target holds only near
the calibration data. On the sealed halves, the gold label was inside the set for 82–91% of items on the news, speech,
Wikipedia, PH-LID-Hard and Tatoeba sets, and for 56% on BasahaCorpus. Between 8% and 29% of sets come back empty, which
means "no label is safe"; treat an empty set as a signal to review, not as a prediction.
* **Evaluation caveats.**
* The news gold sets were labelled by a single, non-native annotator. A native-speaker re-annotation is planned.
* PH-LID-Hard is synthetic and shares design ideas with the training-data augmentation.
* BasahaCorpus accuracy counts Minasbate as `oth`, which flatters every system.
* Real-news results should carry the most weight.
* **Register and region.** The training text is mostly encyclopedic, web and story text in standard orthographies.
Older spellings, regional varieties and social-media slang are less well covered.
## Training data
No training text is distributed with the model. The taggers were trained on sentences from:
* Wikipedia (Tagalog, Cebuano, Waray, Ilocano, Bikol, Kapampangan, Pangasinan, Chavacano, English, Spanish, Indonesian,
Malay and about 30 other languages): CC BY-SA 4.0.
* Tatoeba: CC BY 2.0 FR.
* FineWeb-2 web text for the lower-resource languages, filtered for lexical purity: ODC-By 1.0. Page copyright remains
with the publishers, and this text is used for training only.
* BasahaCorpus children's stories (Imperial & Kochmar, 2023) for Kinaray-a and Hiligaynon: CC BY 4.0 as stated by its
authors.
Training sentences are augmented with synthetic code-switching, casing, SMS-style spelling and accent variants. No
scraped news is in the training data. Existing LID tools (GlotLID, OpenLID, lid.176) were used only to detect
label noise. No third-party model weights are included.
## Versioning
* Releases follow semantic versioning. A version always pins exact weights. `wikalid verify --expect-version X`
checks the version and the sha256 of every bundled file against `MANIFEST.json`.
* Current release: **1.0.1** (2026-11-15). Release history is in `CHANGELOG.md`.
* A retrained model will be released as a new minor version with a fresh evaluation. Weights are never swapped silently.
## License
* **Code:** Apache-2.0.
* **Model weights:** preview release, all rights reserved; redistribution is not permitted. A public licence is under
consideration. Because Wikipedia is the largest training source, CC BY-SA 4.0 is the likely choice. See
`MODEL_LICENSE.md`.
* **Public evaluation sets:** CC BY-SA 4.0 (Wikipedia-derived) and CC BY 2.0 FR (Tatoeba).
## Citation
```bibtex
@software{wikalid_2026,
title = {WikaLID: Language Identification for Philippine Languages},
author = {{sapinsapin}},
year = {2026},
version = {1.0.1},
url = {https://huggingface.co/sapinsapin/wikalid}
}
```
## Acknowledgements and contact
WikaLID builds on public resources from Wikipedia and Tatoeba contributors, FineWeb-2, BasahaCorpus (Imperial &
Kochmar, 2023; stories from Let's Read Asia / The Asia Foundation), FLORES-200 / SIB-200 (used for evaluation only) and
the Philippine Languages Database (used for evaluation only). Baselines: GlotLID (Kargaran et al., 2023), OpenLID
(Burchell et al., 2023) and fastText lid.176 (Joulin et al., 2016).
Questions, error reports and offers to help with native-speaker review are welcome in the Community tab of this
repository.