File size: 15,641 Bytes
863c520
 
8f78c5a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
863c520
8f78c5a
 
 
 
 
 
 
 
 
 
 
 
863c520
 
8f78c5a
863c520
8f78c5a
863c520
8f78c5a
863c520
8f78c5a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
863c520
 
8f78c5a
 
863c520
8f78c5a
 
 
 
 
863c520
8f78c5a
 
 
 
 
 
 
 
 
 
 
 
 
 
863c520
 
8f78c5a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
860c56c
8f78c5a
 
 
 
 
863c520
8f78c5a
 
 
 
 
 
 
 
 
 
 
 
 
863c520
8f78c5a
 
 
 
 
 
 
863c520
8f78c5a
 
863c520
8f78c5a
 
863c520
8f78c5a
863c520
8f78c5a
863c520
8f78c5a
 
 
 
 
 
 
 
 
 
863c520
8f78c5a
 
 
863c520
8f78c5a
 
 
 
 
 
 
 
 
863c520
8f78c5a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
860c56c
8f78c5a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
863c520
 
8f78c5a
 
 
 
 
 
863c520
8f78c5a
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
---
license: other
license_name: wikalid-preview
license_link: https://huggingface.co/sapinsapin/wikalid/blob/main/MODEL_LICENSE.md
language:
- tl
- ceb
- hil
- war
- ilo
- bcl
- pam
- pag
- cbk
- krj
- tsg
- en
- es
- id
- ms
- multilingual
library_name: wikalid
pipeline_tag: text-classification
tags:
- language-identification
- lid
- philippines
- filipino
- tagalog
- taglish
- code-switching
- cebuano
- hiligaynon
- low-resource
- cpu
---

# WikaLID

**Language identification for Philippine languages, with code-switching measurement and calibrated abstention.**

## Overview

General-purpose language identifiers such as GlotLID, OpenLID and fastText `lid.176` cover hundreds of languages, but
they struggle with Philippine web and news text. They mix up the Visayan languages, fail on ALLCAPS headlines and
tabloid style, can't separate Chavacano from Spanish, and treat Tagalog–English code-switching (Taglish) as noise.
WikaLID is a small, CPU-only model built for that setting:

* **17 labels**: 11 Philippine languages, 4 contact languages (English, Spanish, Indonesian, Malay), "other language"
  and "not a language".
* **Token-level tagging.** Every word gets a language, so the output includes the language mixture and the English
  share. Taglish is reported as a measurement, not a separate class.
* **Calibrated uncertainty.** It returns a temperature-scaled confidence and a conformal label set, plus a gate that
  sorts each text into *accept*, *review* or *junk*.
* **Small and deterministic.** Two 26 MB int8 taggers, numpy-only inference, no GPU, no PyTorch, no fastText.

It was built to label a Filipino text corpus for language-model pretraining. The gate exists so that uncertain rows
can be held back instead of being silently mislabelled.

## Supported labels

| code | language | notes |
|---|---|---|
| `tgl` | Tagalog / Filipino | Includes Taglish. Code-switching is reported in `cs_level` / `cs_share` |
| `ceb` | Cebuano | |
| `hil` | Hiligaynon | Most frequent confusions are with `ceb` and `krj` |
| `war` | Waray | |
| `ilo` | Ilocano | |
| `bcl` | Bikol | Central Bikol; Rinconada is also labelled `bcl` |
| `pam` | Kapampangan | |
| `pag` | Pangasinan | Strong on formal text; rare in news, so treat news predictions with care |
| `cbk` | Chavacano | Separated from Spanish |
| `krj` | Kinaray-a | **Experimental** |
| `tsg` | Tausug | **Experimental**: no human-labelled free-text test set |
| `eng` | English | |
| `spa` | Spanish | |
| `ind` | Indonesian | Often confused with `msa` on very short strings |
| `msa` | Malay | |
| `oth` | any other language | Includes Aklanon / Surigaonon detected by marker rules (`oth_sub` = `akl` / `sgd`) |
| `nol` | not a language | Numbered lists, name rosters, score tables (≈ ISO `zxx`) |

## Install

WikaLID is distributed as a wheel with the weights bundled (about 52 MB). It needs Python ≥ 3.10 and numpy.

```bash
pip install wikalid-1.0.1-py3-none-any.whl            # add pyarrow for parquet batch jobs
wikalid verify --expect-version 1.0.1                 # checks the version and the sha256 of every weight file
```

## Quickstart

### Python

```python
import wikalid

lid = wikalid.load()                       # bundled model; deterministic
lid.analyze("Nag-submit na ako ng report pero hindi pa daw na-approve.")
# {'label': 'tgl', 'conf': 0.9968, 'label_set': ['tgl'], 'abstain': False,
#  'mixture': {'tgl': 0.9998}, 'cs_level': 'none', 'cs_share': 2.3e-06, ...}

lid.analyze("Grabe yung traffic kanina sa EDSA, super late na ako sa meeting with the client.")
# {'label': 'tgl', 'conf': 0.9344, 'label_set': ['tgl'], 'mixture': {'tgl': 0.7477, 'eng': 0.252},
#  'cs_level': 'heavy', 'cs_share': 0.252, ...}

wikalid.classify("Indi ko mahibal-an kon ngaa wala sila nagkadto.")   # analysis + gate decision
# {..., 'label': 'hil', 'conf': 0.9993, 'label_final': 'hil', 'oth_sub': None,
#  'gate': 'accept_ph_other', 'gate_reason': 'accepted', 'gate_p_correct': 0.997}
```

### Command line

```bash
$ wikalid predict "Wala ko kahibalo nganong wala sila miadto."
{"text": "Wala ko kahibalo nganong wala sila miadto.", "label": "ceb", "conf": 0.9895, "abstain": false,
 "label_set": ["ceb"], "mixture": {"ceb": 1.0}, "cs_level": "none", ...}

$ echo "Ano ang pangalan mo?" | wikalid predict --gate
{"text": "Ano ang pangalan mo?", "label": "tgl", "conf": 0.9967, ..., "gate": "review", "gate_reason": "meta_low", "gate_p_correct": 0.504}

$ wikalid sidecar docs.parquet labels.parquet --workers 4    # input columns: document_id, source_name, text
```

The outputs above are abridged; values are rounded from real runs of 1.0.1.

`wikalid sidecar` writes one row per document with the model columns (`ourlid_label`, `ourlid_conf`,
`ourlid_label_set`, `ourlid_mixture`, `ourlid_cs_share`, …) and the gate columns (`ourlid_label_final`, `ourlid_oth_sub`,
`gate_v2`, `gate_v2_reason`, `gate_v2_p_correct`). `wikalid score` and `wikalid gate` run the two steps separately. `wikalid info` prints the version, labels and
weight hashes.

### Output fields

| field | meaning |
|---|---|
| `label` | Majority language of the text (standard LID convention) |
| `conf` | Temperature-scaled probability of `label` |
| `label_set`, `abstain` | Split-conformal prediction set at α = 0.10; `abstain` is true when the set is not exactly one label |
| `mixture` | Share of word tokens per language (languages ≥ 1%) |
| `cs_level`, `cs_share` | English share for non-English texts (`none` < 0.03 ≤ `light` < 0.25 ≤ `heavy`) |
| `label_final`, `oth_sub` | `label`, or `oth` + `akl`/`sgd` when the Aklanon/Surigaonon marker rule fires |
| `gate`, `gate_reason`, `gate_p_correct` | Gate decision (below) |

## The gate: accept, review, junk

A label is only useful if you know when to trust it. The gate combines named rules with a small gradient-boosted
"is this label correct?" model. That model was fit only on development data, so its scores are not tuned to the test
sets reported below.

| gate | meaning | example (real outputs) |
|---|---|---|
| `accept` | confident Tagalog | "Nag-submit na ako ng report…" → `tgl`, p_correct 0.995 |
| `accept_ph_other` | confident other Philippine language, or a marker-rule hit | "Indi ko mahibal-an…" → `hil`; an Aklanon sentence → `oth`/`akl`, reason `unsupported_ph_akl` |
| `accept_eng` | confident English | |
| `review` | the model abstained, the text is very short, a sibling language is close, or confidence is below that label's threshold | heavy Taglish → `tgl` but reason `low_conf_tgl`; "Ano ang pangalan mo?" → reason `meta_low` |
| `junk` | not language | a numbered name roster → `nol`, reason `junk_nol` |

The gate is deliberately strict. The English threshold is very high, so many clean English sentences land in `review`.
That suits a corpus where English is opt-in, but you may want your own thresholds for other uses.

Rows the gate accepts were 97.2–100% correct on every sealed evaluation set, and at least 98.5% correct on every set except BasahaCorpus (table below). On about 240k Filipino
news articles, it accepted 88.6% and sent 11.4% to review.

## Evaluation

### Method

* Each evaluation set is split 50/50 by a hash of a group key, so related items stay on the same side. Every modelling
  and threshold decision was made on the *dev* half. The numbers below are from the *sealed* half, which was never
  used for any decision.
* All baselines were run on exactly the same items with their published models.
* "Corrected gold": some gold labels in the Wikipedia-derived sets are known to be wrong (English sentences inside
  regional-language wikis, Spanish inside Chavacano). Where at least two of GlotLID, OpenLID and lid.176 agree on
  English or Spanish with probability ≥ 0.6, the gold is corrected. This rule relies on the baselines, so it can only
  favour them.
* Three sets are public and ship with the source repository: Wikipedia held-out, PH-LID-Hard v0 and the Tatoeba test
  split. The others are private, because they contain copyrighted news text or come from restricted sources.

### Results (sealed halves; accuracy on corrected gold)

| set (n) | description | WikaLID | GlotLID v3 | OpenLID-v3 | fastText lid.176 |
|---|---|---|---|---|---|
| Crawl realism (335) | Real Filipino news spans: Taglish, headlines, lists, bylines | **0.949** | 0.875 | 0.785 | 0.734 |
| News gold spans (327) | Hand-labelled news spans | **0.939** | 0.899 | 0.881 | 0.783 |
| News gold docs (148) | Hand-labelled whole articles | **0.986** | 0.959 | 0.919 | 0.818 |
| PLD read speech (1,653) | Prompts from the Philippine Languages Database, 10 languages | **0.917** | 0.834 | 0.417 | 0.341 |
| Wikipedia held-out (2,563) | Public | **0.925** | 0.907 | 0.710 | 0.664 |
| PH-LID-Hard v0 (6,379) | Public; synthetic noise: caps, SMS spelling, typos, Taglish, lists | **0.901** | 0.829 | 0.641 | 0.599 |
| BasahaCorpus (794) | Children's stories (Kinaray-a, Hiligaynon, …) | **0.739** | 0.583 | 0.026 | 0.031 |
| Tatoeba test (968) | Public; short, clean sentences | 0.928 | **0.955** | 0.758 | 0.805 |
| **Mean** | | **0.911** | 0.855 | 0.642 | 0.597 |

Paired bootstrap against GlotLID (2,000 resamples, 95% CI) on the sealed halves:

* WikaLID is better on seven sets: crawl +7.5 pts [+4.5, +11.0], gold spans +4.0 [+0.9, +7.3], gold docs +2.7
  [+0.7, +5.4], PLD +8.3 [+6.7, +9.9], Wikipedia +1.8 [+1.0, +2.7], PH-LID-Hard +7.2 [+6.3, +8.0], Basaha +15.6
  [+12.3, +18.9].
* **It is worse on Tatoeba: −2.7 pts [−4.6, −0.9].** On short, clean, textbook-style sentences, GlotLID's breadth wins.

### Human-labelled external sets (all items)

| set | WikaLID | GlotLID v3 | OpenLID-v3 | lid.176 |
|---|---|---|---|---|
| FLORES-200 via SIB-200, 9 languages (9,036) | **0.989** | 0.986 | 0.985 | 0.774 |
| Same sentences in UPPER / Title / lower case (5,388) | **0.999** | 0.975 | 0.575 | 0.522 |
| Central Bikol story sentences: Bikol recall (900) | **0.961** | 0.894 | 0.000 | 0.003 |
| 2–5 word fragments of FLORES sentences (4,320) | **0.832** | 0.771 | 0.773 | 0.590 |
| … of which 2 words (1,080) | **0.704** | 0.624 | 0.621 | 0.514 |
| Short Tatoeba sentences (543) | 0.856 | **0.866** | 0.632 | 0.685 |
| Aklanon / Surigaonon sentences → `oth` (110) | **0.345** | 0.300 | 0.082 | 0.055 |

### Gate (sealed halves)

| set | share accepted | accuracy of accepted rows |
|---|---|---|
| Crawl realism | 0.510 | 1.000 |
| News gold spans | 0.514 | 1.000 |
| News gold docs | 0.784 | 0.991 |
| PLD read speech | 0.658 | 0.985 |
| Wikipedia held-out | 0.566 | 0.989 |
| PH-LID-Hard v0 | 0.510 | 0.994 |
| BasahaCorpus | 0.358 | 0.972 |
| Tatoeba test | 0.395 | 0.995 |

These sets are deliberately hard and short, so acceptance rates are much lower than on ordinary news.

### Speed

Measured on a shared 8-core CPU, one BLAS thread per worker: about 7,000 documents per minute per core for 100-word
documents, and about 2,200 per minute per core for full news articles (~400 tokens). Throughput scales roughly
linearly with workers.

## Limitations and bias

* **Languages without a class.** Aklanon, Surigaonon and many smaller Philippine languages have no open training text.
  Hand-written marker rules move only about a third of Aklanon/Surigaonon sentences to `oth`. The rest are labelled
  `hil` or `ceb`. Do not use WikaLID to count speakers of unsupported languages.
* **The Visayan family.** Hiligaynon, Kinaray-a and Cebuano remain the main source of errors. `krj` and `tsg` are
  experimental.
* **English-heavy Taglish.** Tabloid and showbiz text with ALLCAPS leads and English quotes can be labelled `eng`.
  Usually the label set comes back empty and the gate sends the text to `review`. When WikaLID abstains on `eng`, the top
  label is right only about 22% of the time on our gold data.
* **Short strings.** Two-word inputs are 70% accurate, and a third of the remaining errors are Indonesian vs Malay.
* **Rare labels on news.** In a manual sample of news predictions, every `krj` (3/3) and `pag` (2/2) prediction was
  wrong: lists, Aklanon or Hiligaynon text. Route these to review.
* **Calibration across domains.** The conformal sets target 90% coverage at α = 0.10, but that target holds only near
  the calibration data. On the sealed halves, the gold label was inside the set for 82–91% of items on the news, speech,
  Wikipedia, PH-LID-Hard and Tatoeba sets, and for 56% on BasahaCorpus. Between 8% and 29% of sets come back empty, which
  means "no label is safe"; treat an empty set as a signal to review, not as a prediction.
* **Evaluation caveats.**
  * The news gold sets were labelled by a single, non-native annotator. A native-speaker re-annotation is planned.
  * PH-LID-Hard is synthetic and shares design ideas with the training-data augmentation.
  * BasahaCorpus accuracy counts Minasbate as `oth`, which flatters every system.
  * Real-news results should carry the most weight.
* **Register and region.** The training text is mostly encyclopedic, web and story text in standard orthographies.
  Older spellings, regional varieties and social-media slang are less well covered.

## Training data

No training text is distributed with the model. The taggers were trained on sentences from:

* Wikipedia (Tagalog, Cebuano, Waray, Ilocano, Bikol, Kapampangan, Pangasinan, Chavacano, English, Spanish, Indonesian,
  Malay and about 30 other languages): CC BY-SA 4.0.
* Tatoeba: CC BY 2.0 FR.
* FineWeb-2 web text for the lower-resource languages, filtered for lexical purity: ODC-By 1.0. Page copyright remains
  with the publishers, and this text is used for training only.
* BasahaCorpus children's stories (Imperial & Kochmar, 2023) for Kinaray-a and Hiligaynon: CC BY 4.0 as stated by its
  authors.

Training sentences are augmented with synthetic code-switching, casing, SMS-style spelling and accent variants. No
scraped news is in the training data. Existing LID tools (GlotLID, OpenLID, lid.176) were used only to detect
label noise. No third-party model weights are included.

## Versioning

* Releases follow semantic versioning. A version always pins exact weights. `wikalid verify --expect-version X`
  checks the version and the sha256 of every bundled file against `MANIFEST.json`.
* Current release: **1.0.1** (2026-11-15). Release history is in `CHANGELOG.md`.
* A retrained model will be released as a new minor version with a fresh evaluation. Weights are never swapped silently.

## License

* **Code:** Apache-2.0.
* **Model weights:** preview release, all rights reserved; redistribution is not permitted. A public licence is under
  consideration. Because Wikipedia is the largest training source, CC BY-SA 4.0 is the likely choice. See
  `MODEL_LICENSE.md`.
* **Public evaluation sets:** CC BY-SA 4.0 (Wikipedia-derived) and CC BY 2.0 FR (Tatoeba).

## Citation

```bibtex
@software{wikalid_2026,
  title   = {WikaLID: Language Identification for Philippine Languages},
  author  = {{sapinsapin}},
  year    = {2026},
  version = {1.0.1},
  url     = {https://huggingface.co/sapinsapin/wikalid}
}
```

## Acknowledgements and contact

WikaLID builds on public resources from Wikipedia and Tatoeba contributors, FineWeb-2, BasahaCorpus (Imperial &
Kochmar, 2023; stories from Let's Read Asia / The Asia Foundation), FLORES-200 / SIB-200 (used for evaluation only) and
the Philippine Languages Database (used for evaluation only). Baselines: GlotLID (Kargaran et al., 2023), OpenLID
(Burchell et al., 2023) and fastText lid.176 (Joulin et al., 2016).

Questions, error reports and offers to help with native-speaker review are welcome in the Community tab of this
repository.