WikaLID

Language identification for Philippine languages, with code-switching measurement and calibrated abstention.

Overview

General-purpose language identifiers such as GlotLID, OpenLID and fastText lid.176 cover hundreds of languages, but they struggle with Philippine web and news text. They mix up the Visayan languages, fail on ALLCAPS headlines and tabloid style, can't separate Chavacano from Spanish, and treat Tagalog–English code-switching (Taglish) as noise. WikaLID is a small, CPU-only model built for that setting:

  • 17 labels: 11 Philippine languages, 4 contact languages (English, Spanish, Indonesian, Malay), "other language" and "not a language".
  • Token-level tagging. Every word gets a language, so the output includes the language mixture and the English share. Taglish is reported as a measurement, not a separate class.
  • Calibrated uncertainty. It returns a temperature-scaled confidence and a conformal label set, plus a gate that sorts each text into accept, review or junk.
  • Small and deterministic. Two 26 MB int8 taggers, numpy-only inference, no GPU, no PyTorch, no fastText.

It was built to label a Filipino text corpus for language-model pretraining. The gate exists so that uncertain rows can be held back instead of being silently mislabelled.

Supported labels

code language notes
tgl Tagalog / Filipino Includes Taglish. Code-switching is reported in cs_level / cs_share
ceb Cebuano
hil Hiligaynon Most frequent confusions are with ceb and krj
war Waray
ilo Ilocano
bcl Bikol Central Bikol; Rinconada is also labelled bcl
pam Kapampangan
pag Pangasinan Strong on formal text; rare in news, so treat news predictions with care
cbk Chavacano Separated from Spanish
krj Kinaray-a Experimental
tsg Tausug Experimental: no human-labelled free-text test set
eng English
spa Spanish
ind Indonesian Often confused with msa on very short strings
msa Malay
oth any other language Includes Aklanon / Surigaonon detected by marker rules (oth_sub = akl / sgd)
nol not a language Numbered lists, name rosters, score tables (β‰ˆ ISO zxx)

Install

WikaLID is distributed as a wheel with the weights bundled (about 52 MB). It needs Python β‰₯ 3.10 and numpy.

pip install wikalid-1.0.1-py3-none-any.whl            # add pyarrow for parquet batch jobs
wikalid verify --expect-version 1.0.1                 # checks the version and the sha256 of every weight file

Quickstart

Python

import wikalid

lid = wikalid.load()                       # bundled model; deterministic
lid.analyze("Nag-submit na ako ng report pero hindi pa daw na-approve.")
# {'label': 'tgl', 'conf': 0.9968, 'label_set': ['tgl'], 'abstain': False,
#  'mixture': {'tgl': 0.9998}, 'cs_level': 'none', 'cs_share': 2.3e-06, ...}

lid.analyze("Grabe yung traffic kanina sa EDSA, super late na ako sa meeting with the client.")
# {'label': 'tgl', 'conf': 0.9344, 'label_set': ['tgl'], 'mixture': {'tgl': 0.7477, 'eng': 0.252},
#  'cs_level': 'heavy', 'cs_share': 0.252, ...}

wikalid.classify("Indi ko mahibal-an kon ngaa wala sila nagkadto.")   # analysis + gate decision
# {..., 'label': 'hil', 'conf': 0.9993, 'label_final': 'hil', 'oth_sub': None,
#  'gate': 'accept_ph_other', 'gate_reason': 'accepted', 'gate_p_correct': 0.997}

Command line

$ wikalid predict "Wala ko kahibalo nganong wala sila miadto."
{"text": "Wala ko kahibalo nganong wala sila miadto.", "label": "ceb", "conf": 0.9895, "abstain": false,
 "label_set": ["ceb"], "mixture": {"ceb": 1.0}, "cs_level": "none", ...}

$ echo "Ano ang pangalan mo?" | wikalid predict --gate
{"text": "Ano ang pangalan mo?", "label": "tgl", "conf": 0.9967, ..., "gate": "review", "gate_reason": "meta_low", "gate_p_correct": 0.504}

$ wikalid sidecar docs.parquet labels.parquet --workers 4    # input columns: document_id, source_name, text

The outputs above are abridged; values are rounded from real runs of 1.0.1.

wikalid sidecar writes one row per document with the model columns (ourlid_label, ourlid_conf, ourlid_label_set, ourlid_mixture, ourlid_cs_share, …) and the gate columns (ourlid_label_final, ourlid_oth_sub, gate_v2, gate_v2_reason, gate_v2_p_correct). wikalid score and wikalid gate run the two steps separately. wikalid info prints the version, labels and weight hashes.

Output fields

field meaning
label Majority language of the text (standard LID convention)
conf Temperature-scaled probability of label
label_set, abstain Split-conformal prediction set at Ξ± = 0.10; abstain is true when the set is not exactly one label
mixture Share of word tokens per language (languages β‰₯ 1%)
cs_level, cs_share English share for non-English texts (none < 0.03 ≀ light < 0.25 ≀ heavy)
label_final, oth_sub label, or oth + akl/sgd when the Aklanon/Surigaonon marker rule fires
gate, gate_reason, gate_p_correct Gate decision (below)

The gate: accept, review, junk

A label is only useful if you know when to trust it. The gate combines named rules with a small gradient-boosted "is this label correct?" model. That model was fit only on development data, so its scores are not tuned to the test sets reported below.

gate meaning example (real outputs)
accept confident Tagalog "Nag-submit na ako ng report…" β†’ tgl, p_correct 0.995
accept_ph_other confident other Philippine language, or a marker-rule hit "Indi ko mahibal-an…" β†’ hil; an Aklanon sentence β†’ oth/akl, reason unsupported_ph_akl
accept_eng confident English
review the model abstained, the text is very short, a sibling language is close, or confidence is below that label's threshold heavy Taglish β†’ tgl but reason low_conf_tgl; "Ano ang pangalan mo?" β†’ reason meta_low
junk not language a numbered name roster β†’ nol, reason junk_nol

The gate is deliberately strict. The English threshold is very high, so many clean English sentences land in review. That suits a corpus where English is opt-in, but you may want your own thresholds for other uses.

Rows the gate accepts were 97.2–100% correct on every sealed evaluation set, and at least 98.5% correct on every set except BasahaCorpus (table below). On about 240k Filipino news articles, it accepted 88.6% and sent 11.4% to review.

Evaluation

Method

  • Each evaluation set is split 50/50 by a hash of a group key, so related items stay on the same side. Every modelling and threshold decision was made on the dev half. The numbers below are from the sealed half, which was never used for any decision.
  • All baselines were run on exactly the same items with their published models.
  • "Corrected gold": some gold labels in the Wikipedia-derived sets are known to be wrong (English sentences inside regional-language wikis, Spanish inside Chavacano). Where at least two of GlotLID, OpenLID and lid.176 agree on English or Spanish with probability β‰₯ 0.6, the gold is corrected. This rule relies on the baselines, so it can only favour them.
  • Three sets are public and ship with the source repository: Wikipedia held-out, PH-LID-Hard v0 and the Tatoeba test split. The others are private, because they contain copyrighted news text or come from restricted sources.

Results (sealed halves; accuracy on corrected gold)

set (n) description WikaLID GlotLID v3 OpenLID-v3 fastText lid.176
Crawl realism (335) Real Filipino news spans: Taglish, headlines, lists, bylines 0.949 0.875 0.785 0.734
News gold spans (327) Hand-labelled news spans 0.939 0.899 0.881 0.783
News gold docs (148) Hand-labelled whole articles 0.986 0.959 0.919 0.818
PLD read speech (1,653) Prompts from the Philippine Languages Database, 10 languages 0.917 0.834 0.417 0.341
Wikipedia held-out (2,563) Public 0.925 0.907 0.710 0.664
PH-LID-Hard v0 (6,379) Public; synthetic noise: caps, SMS spelling, typos, Taglish, lists 0.901 0.829 0.641 0.599
BasahaCorpus (794) Children's stories (Kinaray-a, Hiligaynon, …) 0.739 0.583 0.026 0.031
Tatoeba test (968) Public; short, clean sentences 0.928 0.955 0.758 0.805
Mean 0.911 0.855 0.642 0.597

Paired bootstrap against GlotLID (2,000 resamples, 95% CI) on the sealed halves:

  • WikaLID is better on seven sets: crawl +7.5 pts [+4.5, +11.0], gold spans +4.0 [+0.9, +7.3], gold docs +2.7 [+0.7, +5.4], PLD +8.3 [+6.7, +9.9], Wikipedia +1.8 [+1.0, +2.7], PH-LID-Hard +7.2 [+6.3, +8.0], Basaha +15.6 [+12.3, +18.9].
  • It is worse on Tatoeba: βˆ’2.7 pts [βˆ’4.6, βˆ’0.9]. On short, clean, textbook-style sentences, GlotLID's breadth wins.

Human-labelled external sets (all items)

set WikaLID GlotLID v3 OpenLID-v3 lid.176
FLORES-200 via SIB-200, 9 languages (9,036) 0.989 0.986 0.985 0.774
Same sentences in UPPER / Title / lower case (5,388) 0.999 0.975 0.575 0.522
Central Bikol story sentences: Bikol recall (900) 0.961 0.894 0.000 0.003
2–5 word fragments of FLORES sentences (4,320) 0.832 0.771 0.773 0.590
… of which 2 words (1,080) 0.704 0.624 0.621 0.514
Short Tatoeba sentences (543) 0.856 0.866 0.632 0.685
Aklanon / Surigaonon sentences β†’ oth (110) 0.345 0.300 0.082 0.055

Gate (sealed halves)

set share accepted accuracy of accepted rows
Crawl realism 0.510 1.000
News gold spans 0.514 1.000
News gold docs 0.784 0.991
PLD read speech 0.658 0.985
Wikipedia held-out 0.566 0.989
PH-LID-Hard v0 0.510 0.994
BasahaCorpus 0.358 0.972
Tatoeba test 0.395 0.995

These sets are deliberately hard and short, so acceptance rates are much lower than on ordinary news.

Speed

Measured on a shared 8-core CPU, one BLAS thread per worker: about 7,000 documents per minute per core for 100-word documents, and about 2,200 per minute per core for full news articles (~400 tokens). Throughput scales roughly linearly with workers.

Limitations and bias

  • Languages without a class. Aklanon, Surigaonon and many smaller Philippine languages have no open training text. Hand-written marker rules move only about a third of Aklanon/Surigaonon sentences to oth. The rest are labelled hil or ceb. Do not use WikaLID to count speakers of unsupported languages.
  • The Visayan family. Hiligaynon, Kinaray-a and Cebuano remain the main source of errors. krj and tsg are experimental.
  • English-heavy Taglish. Tabloid and showbiz text with ALLCAPS leads and English quotes can be labelled eng. Usually the label set comes back empty and the gate sends the text to review. When WikaLID abstains on eng, the top label is right only about 22% of the time on our gold data.
  • Short strings. Two-word inputs are 70% accurate, and a third of the remaining errors are Indonesian vs Malay.
  • Rare labels on news. In a manual sample of news predictions, every krj (3/3) and pag (2/2) prediction was wrong: lists, Aklanon or Hiligaynon text. Route these to review.
  • Calibration across domains. The conformal sets target 90% coverage at Ξ± = 0.10, but that target holds only near the calibration data. On the sealed halves, the gold label was inside the set for 82–91% of items on the news, speech, Wikipedia, PH-LID-Hard and Tatoeba sets, and for 56% on BasahaCorpus. Between 8% and 29% of sets come back empty, which means "no label is safe"; treat an empty set as a signal to review, not as a prediction.
  • Evaluation caveats.
    • The news gold sets were labelled by a single, non-native annotator. A native-speaker re-annotation is planned.
    • PH-LID-Hard is synthetic and shares design ideas with the training-data augmentation.
    • BasahaCorpus accuracy counts Minasbate as oth, which flatters every system.
    • Real-news results should carry the most weight.
  • Register and region. The training text is mostly encyclopedic, web and story text in standard orthographies. Older spellings, regional varieties and social-media slang are less well covered.

Training data

No training text is distributed with the model. The taggers were trained on sentences from:

  • Wikipedia (Tagalog, Cebuano, Waray, Ilocano, Bikol, Kapampangan, Pangasinan, Chavacano, English, Spanish, Indonesian, Malay and about 30 other languages): CC BY-SA 4.0.
  • Tatoeba: CC BY 2.0 FR.
  • FineWeb-2 web text for the lower-resource languages, filtered for lexical purity: ODC-By 1.0. Page copyright remains with the publishers, and this text is used for training only.
  • BasahaCorpus children's stories (Imperial & Kochmar, 2023) for Kinaray-a and Hiligaynon: CC BY 4.0 as stated by its authors.

Training sentences are augmented with synthetic code-switching, casing, SMS-style spelling and accent variants. No scraped news is in the training data. Existing LID tools (GlotLID, OpenLID, lid.176) were used only to detect label noise. No third-party model weights are included.

Versioning

  • Releases follow semantic versioning. A version always pins exact weights. wikalid verify --expect-version X checks the version and the sha256 of every bundled file against MANIFEST.json.
  • Current release: 1.0.1 (2026-11-15). Release history is in CHANGELOG.md.
  • A retrained model will be released as a new minor version with a fresh evaluation. Weights are never swapped silently.

License

  • Code: Apache-2.0.
  • Model weights: preview release, all rights reserved; redistribution is not permitted. A public licence is under consideration. Because Wikipedia is the largest training source, CC BY-SA 4.0 is the likely choice. See MODEL_LICENSE.md.
  • Public evaluation sets: CC BY-SA 4.0 (Wikipedia-derived) and CC BY 2.0 FR (Tatoeba).

Citation

@software{wikalid_2026,
  title   = {WikaLID: Language Identification for Philippine Languages},
  author  = {{sapinsapin}},
  year    = {2026},
  version = {1.0.1},
  url     = {https://huggingface.co/sapinsapin/wikalid}
}

Acknowledgements and contact

WikaLID builds on public resources from Wikipedia and Tatoeba contributors, FineWeb-2, BasahaCorpus (Imperial & Kochmar, 2023; stories from Let's Read Asia / The Asia Foundation), FLORES-200 / SIB-200 (used for evaluation only) and the Philippine Languages Database (used for evaluation only). Baselines: GlotLID (Kargaran et al., 2023), OpenLID (Burchell et al., 2023) and fastText lid.176 (Joulin et al., 2016).

Questions, error reports and offers to help with native-speaker review are welcome in the Community tab of this repository.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support