|
Download README.md from sapinsapin/wikalid: direct link, hf CLI and curl.
- Browser
- Download file 15.6 kB
-
https://huggingface.co/sapinsapin/wikalid/resolve/main/README.md
- Command line
-
hf download hf://sapinsapin/wikalid/README.md
-
curl -L -o README.md https://huggingface.co/sapinsapin/wikalid/resolve/main/README.md
15.6 kB
| license: other | |
| license_name: wikalid-preview | |
| license_link: https://huggingface.co/sapinsapin/wikalid/blob/main/MODEL_LICENSE.md | |
| language: | |
| - tl | |
| - ceb | |
| - hil | |
| - war | |
| - ilo | |
| - bcl | |
| - pam | |
| - pag | |
| - cbk | |
| - krj | |
| - tsg | |
| - en | |
| - es | |
| - id | |
| - ms | |
| - multilingual | |
| library_name: wikalid | |
| pipeline_tag: text-classification | |
| tags: | |
| - language-identification | |
| - lid | |
| - philippines | |
| - filipino | |
| - tagalog | |
| - taglish | |
| - code-switching | |
| - cebuano | |
| - hiligaynon | |
| - low-resource | |
| - cpu | |
| # WikaLID | |
| **Language identification for Philippine languages, with code-switching measurement and calibrated abstention.** | |
| ## Overview | |
| General-purpose language identifiers such as GlotLID, OpenLID and fastText `lid.176` cover hundreds of languages, but | |
| they struggle with Philippine web and news text. They mix up the Visayan languages, fail on ALLCAPS headlines and | |
| tabloid style, can't separate Chavacano from Spanish, and treat Tagalog–English code-switching (Taglish) as noise. | |
| WikaLID is a small, CPU-only model built for that setting: | |
| * **17 labels**: 11 Philippine languages, 4 contact languages (English, Spanish, Indonesian, Malay), "other language" | |
| and "not a language". | |
| * **Token-level tagging.** Every word gets a language, so the output includes the language mixture and the English | |
| share. Taglish is reported as a measurement, not a separate class. | |
| * **Calibrated uncertainty.** It returns a temperature-scaled confidence and a conformal label set, plus a gate that | |
| sorts each text into *accept*, *review* or *junk*. | |
| * **Small and deterministic.** Two 26 MB int8 taggers, numpy-only inference, no GPU, no PyTorch, no fastText. | |
| It was built to label a Filipino text corpus for language-model pretraining. The gate exists so that uncertain rows | |
| can be held back instead of being silently mislabelled. | |
| ## Supported labels | |
| | code | language | notes | | |
| |---|---|---| | |
| | `tgl` | Tagalog / Filipino | Includes Taglish. Code-switching is reported in `cs_level` / `cs_share` | | |
| | `ceb` | Cebuano | | | |
| | `hil` | Hiligaynon | Most frequent confusions are with `ceb` and `krj` | | |
| | `war` | Waray | | | |
| | `ilo` | Ilocano | | | |
| | `bcl` | Bikol | Central Bikol; Rinconada is also labelled `bcl` | | |
| | `pam` | Kapampangan | | | |
| | `pag` | Pangasinan | Strong on formal text; rare in news, so treat news predictions with care | | |
| | `cbk` | Chavacano | Separated from Spanish | | |
| | `krj` | Kinaray-a | **Experimental** | | |
| | `tsg` | Tausug | **Experimental**: no human-labelled free-text test set | | |
| | `eng` | English | | | |
| | `spa` | Spanish | | | |
| | `ind` | Indonesian | Often confused with `msa` on very short strings | | |
| | `msa` | Malay | | | |
| | `oth` | any other language | Includes Aklanon / Surigaonon detected by marker rules (`oth_sub` = `akl` / `sgd`) | | |
| | `nol` | not a language | Numbered lists, name rosters, score tables (≈ ISO `zxx`) | | |
| ## Install | |
| WikaLID is distributed as a wheel with the weights bundled (about 52 MB). It needs Python ≥ 3.10 and numpy. | |
| ```bash | |
| pip install wikalid-1.0.1-py3-none-any.whl # add pyarrow for parquet batch jobs | |
| wikalid verify --expect-version 1.0.1 # checks the version and the sha256 of every weight file | |
| ``` | |
| ## Quickstart | |
| ### Python | |
| ```python | |
| import wikalid | |
| lid = wikalid.load() # bundled model; deterministic | |
| lid.analyze("Nag-submit na ako ng report pero hindi pa daw na-approve.") | |
| # {'label': 'tgl', 'conf': 0.9968, 'label_set': ['tgl'], 'abstain': False, | |
| # 'mixture': {'tgl': 0.9998}, 'cs_level': 'none', 'cs_share': 2.3e-06, ...} | |
| lid.analyze("Grabe yung traffic kanina sa EDSA, super late na ako sa meeting with the client.") | |
| # {'label': 'tgl', 'conf': 0.9344, 'label_set': ['tgl'], 'mixture': {'tgl': 0.7477, 'eng': 0.252}, | |
| # 'cs_level': 'heavy', 'cs_share': 0.252, ...} | |
| wikalid.classify("Indi ko mahibal-an kon ngaa wala sila nagkadto.") # analysis + gate decision | |
| # {..., 'label': 'hil', 'conf': 0.9993, 'label_final': 'hil', 'oth_sub': None, | |
| # 'gate': 'accept_ph_other', 'gate_reason': 'accepted', 'gate_p_correct': 0.997} | |
| ``` | |
| ### Command line | |
| ```bash | |
| $ wikalid predict "Wala ko kahibalo nganong wala sila miadto." | |
| {"text": "Wala ko kahibalo nganong wala sila miadto.", "label": "ceb", "conf": 0.9895, "abstain": false, | |
| "label_set": ["ceb"], "mixture": {"ceb": 1.0}, "cs_level": "none", ...} | |
| $ echo "Ano ang pangalan mo?" | wikalid predict --gate | |
| {"text": "Ano ang pangalan mo?", "label": "tgl", "conf": 0.9967, ..., "gate": "review", "gate_reason": "meta_low", "gate_p_correct": 0.504} | |
| $ wikalid sidecar docs.parquet labels.parquet --workers 4 # input columns: document_id, source_name, text | |
| ``` | |
| The outputs above are abridged; values are rounded from real runs of 1.0.1. | |
| `wikalid sidecar` writes one row per document with the model columns (`ourlid_label`, `ourlid_conf`, | |
| `ourlid_label_set`, `ourlid_mixture`, `ourlid_cs_share`, …) and the gate columns (`ourlid_label_final`, `ourlid_oth_sub`, | |
| `gate_v2`, `gate_v2_reason`, `gate_v2_p_correct`). `wikalid score` and `wikalid gate` run the two steps separately. `wikalid info` prints the version, labels and | |
| weight hashes. | |
| ### Output fields | |
| | field | meaning | | |
| |---|---| | |
| | `label` | Majority language of the text (standard LID convention) | | |
| | `conf` | Temperature-scaled probability of `label` | | |
| | `label_set`, `abstain` | Split-conformal prediction set at α = 0.10; `abstain` is true when the set is not exactly one label | | |
| | `mixture` | Share of word tokens per language (languages ≥ 1%) | | |
| | `cs_level`, `cs_share` | English share for non-English texts (`none` < 0.03 ≤ `light` < 0.25 ≤ `heavy`) | | |
| | `label_final`, `oth_sub` | `label`, or `oth` + `akl`/`sgd` when the Aklanon/Surigaonon marker rule fires | | |
| | `gate`, `gate_reason`, `gate_p_correct` | Gate decision (below) | | |
| ## The gate: accept, review, junk | |
| A label is only useful if you know when to trust it. The gate combines named rules with a small gradient-boosted | |
| "is this label correct?" model. That model was fit only on development data, so its scores are not tuned to the test | |
| sets reported below. | |
| | gate | meaning | example (real outputs) | | |
| |---|---|---| | |
| | `accept` | confident Tagalog | "Nag-submit na ako ng report…" → `tgl`, p_correct 0.995 | | |
| | `accept_ph_other` | confident other Philippine language, or a marker-rule hit | "Indi ko mahibal-an…" → `hil`; an Aklanon sentence → `oth`/`akl`, reason `unsupported_ph_akl` | | |
| | `accept_eng` | confident English | | | |
| | `review` | the model abstained, the text is very short, a sibling language is close, or confidence is below that label's threshold | heavy Taglish → `tgl` but reason `low_conf_tgl`; "Ano ang pangalan mo?" → reason `meta_low` | | |
| | `junk` | not language | a numbered name roster → `nol`, reason `junk_nol` | | |
| The gate is deliberately strict. The English threshold is very high, so many clean English sentences land in `review`. | |
| That suits a corpus where English is opt-in, but you may want your own thresholds for other uses. | |
| Rows the gate accepts were 97.2–100% correct on every sealed evaluation set, and at least 98.5% correct on every set except BasahaCorpus (table below). On about 240k Filipino | |
| news articles, it accepted 88.6% and sent 11.4% to review. | |
| ## Evaluation | |
| ### Method | |
| * Each evaluation set is split 50/50 by a hash of a group key, so related items stay on the same side. Every modelling | |
| and threshold decision was made on the *dev* half. The numbers below are from the *sealed* half, which was never | |
| used for any decision. | |
| * All baselines were run on exactly the same items with their published models. | |
| * "Corrected gold": some gold labels in the Wikipedia-derived sets are known to be wrong (English sentences inside | |
| regional-language wikis, Spanish inside Chavacano). Where at least two of GlotLID, OpenLID and lid.176 agree on | |
| English or Spanish with probability ≥ 0.6, the gold is corrected. This rule relies on the baselines, so it can only | |
| favour them. | |
| * Three sets are public and ship with the source repository: Wikipedia held-out, PH-LID-Hard v0 and the Tatoeba test | |
| split. The others are private, because they contain copyrighted news text or come from restricted sources. | |
| ### Results (sealed halves; accuracy on corrected gold) | |
| | set (n) | description | WikaLID | GlotLID v3 | OpenLID-v3 | fastText lid.176 | | |
| |---|---|---|---|---|---| | |
| | Crawl realism (335) | Real Filipino news spans: Taglish, headlines, lists, bylines | **0.949** | 0.875 | 0.785 | 0.734 | | |
| | News gold spans (327) | Hand-labelled news spans | **0.939** | 0.899 | 0.881 | 0.783 | | |
| | News gold docs (148) | Hand-labelled whole articles | **0.986** | 0.959 | 0.919 | 0.818 | | |
| | PLD read speech (1,653) | Prompts from the Philippine Languages Database, 10 languages | **0.917** | 0.834 | 0.417 | 0.341 | | |
| | Wikipedia held-out (2,563) | Public | **0.925** | 0.907 | 0.710 | 0.664 | | |
| | PH-LID-Hard v0 (6,379) | Public; synthetic noise: caps, SMS spelling, typos, Taglish, lists | **0.901** | 0.829 | 0.641 | 0.599 | | |
| | BasahaCorpus (794) | Children's stories (Kinaray-a, Hiligaynon, …) | **0.739** | 0.583 | 0.026 | 0.031 | | |
| | Tatoeba test (968) | Public; short, clean sentences | 0.928 | **0.955** | 0.758 | 0.805 | | |
| | **Mean** | | **0.911** | 0.855 | 0.642 | 0.597 | | |
| Paired bootstrap against GlotLID (2,000 resamples, 95% CI) on the sealed halves: | |
| * WikaLID is better on seven sets: crawl +7.5 pts [+4.5, +11.0], gold spans +4.0 [+0.9, +7.3], gold docs +2.7 | |
| [+0.7, +5.4], PLD +8.3 [+6.7, +9.9], Wikipedia +1.8 [+1.0, +2.7], PH-LID-Hard +7.2 [+6.3, +8.0], Basaha +15.6 | |
| [+12.3, +18.9]. | |
| * **It is worse on Tatoeba: −2.7 pts [−4.6, −0.9].** On short, clean, textbook-style sentences, GlotLID's breadth wins. | |
| ### Human-labelled external sets (all items) | |
| | set | WikaLID | GlotLID v3 | OpenLID-v3 | lid.176 | | |
| |---|---|---|---|---| | |
| | FLORES-200 via SIB-200, 9 languages (9,036) | **0.989** | 0.986 | 0.985 | 0.774 | | |
| | Same sentences in UPPER / Title / lower case (5,388) | **0.999** | 0.975 | 0.575 | 0.522 | | |
| | Central Bikol story sentences: Bikol recall (900) | **0.961** | 0.894 | 0.000 | 0.003 | | |
| | 2–5 word fragments of FLORES sentences (4,320) | **0.832** | 0.771 | 0.773 | 0.590 | | |
| | … of which 2 words (1,080) | **0.704** | 0.624 | 0.621 | 0.514 | | |
| | Short Tatoeba sentences (543) | 0.856 | **0.866** | 0.632 | 0.685 | | |
| | Aklanon / Surigaonon sentences → `oth` (110) | **0.345** | 0.300 | 0.082 | 0.055 | | |
| ### Gate (sealed halves) | |
| | set | share accepted | accuracy of accepted rows | | |
| |---|---|---| | |
| | Crawl realism | 0.510 | 1.000 | | |
| | News gold spans | 0.514 | 1.000 | | |
| | News gold docs | 0.784 | 0.991 | | |
| | PLD read speech | 0.658 | 0.985 | | |
| | Wikipedia held-out | 0.566 | 0.989 | | |
| | PH-LID-Hard v0 | 0.510 | 0.994 | | |
| | BasahaCorpus | 0.358 | 0.972 | | |
| | Tatoeba test | 0.395 | 0.995 | | |
| These sets are deliberately hard and short, so acceptance rates are much lower than on ordinary news. | |
| ### Speed | |
| Measured on a shared 8-core CPU, one BLAS thread per worker: about 7,000 documents per minute per core for 100-word | |
| documents, and about 2,200 per minute per core for full news articles (~400 tokens). Throughput scales roughly | |
| linearly with workers. | |
| ## Limitations and bias | |
| * **Languages without a class.** Aklanon, Surigaonon and many smaller Philippine languages have no open training text. | |
| Hand-written marker rules move only about a third of Aklanon/Surigaonon sentences to `oth`. The rest are labelled | |
| `hil` or `ceb`. Do not use WikaLID to count speakers of unsupported languages. | |
| * **The Visayan family.** Hiligaynon, Kinaray-a and Cebuano remain the main source of errors. `krj` and `tsg` are | |
| experimental. | |
| * **English-heavy Taglish.** Tabloid and showbiz text with ALLCAPS leads and English quotes can be labelled `eng`. | |
| Usually the label set comes back empty and the gate sends the text to `review`. When WikaLID abstains on `eng`, the top | |
| label is right only about 22% of the time on our gold data. | |
| * **Short strings.** Two-word inputs are 70% accurate, and a third of the remaining errors are Indonesian vs Malay. | |
| * **Rare labels on news.** In a manual sample of news predictions, every `krj` (3/3) and `pag` (2/2) prediction was | |
| wrong: lists, Aklanon or Hiligaynon text. Route these to review. | |
| * **Calibration across domains.** The conformal sets target 90% coverage at α = 0.10, but that target holds only near | |
| the calibration data. On the sealed halves, the gold label was inside the set for 82–91% of items on the news, speech, | |
| Wikipedia, PH-LID-Hard and Tatoeba sets, and for 56% on BasahaCorpus. Between 8% and 29% of sets come back empty, which | |
| means "no label is safe"; treat an empty set as a signal to review, not as a prediction. | |
| * **Evaluation caveats.** | |
| * The news gold sets were labelled by a single, non-native annotator. A native-speaker re-annotation is planned. | |
| * PH-LID-Hard is synthetic and shares design ideas with the training-data augmentation. | |
| * BasahaCorpus accuracy counts Minasbate as `oth`, which flatters every system. | |
| * Real-news results should carry the most weight. | |
| * **Register and region.** The training text is mostly encyclopedic, web and story text in standard orthographies. | |
| Older spellings, regional varieties and social-media slang are less well covered. | |
| ## Training data | |
| No training text is distributed with the model. The taggers were trained on sentences from: | |
| * Wikipedia (Tagalog, Cebuano, Waray, Ilocano, Bikol, Kapampangan, Pangasinan, Chavacano, English, Spanish, Indonesian, | |
| Malay and about 30 other languages): CC BY-SA 4.0. | |
| * Tatoeba: CC BY 2.0 FR. | |
| * FineWeb-2 web text for the lower-resource languages, filtered for lexical purity: ODC-By 1.0. Page copyright remains | |
| with the publishers, and this text is used for training only. | |
| * BasahaCorpus children's stories (Imperial & Kochmar, 2023) for Kinaray-a and Hiligaynon: CC BY 4.0 as stated by its | |
| authors. | |
| Training sentences are augmented with synthetic code-switching, casing, SMS-style spelling and accent variants. No | |
| scraped news is in the training data. Existing LID tools (GlotLID, OpenLID, lid.176) were used only to detect | |
| label noise. No third-party model weights are included. | |
| ## Versioning | |
| * Releases follow semantic versioning. A version always pins exact weights. `wikalid verify --expect-version X` | |
| checks the version and the sha256 of every bundled file against `MANIFEST.json`. | |
| * Current release: **1.0.1** (2026-11-15). Release history is in `CHANGELOG.md`. | |
| * A retrained model will be released as a new minor version with a fresh evaluation. Weights are never swapped silently. | |
| ## License | |
| * **Code:** Apache-2.0. | |
| * **Model weights:** preview release, all rights reserved; redistribution is not permitted. A public licence is under | |
| consideration. Because Wikipedia is the largest training source, CC BY-SA 4.0 is the likely choice. See | |
| `MODEL_LICENSE.md`. | |
| * **Public evaluation sets:** CC BY-SA 4.0 (Wikipedia-derived) and CC BY 2.0 FR (Tatoeba). | |
| ## Citation | |
| ```bibtex | |
| @software{wikalid_2026, | |
| title = {WikaLID: Language Identification for Philippine Languages}, | |
| author = {{sapinsapin}}, | |
| year = {2026}, | |
| version = {1.0.1}, | |
| url = {https://huggingface.co/sapinsapin/wikalid} | |
| } | |
| ``` | |
| ## Acknowledgements and contact | |
| WikaLID builds on public resources from Wikipedia and Tatoeba contributors, FineWeb-2, BasahaCorpus (Imperial & | |
| Kochmar, 2023; stories from Let's Read Asia / The Asia Foundation), FLORES-200 / SIB-200 (used for evaluation only) and | |
| the Philippine Languages Database (used for evaluation only). Baselines: GlotLID (Kargaran et al., 2023), OpenLID | |
| (Burchell et al., 2023) and fastText lid.176 (Joulin et al., 2016). | |
| Questions, error reports and offers to help with native-speaker review are welcome in the Community tab of this | |
| repository. | |