wikalid / docs /TECH_REPORT.md
jpaulpoliquit's picture
OurLID 1.0.0 (private preview): package, weights, model card, reproducible eval
863c520 verified
|
Raw History Blame Contribute Delete
32.5 kB

OurLID: a token-level, mixture-aware and calibrated language identifier for Philippine text

Draft v0.1 - 2026-10-10 - not peer reviewed. All numbers are from eval/ in this repository.

Abstract

Off-the-shelf language identification (LID) is built for documents and closed label sets, yet Philippine text is code-mixed (Taglish), name-dense, noisily spelled and includes many closely related languages (Cebuano/Hiligaynon/Waray; Bikol/Tagalog). We present OurLID, a 26 MB numpy-only system that classifies tokens with a hashed n-gram tagger, aggregates them into a language mixture, scores it with a small head using induced closed-class lexicon evidence, and abstains with split-conformal label sets. Trained only on licence-clean or pointer-only data and synthetic code-switching, it matches or beats GlotLID v3 on a new permissive hard-case benchmark (0.899 vs 0.797 accuracy), on held-out Wikipedia (0.917 vs 0.870) and on our gold news docs (0.963 vs 0.949), ties the best system on gold news spans (0.917), but loses on Hiligaynon spans, read-speech Cebuano and on rejecting unseen related languages. Ablations show that the components we expected to help most (code-switch synthesis, augmentation, morphology features) are not significant on real text, whereas removing document-level context pooling (+4 pts on PLD) and a learned head (+6 pts on the hard benchmark) are.

1 Problem and motivation

Four structural failures of existing LID on Philippine text, each measured on our private gold news spans (593 spans, single annotator) with the same harness for all tools (per-span failure list kept private because it quotes news text; summary here): (i) no not-a-language class: 0 % of 20 list/score/roster spans correct for all tools except OpenLID-v3 zxx (partial); (ii) name-dense Tagalog -> bcl/pag (GlotLID 14, halo-lid 9 errors); (iii) Taglish (heavy CS spans 17-33 % error; lid.176 calls 43 Tagalog spans English); (iv) hil->ceb for every tool without a hil class (26-27 errors of 30). Quality audits of web corpora (Caswell et al. 2020; Kreutzer et al. 2022) show the same pattern: high held-out F1, low human-judged precision for low-resource languages.

2 Related work (verified sources; table in study/STACK_STUDY.md)

fastText lid.176 / "Bag of Tricks" (Joulin et al. 2016); langid.py (Lui & Baldwin 2012); CLD2/CLD3; lingua; GlotLID (Kargaran et al., EMNLP Findings 2023); OpenLID and OpenLID-v3 (Burchell et al. 2023; Fedorova et al., VarDial 2026); NLLB lid201; MMS-LID (audio); Google LangID/semi-supervised filtering (Caswell 2020; Bapna 2022); IndicLID cascade (Madhani et al. 2023); FineWeb-2 filtering (arXiv 2506.20920); LID survey (Jauhiainen et al. 2019); similar-language/variety identification (VarDial 2023 findings; Haas & Derczynski 2021); word-level code-switching LID (LinCE; TweetTaglish, Herrera et al. 2022; TagLID); Fightin' Words log-odds (Monroe et al. 2008); conformal prediction (Angelopoulos & Bates 2021); FLORES+/OLDI (CC BY-SA 4.0; ceb, ilo, pag, war present, hil/bcl/pam absent).

3 Method

Tokens. Script-aware tokenizer keeping hyphenated Tagalog affixation (nag-submit) and quotes. Per-token features hashed to 2^19 buckets: char 1-5-grams with boundary markers, folded word identity, morphology signatures (prefix/infix/suffix classes such as -um-/-in-, reduplication, clitics), word-shape and script. Tagger. EmbeddingBag(2^19 x 48) -> linear(60->128) -> conv(k5) -> conv(k3, dilation 2) with residual -> linear -> 17-way softmax (11 PH classes, eng, spa, ind, msa, oth, neutral x). No document-level pooling (Section 5). Supervision. Monolingual sentences with token labels (neutral for punctuation/digits/capitalised non-initial tokens); programmatic code-switch synthesis (English chunk insertion, inter-sentential mixing, quote frames, affixed English), augmentation (case, SMS spelling, typos, #/@ names, emoji, listing noise); weak purification: lowercase tokens in non-English training sentences that belong to a contrast-derived pure-English list (1,238 types) are relabelled eng. Document layer. Token posteriors -> mixture over all words and over unquoted words, lexical mass, entropy, quote fraction; closed-class lexicon block = mean log-odds evidence per language from words that are discriminative (Monroe z >= 2) in both Wikipedia and web samples; logistic-regression head (57 features) trained on 24k synthetic documents; temperature scaling; Mondrian split-conformal thresholds per predicted class on 8k calibration documents. cs_level from English share (<0.03 none, <0.25 light, else heavy). Data. docs/DATA_CARD.md: 2.64M training sentences (R: Wikipedia/Tatoeba redistributable; W/G: FineWeb-2 web pointer-only), domain-disjoint dev/test.

4 Experimental setup

Test sets: private gold news spans/docs (single annotator, non-native), PLD read-speech sentences (3,222; halo-lid trained on PLD), held-out Wikipedia/Tatoeba-source sentences (5,037, label-noisy for pam/pag/cbk), PH-LID-Hard v0 (12,880, synthetic transformations of Tier-R test text; docs/BENCHMARK_CARD.md). Baselines run by us: GlotLID v3, OpenLID-v3, halo-lid, lingua, lid.176, lid201, langid.py, and the earlier "phlid" stack (GlotLID+halo-lid+OpenLID+lexicon LR). Metrics: accuracy, macro-F1 (classes n>=20), ECE (10 bins), AURC, per-category accuracy, conformal coverage; paired bootstrap 95 % CIs (500 resamples) for ablations. quoted_english is excluded from headline numbers because its label (frame language) is a convention.

5 Results

Main table (acc / macro-F1; eval/FINAL_RESULTS.md):

system gold spans gold docs PLD held-out Wiki PH-LID-Hard
OurLID v0.1 0.917 / 0.807 0.963 / 0.985 0.888 / 0.875 0.917 / 0.920 0.899 / 0.879
phlid stack 0.917 / 0.769 0.939 / 0.979 (0.934 / 0.948) 0.704 / 0.688 0.717 / 0.596
GlotLID v3 0.899 / 0.743 0.949 / 0.957 0.841 / 0.848 0.870 / 0.873 0.797 / 0.729
OpenLID-v3 0.865 / 0.598 0.895 / 0.827 0.436 / 0.405 0.676 / 0.607 0.617 / 0.539
halo-lid 0.890 / 0.728 0.942 / 0.970 (0.987 / 0.988) 0.554 / 0.492 0.662 / 0.468
lingua / lid.176 / lid201 / langid 0.79 / 0.78 / 0.83 / 0.76 acc 0.80 / 0.80 / 0.88 / 0.77 0.20 / 0.36 / 0.44 / 0.19 0.41 / 0.62 / 0.67 / 0.38 0.38 / 0.57 / 0.62 / 0.35
Parentheses: system trained on PLD (leakage).
Calibration. ECE 0.021-0.068, AURC 0.001-0.025. Abstention (eval/ABSTAIN.md): at alpha=0.05 set coverage 0.88 (Hard), 0.89 (gold spans), 0.95 (gold docs), 0.87 (PLD), 0.91 (Wikipedia) - below the nominal 0.95 because calibration is synthetic; accuracy on singleton sets 0.92-0.97, abstaining on 10-25 % of items.
Ablations (320k items, 1 epoch; eval/ABLATIONS.md). Global mean-pooled context fell 4.0 pts on PLD (CI 2.8-5.0) and 2.0 pts on gold spans versus no pooling; local +-8 pooling was in between. Code-switch synthesis, augmentation, morphology features: differences within +-1 pt, CIs include 0 on real sets (augmentation matters for ALLCAPS: 0.88 vs 0.93 on Hard). Uniform class sampling hurt PLD (-1.6 pts). English purification: token-level English confidence in an English quote 0.40 -> 0.98, gold +1.3 pts, PLD -0.9 pts.
Per-category (Hard): listings 0.97 (only OpenLID 0.56 else 0.00), ALLCAPS 0.94, SMS 0.91, short strings 0.74 (GlotLID 0.60), clean sentences 0.92 (GlotLID 0.87).
Errors we make: gold: oth->hil 11 (unsupported Aklanon/Chavacano-like text), eng->tgl 8, nol->tgl 4; PLD: bcl->tgl 62, ceb->tgl 53, ceb->bcl 35; Wikipedia: cbk<->spa, ind<->msa.
Speed (one process): ~19k docs/min at ~100 words, ~60k docs/min at 25 words, ~5.9k docs/min at ~400 tokens; RSS 115 MB.

6 Release-scale application

v0.2.9.5 news release (229,719 documents): see docs/RELEASE_SIDECAR.md for label distribution and the disagreement analysis against the earlier phlid sidecar.

7 Limitations

Synthetic calibration, single non-native annotator, benchmark partly shares design with training, Wikipedia-trained baselines favoured on Wikipedia text but so are we (same sources), no real Taglish mixture gold, no unseen-language rejection beyond lists/noise, uneven classes (krj/tsg experimental; hil has no redistributable text), no speech/romanisation/other-script evaluation, weights licence unresolved.

8 Ethics and community note

LID mistakes silently erase regional-language text from corpora; the abstain output should stay on. Native-speaker annotation is needed (docs/NATIVE_REVIEW.md). No personal data is redistributed; web training text is pointer-only. NC/GPL models were used for comparison only.

9 Reproducibility

docs/REPRODUCE.md, fixed seeds, 12 unit/integration tests.

Round 2 addendum (draft)

We add a private crawl-realism held-out set (661 scored spans from 12 news sources and 23 hard classes), a 1,712-token word-level Taglish gold set, near-neighbour negatives (FineWeb-2 languages of the Philippines plus synthetic Aklanon-like text), v3 synthetic augmentations and real-data conformal calibration. On crawl-realism (ambiguous-frame items scored leniently, macro-F1 over classes with >=8 items) OurLID r2 reaches 0.91 accuracy / 0.851 macro-F1 versus 0.90 / 0.736 for r1, 0.88 / 0.742 for the phlid stack and 0.88 / 0.726 for GlotLID (strict forced-label accuracy r1 0.877 -> r2 0.894); improvements concentrate on nol and unseen neighbouring languages and are small elsewhere (+1.7 to +2.0 pt on real news sets, -0.4 pt on PH-LID-Hard). English recall on short and ALLCAPS crawl spans (0.79) and 1-5 word strings (0.59 for all tools) remain open. See eval/R2_VS_R1.md, eval/CRAWL_REALISM.md, eval/GATE_POLICY.md.

OurLID improve loop (rounds 0-6; best = c3: 2-tagger ensemble + head, r3 inference options)

Protocol: every eval set is split 50/50 (md5 of group id; eval/sets.py) into a dev half used for all decisions and a sealed half used only for informational snapshots. Gold labels of wiki/hard/pld/cbk are noisy (English text labelled pam/bcl/ilo/war, Spanish text labelled Chavacano); eval/loop_report.py therefore reports raw and corrected gold (non-eng gold with >=2 of GlotLID/OpenLID/lid176 saying eng -> eng; cbk gold with >=2 saying spa and no Chavacano marker -> spa). Caveat: the corrected gold shares detectors with the round-1 training-label cleaning. Full log: eval/LOOP_LOG.md.

round change outcome (dev halves, paired bootstrap vs previous kept round; * = 95% CI excludes 0) kept
0 r3c baseline (case-fold view, HMM smoothing, +English-with-names head data) crawl +2.8*, goldspan +1.1, golddoc +2.7 vs r2; raw wiki -1.0 / hard -0.9 but corrected-gold wiki +0.6 / hard +0.5*: the "tradeoff" was label noise yes
1 consensus label cleaning of the training corpus (GlotLID/OpenLID/lid176 as detectors; 36k docs -> eng, 2.3k cbk -> spa) corrected wiki +0.8*, hard +0.5*; raw wiki/hard -1 (consistently-correct English no longer called pam) yes
1b stress-accent-stripped second view at inference PLD +1.7* (Cebuano with stress marks 31/113 -> 98/113 recalled); others +-0.1 yes
2 story-disjoint BasahaCorpus (krj, hil real; Minasbate/Rinconada -> oth) + accent-injection augmentation krj F1 0.78->0.83, hil 0.72->0.80; basaha acc +19* (inflated by mbt/rin->oth); crawl +1.8, golddoc +1.4, goldspan -1.5 ns, tato -0.6 ns yes
3 2-seed tagger ensemble + ensemble head/conformal hard +0.8*, basaha +2.4*, crawl +0.9, tato +0.7 yes
4 speed: shared featurisation, 1 BLAS thread / worker identical labels; 39k docs/min (100-word docs, 7 workers, loaded box) yes
5 + FineWeb-2 _removed minority data (pag/pam/cbk/ilo/hil/bcl/krj/war/tsg, ~200k sent.) + 60k Tatoeba English no significant gain; crawl -1.5, golddoc -1.4, tato -1.4*, cbk(hard) F1 .76->.68; krj/hil +.02-.03 no (reverted)
6 gate v2: learned tree head + named rules over c3 outputs (dev-fit) sealed precision of accepted 0.964 -> 0.987 at equal coverage (64%); pld 0.85 -> 0.96 yes

BasahaCorpus licence check (2026-10-10): the EMNLP 2023 paper (Imperial & Kochmar, sec. 2.1) states the stories were obtained from Let's Read Asia 'with explicit permission ... licensed under Creative Commons BY 4.0'; the GitHub repo (imperialite/BasahaCorpus-HierarchicalCrosslingualARA) carries no LICENSE file, only the permission statement. Status: licence stated by the authors, not independently verified at the upstream site / per story -> attribution required (Imperial & Kochmar 2023; Let's Read Asia / The Asia Foundation); if a downstream user needs a clean chain, retrain without data/work/basaha_train.parquet (scripts/build_basaha_split.py) - expected cost: Kinaray-a F1 ~0.78 instead of ~0.84, Hiligaynon ~0.72 instead of ~0.82 (round 2/3 ablation).

Training-label cleaning uses GlotLID/OpenLID/lid176 only as detectors (rule-based relabelling); no external weights are in the model. Training data of c3: Wikipedia (CC BY-SA), Tatoeba (CC BY), FineWeb-2 (ODC-By, GlotLID-labelled, lexical-purity filtered), BasahaCorpus (see above). No scraped news is in the training data. Eval sets crawl-realism / gold spans / gold docs / PLD are private and not published.

Results (accuracy raw/corrected gold, macro-F1 corrected in brackets)

sealed halves: accuracy raw / corrected gold (macro-F1 corrected)

system crawl(335) goldspan(327) golddoc(148) pld(1653) wiki(2563) hard(6379) basaha(794) tato(968) mean corrected
c3 0.943/0.943 (0.94) 0.942/0.942 (0.94) 1.000/1.000 (1.00) 0.911/0.913 (0.92) 0.898/0.929 (0.93) 0.878/0.900 (0.88) 0.759/0.759 (0.58) 0.928/0.925 (0.94) 0.914
c2 0.934/0.934 (0.93) 0.927/0.927 (0.92) 0.993/0.993 (1.00) 0.909/0.910 (0.91) 0.895/0.925 (0.93) 0.873/0.896 (0.87) 0.722/0.722 (0.56) 0.925/0.924 (0.93) 0.904
ourlid_r2 0.919/0.919 (0.91) 0.930/0.930 (0.94) 1.000/1.000 (1.00) 0.884/0.885 (0.88) 0.916/0.909 (0.91) 0.891/0.885 (0.86) 0.540/0.540 (0.54) 0.937/0.934 (0.94) 0.875
glotlid 0.875/0.875 (0.93) 0.899/0.899 (0.92) 0.959/0.959 (1.00) 0.830/0.834 (0.84) 0.869/0.907 (0.91) 0.799/0.829 (0.76) 0.583/0.583 (0.64) 0.949/0.955 (0.97) 0.855
openlid3 0.785/0.785 (0.62) 0.881/0.881 (0.87) 0.919/0.919 (0.99) 0.417/0.417 (0.40) 0.676/0.710 (0.63) 0.619/0.641 (0.55) 0.026/0.026 (0.01) 0.754/0.758 (0.68) 0.642
lid176 0.734/0.734 (0.57) 0.783/0.783 (0.74) 0.818/0.818 (0.90) 0.337/0.341 (0.29) 0.620/0.664 (0.55) 0.567/0.599 (0.45) 0.031/0.031 (0.03) 0.801/0.805 (0.80) 0.597
halolid 0.863/0.863 (0.93) 0.893/0.893 (0.91) 0.953/0.953 (1.00) 0.987/0.985 (0.98) 0.542/0.566 (0.50) 0.659/0.679 (0.48) 0.361/0.361 (0.30) n/a 0.757
lid201 0.797/0.797 (0.62) 0.841/0.841 (0.84) 0.912/0.912 (0.99) 0.420/0.421 (0.38) 0.675/0.700 (0.61) 0.619/0.637 (0.52) 0.055/0.055 (0.03) n/a 0.623
lingua 0.740/0.740 (0.43) 0.789/0.789 (0.60) 0.811/0.811 (0.90) 0.191/0.193 (0.12) 0.406/0.448 (0.30) 0.376/0.405 (0.26) 0.052/0.052 (0.04) n/a 0.491

dev halves: accuracy raw / corrected gold (macro-F1 corrected)

system crawl(326) goldspan(266) golddoc(146) pld(1569) wiki(2474) hard(6141) basaha(806) tato(879) mean corrected
c3 0.960/0.957 (0.94) 0.955/0.955 (0.95) 1.000/1.000 (1.00) 0.902/0.902 (0.91) 0.893/0.922 (0.92) 0.883/0.907 (0.89) 0.782/0.782 (0.59) 0.941/0.937 (0.95) 0.920
c2 0.951/0.948 (0.93) 0.951/0.951 (0.94) 1.000/1.000 (1.00) 0.902/0.903 (0.91) 0.892/0.921 (0.92) 0.875/0.898 (0.88) 0.758/0.758 (0.58) 0.932/0.931 (0.94) 0.914
ourlid_r2 0.905/0.902 (0.88) 0.940/0.940 (0.91) 0.966/0.966 (0.98) 0.894/0.895 (0.89) 0.913/0.907 (0.91) 0.899/0.892 (0.87) 0.562/0.562 (0.54) 0.937/0.929 (0.94) 0.874
glotlid 0.893/0.890 (0.90) 0.898/0.898 (0.94) 0.938/0.938 (0.98) 0.853/0.855 (0.87) 0.872/0.908 (0.91) 0.795/0.826 (0.76) 0.550/0.550 (0.60) 0.941/0.943 (0.95) 0.851
openlid3 0.770/0.773 (0.58) 0.846/0.846 (0.62) 0.870/0.870 (0.98) 0.457/0.458 (0.41) 0.675/0.701 (0.61) 0.614/0.636 (0.55) 0.016/0.016 (0.02) 0.767/0.774 (0.69) 0.634
lid176 0.733/0.736 (0.53) 0.771/0.771 (0.50) 0.774/0.774 (0.87) 0.375/0.378 (0.30) 0.626/0.665 (0.56) 0.568/0.602 (0.45) 0.031/0.031 (0.04) 0.799/0.807 (0.79) 0.595
halolid 0.868/0.868 (0.90) 0.887/0.887 (0.91) 0.932/0.932 (0.98) 0.987/0.985 (0.98) 0.566/0.589 (0.52) 0.666/0.684 (0.48) 0.376/0.376 (0.31) n/a 0.760
lid201 0.776/0.779 (0.58) 0.823/0.823 (0.60) 0.849/0.849 (0.97) 0.458/0.460 (0.40) 0.672/0.693 (0.60) 0.614/0.630 (0.52) 0.041/0.041 (0.03) n/a 0.611
lingua 0.745/0.748 (0.56) 0.786/0.786 (0.59) 0.795/0.795 (0.89) 0.204/0.206 (0.12) 0.418/0.455 (0.30) 0.385/0.414 (0.26) 0.035/0.035 (0.03) n/a 0.491

all halves: accuracy raw / corrected gold (macro-F1 corrected)

system crawl(661) goldspan(593) golddoc(294) pld(3222) wiki(5037) hard(12520) basaha(1600) tato(1847) mean corrected
c3 0.952/0.950 (0.92) 0.948/0.948 (0.86) 1.000/1.000 (1.00) 0.907/0.908 (0.92) 0.896/0.925 (0.93) 0.880/0.903 (0.88) 0.771/0.771 (0.58) 0.934/0.931 (0.94) 0.917
c2 0.943/0.941 (0.91) 0.938/0.938 (0.84) 0.997/0.997 (0.99) 0.906/0.907 (0.91) 0.894/0.923 (0.93) 0.874/0.897 (0.88) 0.740/0.740 (0.57) 0.928/0.927 (0.94) 0.909
ourlid_r2 0.912/0.911 (0.87) 0.934/0.934 (0.82) 0.983/0.983 (1.00) 0.889/0.890 (0.88) 0.914/0.908 (0.91) 0.895/0.888 (0.86) 0.551/0.551 (0.54) 0.937/0.932 (0.94) 0.875
glotlid 0.884/0.882 (0.81) 0.899/0.899 (0.66) 0.949/0.949 (0.93) 0.841/0.845 (0.85) 0.870/0.907 (0.91) 0.797/0.828 (0.76) 0.566/0.566 (0.62) 0.945/0.949 (0.96) 0.853
openlid3 0.778/0.779 (0.52) 0.865/0.865 (0.52) 0.895/0.895 (0.55) 0.436/0.437 (0.40) 0.676/0.706 (0.62) 0.617/0.639 (0.55) 0.021/0.021 (0.01) 0.760/0.766 (0.69) 0.638
lid176 0.734/0.735 (0.42) 0.777/0.777 (0.37) 0.796/0.796 (0.57) 0.356/0.359 (0.29) 0.623/0.664 (0.56) 0.568/0.601 (0.45) 0.031/0.031 (0.03) 0.800/0.806 (0.80) 0.596
halolid 0.865/0.865 (0.65) 0.890/0.890 (0.61) 0.942/0.942 (0.91) 0.987/0.985 (0.98) 0.554/0.577 (0.51) 0.662/0.681 (0.48) 0.369/0.369 (0.30) n/a 0.759
lid201 0.787/0.788 (0.48) 0.833/0.833 (0.41) 0.881/0.881 (0.54) 0.439/0.440 (0.39) 0.674/0.697 (0.61) 0.617/0.634 (0.52) 0.048/0.048 (0.03) n/a 0.617
lingua 0.743/0.744 (0.24) 0.788/0.788 (0.30) 0.803/0.803 (0.30) 0.197/0.199 (0.12) 0.412/0.452 (0.30) 0.380/0.410 (0.26) 0.043/0.043 (0.04) n/a 0.491

per-class F1 (corrected gold, all rows; support) -- best vs glotlid vs ourlid_r2

class crawl goldspan golddoc pld wiki hard basaha tato
tgl 0.97/0.94/0.95 (397) 0.98/0.96/0.98 (414) 1.00/0.99/0.99 (231) 0.89/0.82/0.85 (487) 0.95/0.95/0.94 (395) 0.96/0.93/0.96 (2032) - 0.98/0.99/0.97 (400)
eng 0.92/0.90/0.81 (114) 0.92/0.91/0.87 (60) - 0.94/0.97/0.96 (166) 0.91/0.93/0.89 (518) 0.85/0.92/0.89 (1407) - 1.00/1.00/1.00 (400)
ceb 0.96/0.93/0.90 (38) 0.97/0.88/0.95 (39) 1.00/0.93/1.00 (21) 0.89/0.73/0.73 (274) 0.99/0.98/0.98 (400) 0.96/0.91/0.94 (1124) - 0.86/0.92/0.83 (101)
hil 0.93/0.90/0.91 (37) 0.91/0.97/0.85 (30) 1.00/0.86/1.00 (17) 0.89/0.87/0.89 (251) - - 0.80/0.72/0.72 (400) -
war 1.00/1.00/0.97 (16) - - 0.86/0.85/0.85 (176) 0.98/0.98/0.97 (392) 0.96/0.93/0.94 (1141) - 0.88/0.94/0.87 (119)
ilo - - - 0.97/0.93/0.96 (414) 0.95/0.94/0.95 (390) 0.95/0.93/0.95 (1138) - 0.97/0.98/0.97 (120)
bcl - - - 0.90/0.86/0.89 (788) 0.92/0.93/0.91 (385) 0.92/0.88/0.91 (1117) 0.14/0.57/0.60 (400) -
pam - - - 0.98/0.93/0.97 (574) 0.88/0.86/0.79 (331) 0.87/0.78/0.82 (1029) - 0.96/0.97/0.95 (72)
pag - - - 0.95/0.85/0.85 (44) 0.96/0.94/0.96 (235) 0.89/0.85/0.86 (411) - -
cbk 0.91/0.97/0.91 (16) - - - 0.88/0.83/0.85 (306) 0.75/0.64/0.68 (351) - 0.97/0.93/0.96 (219)
krj - - - - - - 0.85/0.57/0.82 (400) -
spa - - - - 0.91/0.91/0.89 (493) 0.89/0.87/0.82 (741) - -
ind - - - - 0.89/0.89/0.90 (398) 0.85/0.79/0.84 (577) - 0.94/0.97/0.95 (400)
msa - - - - 0.89/0.87/0.90 (396) 0.85/0.76/0.85 (577) - -

crawl-realism per source (all rows): best vs GlotLID accuracy

source n best glotlid
hataw_news 183 0.951 0.869
rmn_news 147 0.946 0.925
remate_news 92 0.935 0.837
pilipino_mirror_news 69 0.986 0.783
liwayway_literary 34 0.941 0.824
notesnikiko_literary 24 1.000 1.000
pinoyweekly_news 24 0.917 0.917
bombo_news 24 0.958 0.917
radyo_pilipinas_news 19 1.000 1.000
philstar_psn 17 0.882 0.941
gma_news 14 0.929 0.929
inquirer_news 14 1.000 1.000

Measured limitations of c3

  • wiki/hard pam/bcl/ilo/war gold contains English text; raw numbers understate real accuracy by ~2-3 pts there; Chavacano wiki is Spanish-like, so cbk gold is unreliable except Tatoeba (cbk F1 0.97).
  • basaha accuracy is inflated by counting Minasbate/Rinconada as oth/lenient; honest numbers are the krj / hil F1.
  • 1-5 word strings ~0.71 raw on hard (mostly intrinsically ambiguous items); Aklanon/Surigaonon have no class (fall into hil/oth).
  • Round 5 (more real minority data from FineWeb-2 _removed) and round-5 English additions did not help; FLORES+ (clean human pag/ilo/ceb/war evaluation) is gated on Hugging Face and needs the owner to accept its terms.
  • Gate v2 (tree head + named rules) is fit on dev halves of ~10k labelled items; its head is a small sklearn model (pickle gitignored; re-fit with scripts/gate_v2.py c3).
  • 2x tagger cost: ~7k docs/min/core on 100-word docs, ~2.2k docs/min/core on full news docs.

Gate v2 (sealed halves; thresholds and tree head fit on dev only)

policy accepted frac precision accepted precision rejected crawl goldspan pld tato basaha
DEV: per-class thresholds 0.630 0.9713 0.754 1.000 1.000 0.970 0.991 0.873
DEV: tree head, tau for dev precision .98 (tau=0.980) 0.517 0.9803 0.795 0.993 0.976 0.980 0.986 0.960
DEV: tree head, same coverage as thresholds (tau=0.962) 0.630 0.9713 0.754 0.995 0.976 0.976 0.988 0.876
DEV: both (threshold AND tree>=tau) 0.437 0.9910 0.813 1.000 1.000 0.985 0.996 0.987
SEALED: per-class thresholds 0.636 0.9644 0.768 0.984 0.989 0.970 0.987 0.847
SEALED: tree head, tau for dev precision .98 (tau=0.980) 0.521 0.9924 0.785 1.000 1.000 0.995 0.996 0.972
SEALED: tree head, same coverage as thresholds (tau=0.962) 0.647 0.9870 0.720 0.996 0.996 0.987 0.995 0.961
SEALED: both (threshold AND tree>=tau) 0.439 0.9938 0.814 1.000 1.000 0.994 0.997 0.975

On the v0.2.9.5 release (c3 sidecar, out/ourlid_v0.2.9.5_c3.gate_v2.parquet): accept 196,575 + accept_ph_other 10,498 (= 90.1%), review 22,644 (9.9%; low_conf_tgl 16.8k, abstain_set 3.9k, meta_low 1.8k, low_conf_eng 51), junk 2. Previous per-class-threshold gate (dev-fit): accept 86.1%, review 9.1%, accept_ph_other 4.8%.

Weak spots re-examined on trustworthy sets (2026-10-11)

The weak-spot numbers used earlier came from our own noisy gold (Wikipedia bot text, Spanish-like cbk-zam, English inside pam/bcl wiki). They were re-measured on human-labelled external sets that no OurLID model produced (scripts/build_trust_sets.py; research and licences in docs/LID_WEAKSPOTS_RESEARCH.md). Cells are accuracy (recall for a class) with 95% Wilson interval, all rows (dev+sealed halves are reported separately in eval/LOOP_LOG.md). FLORES-200 text (via SIB-200, CC BY-SA 4.0) is evaluation-only and never enters training.

weak spot (trustworthy set) glotlid c3r c7r
Pangasinan recall (FLORES, n=1004) 1.000 [1.00-1.00] 0.999 [0.99-1.00] 0.997 [0.99-1.00]
English recall (FLORES, n=1004) 1.000 [1.00-1.00] 1.000 [1.00-1.00] 1.000 [1.00-1.00]
Spanish recall, not called Chavacano (FLORES, n=1004) 0.998 [0.99-1.00] 0.998 [0.99-1.00] 0.997 [0.99-1.00]
Chavacano recall (Tatoeba test, n=231) 0.840 [0.79-0.88] 0.970 [0.94-0.99] 0.974 [0.94-0.99]
Bikol recall (story sentences, clean) 0.894 [0.87-0.91] 0.920 [0.90-0.94] 0.961 [0.95-0.97]
Aklanon+Surigaonon -> oth (n=110, templated) 0.300 [0.22-0.39] 0.327 [0.25-0.42] 0.345 [0.26-0.44]
Short strings 2-5 words, all langs 0.771 [0.76-0.78] 0.837 [0.83-0.85] 0.832 [0.82-0.84]
of which 2 words 0.624 [0.59-0.65] 0.708 [0.68-0.73] 0.704 [0.68-0.73]
of which 3 words 0.739 [0.71-0.76] 0.819 [0.80-0.84] 0.815 [0.79-0.84]
of which 4-5 words 0.861 [0.85-0.88] 0.911 [0.90-0.92] 0.905 [0.89-0.92]
Short strings excl. ind/msa 0.819 [0.81-0.83] 0.866 [0.85-0.88] 0.861 [0.85-0.87]
Tatoeba <=4 words 0.866 [0.83-0.89] 0.858 [0.83-0.89] 0.856 [0.82-0.88]
ALLCAPS/Title/lower variants 0.975 [0.97-0.98] 0.999 [1.00-1.00] 0.999 [1.00-1.00]
English recall, ALLCAPS/Title/lower 0.931 [0.91-0.95] 1.000 [1.00-1.00] 1.000 [1.00-1.00]
Whole FLORES set (9 langs) 0.986 [0.98-0.99] 0.989 [0.99-0.99] 0.989 [0.99-0.99]

Verdict per weak spot (final model = c7 + marker rules c7r; before = c3r = c3 + rules; GlotLID shown as the strongest external baseline)

  • Pangasinan: not a weak spot on trustworthy data. FLORES-200 recall 0.997-0.999 (n=1,004, 95% CI >= 0.99); GlotLID 1.000. The earlier "weak" reading came from noisy wiki/PLD gold. No change made. Caveat: FLORES is formal translated text; Pangasinan web/social text is untested.
  • English recall / ALLCAPS: FLORES English 1.000, ALLCAPS/Title/lower variants 1.000 (GlotLID 0.931). On the messy crawl gold English recall is 0.94 (n=114, +-0.04) with GlotLID 0.87; the misses are Taglish frame ambiguities (eng<->tgl), not language-ID errors. No change.
  • Chavacano vs Spanish: Chavacano recall 0.974 on Tatoeba test (n=231, CI 0.94-0.99; GlotLID 0.84); FLORES Spanish recall 0.997 (n=1,004). cbk-zam Wikipedia gold is Spanish-like and stays flagged as noisy. Extra gate features (count of Chavacano-not-Spanish function words) are in gate v2. Free-text Chavacano (news/social) is not covered by any trustworthy set, and the eBible Chavacano NT could not be downloaded and is NC/ND-licensed anyway.
  • Bikol: real fix. Rinconada (a Bikol language) had been trained as oth, which pulled Central Bikol sentences into oth (40/900). Counting it as bcl: Bikol recall 0.920 -> 0.961 (paired dev +1.7 [+0.4,+3.0], sealed +2.3 [+1.1,+3.6]); GlotLID 0.894. Cost (honest): BasahaCorpus Minasbate (a Visayan language, deliberately in oth) now leaks into bcl/hil: oth recall 266 -> 236 of 400; the basaha set therefore shows a significant -2.9 (dev) / -2.0 (sealed) drop even though Rinconada itself went from 34 to 311 correct of 400 under the new class definition. crawl/goldspan/golddoc/pld show no significant change (all CIs include 0). Only Central Bikol sentences were testable; Legazpi/Miraya/Rinconada free text has no trustworthy eval.
  • Aklanon / Surigaonon: no open training corpus exists (only ~28k chars of templated Wikimedia Incubator stubs + 28 Tatoeba sentences), so no class could be trained. Instead an explicit unsupported Philippine language bucket: ourlid.rules (hand-written, unit-tested markers; ALLCAPS acronyms ignored) overturns a Visayan/Tagalog-looking prediction to oth and records sub=akl|sgd. Recall on the 110-item templated set 0.291 -> 0.345 (it is low because single stub sentences rarely carry two markers; intervals 0.26-0.44). What the rules did on real data: 0 false fires on 125k supported-class dev sentences and on every trust/gold set; on the real 229k-doc release they flag 173 Aklanon and 30 Surigaonon docs (all rmn_news), of which 39 were previously labelled ceb/bcl/tgl/hil/krj/war and 1 gold-ceb golddoc that is in fact Surigaonon (gold error found by the rule). They are not a classifier: recall on free Aklanon/Surigaonon text is unknown. An orthographic Aklanon e-for-l rule was tried and rejected (fires on Tagalog babae, English health, Spanish words).
  • Strings under five words: not fixed. 2 words 0.704, 3 words 0.815, 4-5 words 0.905 (c7r) vs GlotLID 0.624 / 0.739 / 0.861. Raising short-fragment augmentation from 10% to 20% (round W2) gave no gain and cost crawl accuracy (3 tgl->eng flips) so it was reverted. Roughly a third of the remaining 2-5 word errors are Indonesian<->Malay, intrinsically ambiguous. Use the gate: with c3, for 2-3 word strings confidence >= 0.99 and not abstaining keeps 41% of items at 0.914 precision (4-5 words: 60% at 0.961).

Final artefacts: model models/ourlid-c7 (weights gitignored; staged in release_stage/ for the private release); sidecar out/ourlid_v0.2.9.5_c7.gate_v2.parquet with new columns ourlid_label_final (rule-adjusted label), ourlid_oth_sub (akl|sgd), gate_v2, gate_v2_reason, gate_v2_p_correct. Gate v2 counts on the 229,719 release docs (thresholds and tree head fit on dev halves only; policy tau = same coverage as thresholds): accept 192,193; review 26,535 (low_conf_tgl 22,056, abstain_set 4,119, low_conf_hil 275, meta_low 38, other 47); accept_ph_other 10,990 (incl. unsupported_ph_akl 173, unsupported_ph_sgd 30); junk 1. Gate v2 sealed precision of accepted at matched coverage 0.986. Known side effect of c7 vs c3: 106 release labels differ, 76 of them tgl->eng on Taglish-frame docs.

What still needs native review: Aklanon and Surigaonon (the only open text is ~28k characters of templated Wikimedia Incubator stubs plus 28 Tatoeba sentences; the marker lists are derived from grammar sketches and are unverified by native speakers), Bikol varieties beyond Central Bikol (Rinconada/Miraya/Legazpi have no trustworthy eval text), Chavacano vs Spanish on free web text (Tatoeba n=231 is short, simple sentences), and the review bucket via labeling/ (scripts/ingest_labels.py).

Speed and English/short-string follow-up (2026-10-12; model c7 unchanged)

  • Speed (round S1): profiling shows ~75% of CPU in the two tagger GEMMs; ONNX would not help on 1 BLAS thread/worker and was not added. New opt-in cascade ourlid.r3.analyze_fast (OURLID_FAST=100:0.95 for scripts/run_release.py): analyse the first 100 words, escalate to the full doc if that analysis abstains or conf < 0.95. 7 worker processes, shared box: full ~291-word news docs 17.5k -> 46.9k docs/min; ~100-word docs 57k docs/min (exact). Exactness vs full analysis on 2,376 fresh release docs: label equal 99.83%, label+abstain equal 99.3%; crawl/goldspan/PLD identical (short docs). It is NOT label-identical on long docs (0.17% differ) and conf/mixture/cs_share of fast-path docs describe the first 100 words (ourlid_fast column). Default remains exact.
  • English recall / short strings (round S2): English recall is already 97.7% on 2-5-word fragments, 100% on short Tatoeba/ALLCAPS/FLORES; remaining eng<->tgl crawl "errors" are Taglish-frame ambiguity or name-only strings (gold nol). Residual short-string loss is msa<->ind, war->ceb/hil and spa->cbk, not English. Only gate-level change kept: English/name/Tagalog function-word features in the gate tree head (sealed precision at equal coverage +0.1 to +0.3 pt, no regression; dev gain at low coverage does not transfer). 42 of 229,719 v0.2.9.5 docs change gate decision; labels unchanged. A label-agnostic accept_taglish rule was tried and reverted (no precision gain).