|
Download docs/TECH_REPORT.md from sapinsapin/wikalid: direct link, hf CLI and curl.
- Browser
- Download file 32.5 kB
-
https://huggingface.co/sapinsapin/wikalid/resolve/main/docs/TECH_REPORT.md
- Command line
-
hf download hf://sapinsapin/wikalid/docs/TECH_REPORT.md
-
curl -L -o TECH_REPORT.md https://huggingface.co/sapinsapin/wikalid/resolve/main/docs/TECH_REPORT.md
32.5 kB
| # OurLID: a token-level, mixture-aware and calibrated language identifier for Philippine text | |
| *Draft v0.1 - 2026-10-10 - not peer reviewed. All numbers are from `eval/` in this repository.* | |
| ## Abstract | |
| Off-the-shelf language identification (LID) is built for documents and closed label sets, yet Philippine text is code-mixed (Taglish), name-dense, noisily spelled and includes many closely related languages (Cebuano/Hiligaynon/Waray; Bikol/Tagalog). We present OurLID, a 26 MB numpy-only system that classifies *tokens* with a hashed n-gram tagger, aggregates them into a language *mixture*, scores it with a small head using induced closed-class lexicon evidence, and abstains with split-conformal label sets. Trained only on licence-clean or pointer-only data and synthetic code-switching, it matches or beats GlotLID v3 on a new permissive hard-case benchmark (0.899 vs 0.797 accuracy), on held-out Wikipedia (0.917 vs 0.870) and on our gold news docs (0.963 vs 0.949), ties the best system on gold news spans (0.917), but loses on Hiligaynon spans, read-speech Cebuano and on rejecting unseen related languages. Ablations show that the components we expected to help most (code-switch synthesis, augmentation, morphology features) are not significant on real text, whereas **removing document-level context pooling** (+4 pts on PLD) and a **learned head** (+6 pts on the hard benchmark) are. | |
| ## 1 Problem and motivation | |
| Four structural failures of existing LID on Philippine text, each measured on our private gold news spans (593 spans, single annotator) with the same harness for all tools (per-span failure list kept private because it quotes news text; summary here): (i) *no not-a-language class*: 0 % of 20 list/score/roster spans correct for all tools except OpenLID-v3 `zxx` (partial); (ii) *name-dense Tagalog* -> bcl/pag (GlotLID 14, halo-lid 9 errors); (iii) *Taglish* (heavy CS spans 17-33 % error; lid.176 calls 43 Tagalog spans English); (iv) *hil->ceb* for every tool without a hil class (26-27 errors of 30). Quality audits of web corpora (Caswell et al. 2020; Kreutzer et al. 2022) show the same pattern: high held-out F1, low human-judged precision for low-resource languages. | |
| ## 2 Related work (verified sources; table in `study/STACK_STUDY.md`) | |
| fastText lid.176 / "Bag of Tricks" (Joulin et al. 2016); langid.py (Lui & Baldwin 2012); CLD2/CLD3; lingua; GlotLID (Kargaran et al., EMNLP Findings 2023); OpenLID and OpenLID-v3 (Burchell et al. 2023; Fedorova et al., VarDial 2026); NLLB lid201; MMS-LID (audio); Google LangID/semi-supervised filtering (Caswell 2020; Bapna 2022); IndicLID cascade (Madhani et al. 2023); FineWeb-2 filtering (arXiv 2506.20920); LID survey (Jauhiainen et al. 2019); similar-language/variety identification (VarDial 2023 findings; Haas & Derczynski 2021); word-level code-switching LID (LinCE; TweetTaglish, Herrera et al. 2022; TagLID); Fightin' Words log-odds (Monroe et al. 2008); conformal prediction (Angelopoulos & Bates 2021); FLORES+/OLDI (CC BY-SA 4.0; ceb, ilo, pag, war present, hil/bcl/pam absent). | |
| ## 3 Method | |
| **Tokens.** Script-aware tokenizer keeping hyphenated Tagalog affixation (`nag-submit`) and quotes. Per-token features hashed to 2^19 buckets: char 1-5-grams with boundary markers, folded word identity, morphology signatures (prefix/infix/suffix classes such as -um-/-in-, reduplication, clitics), word-shape and script. | |
| **Tagger.** EmbeddingBag(2^19 x 48) -> linear(60->128) -> conv(k5) -> conv(k3, dilation 2) with residual -> linear -> 17-way softmax (11 PH classes, eng, spa, ind, msa, oth, neutral `x`). *No document-level pooling* (Section 5). | |
| **Supervision.** Monolingual sentences with token labels (neutral for punctuation/digits/capitalised non-initial tokens); programmatic code-switch synthesis (English chunk insertion, inter-sentential mixing, quote frames, affixed English), augmentation (case, SMS spelling, typos, `#`/`@` names, emoji, listing noise); *weak purification*: lowercase tokens in non-English training sentences that belong to a contrast-derived pure-English list (1,238 types) are relabelled `eng`. | |
| **Document layer.** Token posteriors -> mixture over all words and over unquoted words, lexical mass, entropy, quote fraction; closed-class lexicon block = mean log-odds evidence per language from words that are discriminative (Monroe z >= 2) in both Wikipedia and web samples; logistic-regression head (57 features) trained on 24k synthetic documents; temperature scaling; Mondrian split-conformal thresholds per predicted class on 8k calibration documents. `cs_level` from English share (<0.03 none, <0.25 light, else heavy). | |
| **Data.** `docs/DATA_CARD.md`: 2.64M training sentences (R: Wikipedia/Tatoeba redistributable; W/G: FineWeb-2 web pointer-only), domain-disjoint dev/test. | |
| ## 4 Experimental setup | |
| Test sets: private gold news spans/docs (single annotator, non-native), PLD read-speech sentences (3,222; halo-lid trained on PLD), held-out Wikipedia/Tatoeba-source sentences (5,037, label-noisy for pam/pag/cbk), PH-LID-Hard v0 (12,880, synthetic transformations of Tier-R test text; `docs/BENCHMARK_CARD.md`). Baselines run by us: GlotLID v3, OpenLID-v3, halo-lid, lingua, lid.176, lid201, langid.py, and the earlier "phlid" stack (GlotLID+halo-lid+OpenLID+lexicon LR). Metrics: accuracy, macro-F1 (classes n>=20), ECE (10 bins), AURC, per-category accuracy, conformal coverage; paired bootstrap 95 % CIs (500 resamples) for ablations. `quoted_english` is excluded from headline numbers because its label (frame language) is a convention. | |
| ## 5 Results | |
| **Main table** (acc / macro-F1; `eval/FINAL_RESULTS.md`): | |
| | system | gold spans | gold docs | PLD | held-out Wiki | PH-LID-Hard | | |
| |---|---|---|---|---|---| | |
| | OurLID v0.1 | 0.917 / 0.807 | 0.963 / 0.985 | 0.888 / 0.875 | 0.917 / 0.920 | 0.899 / 0.879 | | |
| | phlid stack | 0.917 / 0.769 | 0.939 / 0.979 | (0.934 / 0.948) | 0.704 / 0.688 | 0.717 / 0.596 | | |
| | GlotLID v3 | 0.899 / 0.743 | 0.949 / 0.957 | 0.841 / 0.848 | 0.870 / 0.873 | 0.797 / 0.729 | | |
| | OpenLID-v3 | 0.865 / 0.598 | 0.895 / 0.827 | 0.436 / 0.405 | 0.676 / 0.607 | 0.617 / 0.539 | | |
| | halo-lid | 0.890 / 0.728 | 0.942 / 0.970 | (0.987 / 0.988) | 0.554 / 0.492 | 0.662 / 0.468 | | |
| | lingua / lid.176 / lid201 / langid | 0.79 / 0.78 / 0.83 / 0.76 acc | 0.80 / 0.80 / 0.88 / 0.77 | 0.20 / 0.36 / 0.44 / 0.19 | 0.41 / 0.62 / 0.67 / 0.38 | 0.38 / 0.57 / 0.62 / 0.35 | | |
| Parentheses: system trained on PLD (leakage). | |
| **Calibration.** ECE 0.021-0.068, AURC 0.001-0.025. **Abstention** (`eval/ABSTAIN.md`): at alpha=0.05 set coverage 0.88 (Hard), 0.89 (gold spans), 0.95 (gold docs), 0.87 (PLD), 0.91 (Wikipedia) - below the nominal 0.95 because calibration is synthetic; accuracy on singleton sets 0.92-0.97, abstaining on 10-25 % of items. | |
| **Ablations** (320k items, 1 epoch; `eval/ABLATIONS.md`). Global mean-pooled context fell 4.0 pts on PLD (CI 2.8-5.0) and 2.0 pts on gold spans versus no pooling; local +-8 pooling was in between. Code-switch synthesis, augmentation, morphology features: differences within +-1 pt, CIs include 0 on real sets (augmentation matters for ALLCAPS: 0.88 vs 0.93 on Hard). Uniform class sampling hurt PLD (-1.6 pts). English purification: token-level English confidence in an English quote 0.40 -> 0.98, gold +1.3 pts, PLD -0.9 pts. | |
| **Per-category** (Hard): listings 0.97 (only OpenLID 0.56 else 0.00), ALLCAPS 0.94, SMS 0.91, short strings 0.74 (GlotLID 0.60), clean sentences 0.92 (GlotLID 0.87). | |
| **Errors we make:** gold: oth->hil 11 (unsupported Aklanon/Chavacano-like text), eng->tgl 8, nol->tgl 4; PLD: bcl->tgl 62, ceb->tgl 53, ceb->bcl 35; Wikipedia: cbk<->spa, ind<->msa. | |
| **Speed** (one process): ~19k docs/min at ~100 words, ~60k docs/min at 25 words, ~5.9k docs/min at ~400 tokens; RSS 115 MB. | |
| ## 6 Release-scale application | |
| v0.2.9.5 news release (229,719 documents): see `docs/RELEASE_SIDECAR.md` for label distribution and the disagreement analysis against the earlier phlid sidecar. | |
| ## 7 Limitations | |
| Synthetic calibration, single non-native annotator, benchmark partly shares design with training, Wikipedia-trained baselines favoured on Wikipedia text but so are we (same sources), no real Taglish mixture gold, no unseen-language rejection beyond lists/noise, uneven classes (krj/tsg experimental; hil has no redistributable text), no speech/romanisation/other-script evaluation, weights licence unresolved. | |
| ## 8 Ethics and community note | |
| LID mistakes silently erase regional-language text from corpora; the abstain output should stay on. Native-speaker annotation is needed (`docs/NATIVE_REVIEW.md`). No personal data is redistributed; web training text is pointer-only. NC/GPL models were used for comparison only. | |
| ## 9 Reproducibility | |
| `docs/REPRODUCE.md`, fixed seeds, 12 unit/integration tests. | |
| ## Round 2 addendum (draft) | |
| We add a private crawl-realism held-out set (661 scored spans from 12 news sources and 23 hard classes), a 1,712-token word-level Taglish gold set, near-neighbour negatives (FineWeb-2 languages of the Philippines plus synthetic Aklanon-like text), v3 synthetic augmentations and real-data conformal calibration. On crawl-realism (ambiguous-frame items scored leniently, macro-F1 over classes with >=8 items) OurLID r2 reaches 0.91 accuracy / 0.851 macro-F1 versus 0.90 / 0.736 for r1, 0.88 / 0.742 for the phlid stack and 0.88 / 0.726 for GlotLID (strict forced-label accuracy r1 0.877 -> r2 0.894); improvements concentrate on `nol` and unseen neighbouring languages and are small elsewhere (+1.7 to +2.0 pt on real news sets, -0.4 pt on PH-LID-Hard). English recall on short and ALLCAPS crawl spans (0.79) and 1-5 word strings (0.59 for all tools) remain open. See `eval/R2_VS_R1.md`, `eval/CRAWL_REALISM.md`, `eval/GATE_POLICY.md`. | |
| <!-- LOOP:BEGIN --> | |
| ## OurLID improve loop (rounds 0-6; best = `c3`: 2-tagger ensemble + head, r3 inference options) | |
| Protocol: every eval set is split 50/50 (md5 of group id; `eval/sets.py`) into a *dev* half used for all decisions and a *sealed* half used only for informational snapshots. Gold labels of wiki/hard/pld/cbk are noisy (English text labelled pam/bcl/ilo/war, Spanish text labelled Chavacano); `eval/loop_report.py` therefore reports raw and *corrected* gold (non-eng gold with >=2 of GlotLID/OpenLID/lid176 saying eng -> eng; cbk gold with >=2 saying spa and no Chavacano marker -> spa). Caveat: the corrected gold shares detectors with the round-1 training-label cleaning. Full log: `eval/LOOP_LOG.md`. | |
| | round | change | outcome (dev halves, paired bootstrap vs previous kept round; * = 95% CI excludes 0) | kept | | |
| |---|---|---|---| | |
| | 0 | r3c baseline (case-fold view, HMM smoothing, +English-with-names head data) | crawl +2.8*, goldspan +1.1, golddoc +2.7 vs r2; raw wiki -1.0 / hard -0.9 but corrected-gold wiki +0.6 / hard +0.5*: the "tradeoff" was label noise | yes | | |
| | 1 | consensus label cleaning of the training corpus (GlotLID/OpenLID/lid176 as detectors; 36k docs -> eng, 2.3k cbk -> spa) | corrected wiki +0.8*, hard +0.5*; raw wiki/hard -1 (consistently-correct English no longer called pam) | yes | | |
| | 1b | stress-accent-stripped second view at inference | PLD +1.7* (Cebuano with stress marks 31/113 -> 98/113 recalled); others +-0.1 | yes | | |
| | 2 | story-disjoint BasahaCorpus (krj, hil real; Minasbate/Rinconada -> oth) + accent-injection augmentation | krj F1 0.78->0.83, hil 0.72->0.80; basaha acc +19* (inflated by mbt/rin->oth); crawl +1.8, golddoc +1.4, goldspan -1.5 ns, tato -0.6 ns | yes | | |
| | 3 | 2-seed tagger ensemble + ensemble head/conformal | hard +0.8*, basaha +2.4*, crawl +0.9, tato +0.7 | yes | | |
| | 4 | speed: shared featurisation, 1 BLAS thread / worker | identical labels; 39k docs/min (100-word docs, 7 workers, loaded box) | yes | | |
| | 5 | + FineWeb-2 `_removed` minority data (pag/pam/cbk/ilo/hil/bcl/krj/war/tsg, ~200k sent.) + 60k Tatoeba English | no significant gain; crawl -1.5, golddoc -1.4, tato -1.4*, cbk(hard) F1 .76->.68; krj/hil +.02-.03 | **no (reverted)** | | |
| | 6 | gate v2: learned tree head + named rules over c3 outputs (dev-fit) | sealed precision of accepted 0.964 -> 0.987 at equal coverage (64%); pld 0.85 -> 0.96 | yes | | |
| **BasahaCorpus licence check (2026-10-10):** the EMNLP 2023 paper (Imperial & Kochmar, sec. 2.1) states the stories were obtained from Let's Read Asia 'with explicit permission ... licensed under Creative Commons BY 4.0'; the GitHub repo (imperialite/BasahaCorpus-HierarchicalCrosslingualARA) carries no LICENSE file, only the permission statement. Status: **licence stated by the authors, not independently verified at the upstream site / per story** -> attribution required (Imperial & Kochmar 2023; Let's Read Asia / The Asia Foundation); if a downstream user needs a clean chain, retrain without `data/work/basaha_train.parquet` (scripts/build_basaha_split.py) - expected cost: Kinaray-a F1 ~0.78 instead of ~0.84, Hiligaynon ~0.72 instead of ~0.82 (round 2/3 ablation). | |
| Training-label cleaning uses GlotLID/OpenLID/lid176 only as *detectors* (rule-based relabelling); no external weights are in the model. Training data of `c3`: Wikipedia (CC BY-SA), Tatoeba (CC BY), FineWeb-2 (ODC-By, GlotLID-labelled, lexical-purity filtered), BasahaCorpus (see above). No scraped news is in the training data. Eval sets crawl-realism / gold spans / gold docs / PLD are private and not published. | |
| ### Results (accuracy raw/corrected gold, macro-F1 corrected in brackets) | |
| ### sealed halves: accuracy raw / corrected gold (macro-F1 corrected) | |
| | system | crawl(335) | goldspan(327) | golddoc(148) | pld(1653) | wiki(2563) | hard(6379) | basaha(794) | tato(968) | mean corrected | | |
| |---|---|---|---|---|---|---|---|---|---| | |
| | c3 | 0.943/0.943 (0.94) | 0.942/0.942 (0.94) | 1.000/1.000 (1.00) | 0.911/0.913 (0.92) | 0.898/0.929 (0.93) | 0.878/0.900 (0.88) | 0.759/0.759 (0.58) | 0.928/0.925 (0.94) | 0.914 | | |
| | c2 | 0.934/0.934 (0.93) | 0.927/0.927 (0.92) | 0.993/0.993 (1.00) | 0.909/0.910 (0.91) | 0.895/0.925 (0.93) | 0.873/0.896 (0.87) | 0.722/0.722 (0.56) | 0.925/0.924 (0.93) | 0.904 | | |
| | ourlid_r2 | 0.919/0.919 (0.91) | 0.930/0.930 (0.94) | 1.000/1.000 (1.00) | 0.884/0.885 (0.88) | 0.916/0.909 (0.91) | 0.891/0.885 (0.86) | 0.540/0.540 (0.54) | 0.937/0.934 (0.94) | 0.875 | | |
| | glotlid | 0.875/0.875 (0.93) | 0.899/0.899 (0.92) | 0.959/0.959 (1.00) | 0.830/0.834 (0.84) | 0.869/0.907 (0.91) | 0.799/0.829 (0.76) | 0.583/0.583 (0.64) | 0.949/0.955 (0.97) | 0.855 | | |
| | openlid3 | 0.785/0.785 (0.62) | 0.881/0.881 (0.87) | 0.919/0.919 (0.99) | 0.417/0.417 (0.40) | 0.676/0.710 (0.63) | 0.619/0.641 (0.55) | 0.026/0.026 (0.01) | 0.754/0.758 (0.68) | 0.642 | | |
| | lid176 | 0.734/0.734 (0.57) | 0.783/0.783 (0.74) | 0.818/0.818 (0.90) | 0.337/0.341 (0.29) | 0.620/0.664 (0.55) | 0.567/0.599 (0.45) | 0.031/0.031 (0.03) | 0.801/0.805 (0.80) | 0.597 | | |
| | halolid | 0.863/0.863 (0.93) | 0.893/0.893 (0.91) | 0.953/0.953 (1.00) | 0.987/0.985 (0.98) | 0.542/0.566 (0.50) | 0.659/0.679 (0.48) | 0.361/0.361 (0.30) | n/a | 0.757 | | |
| | lid201 | 0.797/0.797 (0.62) | 0.841/0.841 (0.84) | 0.912/0.912 (0.99) | 0.420/0.421 (0.38) | 0.675/0.700 (0.61) | 0.619/0.637 (0.52) | 0.055/0.055 (0.03) | n/a | 0.623 | | |
| | lingua | 0.740/0.740 (0.43) | 0.789/0.789 (0.60) | 0.811/0.811 (0.90) | 0.191/0.193 (0.12) | 0.406/0.448 (0.30) | 0.376/0.405 (0.26) | 0.052/0.052 (0.04) | n/a | 0.491 | | |
| ### dev halves: accuracy raw / corrected gold (macro-F1 corrected) | |
| | system | crawl(326) | goldspan(266) | golddoc(146) | pld(1569) | wiki(2474) | hard(6141) | basaha(806) | tato(879) | mean corrected | | |
| |---|---|---|---|---|---|---|---|---|---| | |
| | c3 | 0.960/0.957 (0.94) | 0.955/0.955 (0.95) | 1.000/1.000 (1.00) | 0.902/0.902 (0.91) | 0.893/0.922 (0.92) | 0.883/0.907 (0.89) | 0.782/0.782 (0.59) | 0.941/0.937 (0.95) | 0.920 | | |
| | c2 | 0.951/0.948 (0.93) | 0.951/0.951 (0.94) | 1.000/1.000 (1.00) | 0.902/0.903 (0.91) | 0.892/0.921 (0.92) | 0.875/0.898 (0.88) | 0.758/0.758 (0.58) | 0.932/0.931 (0.94) | 0.914 | | |
| | ourlid_r2 | 0.905/0.902 (0.88) | 0.940/0.940 (0.91) | 0.966/0.966 (0.98) | 0.894/0.895 (0.89) | 0.913/0.907 (0.91) | 0.899/0.892 (0.87) | 0.562/0.562 (0.54) | 0.937/0.929 (0.94) | 0.874 | | |
| | glotlid | 0.893/0.890 (0.90) | 0.898/0.898 (0.94) | 0.938/0.938 (0.98) | 0.853/0.855 (0.87) | 0.872/0.908 (0.91) | 0.795/0.826 (0.76) | 0.550/0.550 (0.60) | 0.941/0.943 (0.95) | 0.851 | | |
| | openlid3 | 0.770/0.773 (0.58) | 0.846/0.846 (0.62) | 0.870/0.870 (0.98) | 0.457/0.458 (0.41) | 0.675/0.701 (0.61) | 0.614/0.636 (0.55) | 0.016/0.016 (0.02) | 0.767/0.774 (0.69) | 0.634 | | |
| | lid176 | 0.733/0.736 (0.53) | 0.771/0.771 (0.50) | 0.774/0.774 (0.87) | 0.375/0.378 (0.30) | 0.626/0.665 (0.56) | 0.568/0.602 (0.45) | 0.031/0.031 (0.04) | 0.799/0.807 (0.79) | 0.595 | | |
| | halolid | 0.868/0.868 (0.90) | 0.887/0.887 (0.91) | 0.932/0.932 (0.98) | 0.987/0.985 (0.98) | 0.566/0.589 (0.52) | 0.666/0.684 (0.48) | 0.376/0.376 (0.31) | n/a | 0.760 | | |
| | lid201 | 0.776/0.779 (0.58) | 0.823/0.823 (0.60) | 0.849/0.849 (0.97) | 0.458/0.460 (0.40) | 0.672/0.693 (0.60) | 0.614/0.630 (0.52) | 0.041/0.041 (0.03) | n/a | 0.611 | | |
| | lingua | 0.745/0.748 (0.56) | 0.786/0.786 (0.59) | 0.795/0.795 (0.89) | 0.204/0.206 (0.12) | 0.418/0.455 (0.30) | 0.385/0.414 (0.26) | 0.035/0.035 (0.03) | n/a | 0.491 | | |
| ### all halves: accuracy raw / corrected gold (macro-F1 corrected) | |
| | system | crawl(661) | goldspan(593) | golddoc(294) | pld(3222) | wiki(5037) | hard(12520) | basaha(1600) | tato(1847) | mean corrected | | |
| |---|---|---|---|---|---|---|---|---|---| | |
| | c3 | 0.952/0.950 (0.92) | 0.948/0.948 (0.86) | 1.000/1.000 (1.00) | 0.907/0.908 (0.92) | 0.896/0.925 (0.93) | 0.880/0.903 (0.88) | 0.771/0.771 (0.58) | 0.934/0.931 (0.94) | 0.917 | | |
| | c2 | 0.943/0.941 (0.91) | 0.938/0.938 (0.84) | 0.997/0.997 (0.99) | 0.906/0.907 (0.91) | 0.894/0.923 (0.93) | 0.874/0.897 (0.88) | 0.740/0.740 (0.57) | 0.928/0.927 (0.94) | 0.909 | | |
| | ourlid_r2 | 0.912/0.911 (0.87) | 0.934/0.934 (0.82) | 0.983/0.983 (1.00) | 0.889/0.890 (0.88) | 0.914/0.908 (0.91) | 0.895/0.888 (0.86) | 0.551/0.551 (0.54) | 0.937/0.932 (0.94) | 0.875 | | |
| | glotlid | 0.884/0.882 (0.81) | 0.899/0.899 (0.66) | 0.949/0.949 (0.93) | 0.841/0.845 (0.85) | 0.870/0.907 (0.91) | 0.797/0.828 (0.76) | 0.566/0.566 (0.62) | 0.945/0.949 (0.96) | 0.853 | | |
| | openlid3 | 0.778/0.779 (0.52) | 0.865/0.865 (0.52) | 0.895/0.895 (0.55) | 0.436/0.437 (0.40) | 0.676/0.706 (0.62) | 0.617/0.639 (0.55) | 0.021/0.021 (0.01) | 0.760/0.766 (0.69) | 0.638 | | |
| | lid176 | 0.734/0.735 (0.42) | 0.777/0.777 (0.37) | 0.796/0.796 (0.57) | 0.356/0.359 (0.29) | 0.623/0.664 (0.56) | 0.568/0.601 (0.45) | 0.031/0.031 (0.03) | 0.800/0.806 (0.80) | 0.596 | | |
| | halolid | 0.865/0.865 (0.65) | 0.890/0.890 (0.61) | 0.942/0.942 (0.91) | 0.987/0.985 (0.98) | 0.554/0.577 (0.51) | 0.662/0.681 (0.48) | 0.369/0.369 (0.30) | n/a | 0.759 | | |
| | lid201 | 0.787/0.788 (0.48) | 0.833/0.833 (0.41) | 0.881/0.881 (0.54) | 0.439/0.440 (0.39) | 0.674/0.697 (0.61) | 0.617/0.634 (0.52) | 0.048/0.048 (0.03) | n/a | 0.617 | | |
| | lingua | 0.743/0.744 (0.24) | 0.788/0.788 (0.30) | 0.803/0.803 (0.30) | 0.197/0.199 (0.12) | 0.412/0.452 (0.30) | 0.380/0.410 (0.26) | 0.043/0.043 (0.04) | n/a | 0.491 | | |
| ### per-class F1 (corrected gold, all rows; support) -- best vs glotlid vs ourlid_r2 | |
| | class | crawl | goldspan | golddoc | pld | wiki | hard | basaha | tato | | |
| |---|---|---|---|---|---|---|---|---| | |
| | tgl | 0.97/0.94/0.95 (397) | 0.98/0.96/0.98 (414) | 1.00/0.99/0.99 (231) | 0.89/0.82/0.85 (487) | 0.95/0.95/0.94 (395) | 0.96/0.93/0.96 (2032) | - | 0.98/0.99/0.97 (400) | | |
| | eng | 0.92/0.90/0.81 (114) | 0.92/0.91/0.87 (60) | - | 0.94/0.97/0.96 (166) | 0.91/0.93/0.89 (518) | 0.85/0.92/0.89 (1407) | - | 1.00/1.00/1.00 (400) | | |
| | ceb | 0.96/0.93/0.90 (38) | 0.97/0.88/0.95 (39) | 1.00/0.93/1.00 (21) | 0.89/0.73/0.73 (274) | 0.99/0.98/0.98 (400) | 0.96/0.91/0.94 (1124) | - | 0.86/0.92/0.83 (101) | | |
| | hil | 0.93/0.90/0.91 (37) | 0.91/0.97/0.85 (30) | 1.00/0.86/1.00 (17) | 0.89/0.87/0.89 (251) | - | - | 0.80/0.72/0.72 (400) | - | | |
| | war | 1.00/1.00/0.97 (16) | - | - | 0.86/0.85/0.85 (176) | 0.98/0.98/0.97 (392) | 0.96/0.93/0.94 (1141) | - | 0.88/0.94/0.87 (119) | | |
| | ilo | - | - | - | 0.97/0.93/0.96 (414) | 0.95/0.94/0.95 (390) | 0.95/0.93/0.95 (1138) | - | 0.97/0.98/0.97 (120) | | |
| | bcl | - | - | - | 0.90/0.86/0.89 (788) | 0.92/0.93/0.91 (385) | 0.92/0.88/0.91 (1117) | 0.14/0.57/0.60 (400) | - | | |
| | pam | - | - | - | 0.98/0.93/0.97 (574) | 0.88/0.86/0.79 (331) | 0.87/0.78/0.82 (1029) | - | 0.96/0.97/0.95 (72) | | |
| | pag | - | - | - | 0.95/0.85/0.85 (44) | 0.96/0.94/0.96 (235) | 0.89/0.85/0.86 (411) | - | - | | |
| | cbk | 0.91/0.97/0.91 (16) | - | - | - | 0.88/0.83/0.85 (306) | 0.75/0.64/0.68 (351) | - | 0.97/0.93/0.96 (219) | | |
| | krj | - | - | - | - | - | - | 0.85/0.57/0.82 (400) | - | | |
| | spa | - | - | - | - | 0.91/0.91/0.89 (493) | 0.89/0.87/0.82 (741) | - | - | | |
| | ind | - | - | - | - | 0.89/0.89/0.90 (398) | 0.85/0.79/0.84 (577) | - | 0.94/0.97/0.95 (400) | | |
| | msa | - | - | - | - | 0.89/0.87/0.90 (396) | 0.85/0.76/0.85 (577) | - | - | | |
| ### crawl-realism per source (all rows): best vs GlotLID accuracy | |
| | source | n | best | glotlid | | |
| |---|---|---|---| | |
| | hataw_news | 183 | 0.951 | 0.869 | | |
| | rmn_news | 147 | 0.946 | 0.925 | | |
| | remate_news | 92 | 0.935 | 0.837 | | |
| | pilipino_mirror_news | 69 | 0.986 | 0.783 | | |
| | liwayway_literary | 34 | 0.941 | 0.824 | | |
| | notesnikiko_literary | 24 | 1.000 | 1.000 | | |
| | pinoyweekly_news | 24 | 0.917 | 0.917 | | |
| | bombo_news | 24 | 0.958 | 0.917 | | |
| | radyo_pilipinas_news | 19 | 1.000 | 1.000 | | |
| | philstar_psn | 17 | 0.882 | 0.941 | | |
| | gma_news | 14 | 0.929 | 0.929 | | |
| | inquirer_news | 14 | 1.000 | 1.000 | | |
| ### Measured limitations of c3 | |
| * wiki/hard pam/bcl/ilo/war gold contains English text; raw numbers understate real accuracy by ~2-3 pts there; Chavacano wiki is Spanish-like, so cbk gold is unreliable except Tatoeba (cbk F1 0.97). | |
| * basaha accuracy is inflated by counting Minasbate/Rinconada as `oth`/lenient; honest numbers are the krj / hil F1. | |
| * 1-5 word strings ~0.71 raw on hard (mostly intrinsically ambiguous items); Aklanon/Surigaonon have no class (fall into hil/oth). | |
| * Round 5 (more real minority data from FineWeb-2 `_removed`) and round-5 English additions did not help; FLORES+ (clean human pag/ilo/ceb/war evaluation) is gated on Hugging Face and needs the owner to accept its terms. | |
| * Gate v2 (tree head + named rules) is fit on dev halves of ~10k labelled items; its head is a small sklearn model (pickle gitignored; re-fit with scripts/gate_v2.py c3). | |
| * 2x tagger cost: ~7k docs/min/core on 100-word docs, ~2.2k docs/min/core on full news docs. | |
| ### Gate v2 (sealed halves; thresholds and tree head fit on dev only) | |
| | policy | accepted frac | precision accepted | precision rejected | crawl | goldspan | pld | tato | basaha | | |
| |---|---|---|---|---|---|---|---|---| | |
| | DEV: per-class thresholds | 0.630 | 0.9713 | 0.754 | 1.000 | 1.000 | 0.970 | 0.991 | 0.873 | | |
| | DEV: tree head, tau for dev precision .98 (tau=0.980) | 0.517 | 0.9803 | 0.795 | 0.993 | 0.976 | 0.980 | 0.986 | 0.960 | | |
| | DEV: tree head, same coverage as thresholds (tau=0.962) | 0.630 | 0.9713 | 0.754 | 0.995 | 0.976 | 0.976 | 0.988 | 0.876 | | |
| | DEV: both (threshold AND tree>=tau) | 0.437 | 0.9910 | 0.813 | 1.000 | 1.000 | 0.985 | 0.996 | 0.987 | | |
| | SEALED: per-class thresholds | 0.636 | 0.9644 | 0.768 | 0.984 | 0.989 | 0.970 | 0.987 | 0.847 | | |
| | SEALED: tree head, tau for dev precision .98 (tau=0.980) | 0.521 | 0.9924 | 0.785 | 1.000 | 1.000 | 0.995 | 0.996 | 0.972 | | |
| | SEALED: tree head, same coverage as thresholds (tau=0.962) | 0.647 | 0.9870 | 0.720 | 0.996 | 0.996 | 0.987 | 0.995 | 0.961 | | |
| | SEALED: both (threshold AND tree>=tau) | 0.439 | 0.9938 | 0.814 | 1.000 | 1.000 | 0.994 | 0.997 | 0.975 | | |
| On the v0.2.9.5 release (c3 sidecar, `out/ourlid_v0.2.9.5_c3.gate_v2.parquet`): accept 196,575 + accept_ph_other 10,498 (= 90.1%), review 22,644 (9.9%; low_conf_tgl 16.8k, abstain_set 3.9k, meta_low 1.8k, low_conf_eng 51), junk 2. Previous per-class-threshold gate (dev-fit): accept 86.1%, review 9.1%, accept_ph_other 4.8%. | |
| <!-- LOOP:END --> | |
| <!-- WEAK:BEGIN --> | |
| ## Weak spots re-examined on trustworthy sets (2026-10-11) | |
| The weak-spot numbers used earlier came from our own noisy gold (Wikipedia bot text, Spanish-like cbk-zam, English inside pam/bcl wiki). They were re-measured on **human-labelled external sets** that no OurLID model produced | |
| (`scripts/build_trust_sets.py`; research and licences in `docs/LID_WEAKSPOTS_RESEARCH.md`). Cells are accuracy (recall for a class) with 95% Wilson interval, all rows (dev+sealed halves are reported separately in `eval/LOOP_LOG.md`). | |
| FLORES-200 text (via SIB-200, CC BY-SA 4.0) is evaluation-only and never enters training. | |
| | weak spot (trustworthy set) | glotlid | c3r | c7r | | |
| |---|---|---|---| | |
| | Pangasinan recall (FLORES, n=1004) | 1.000 [1.00-1.00] | 0.999 [0.99-1.00] | 0.997 [0.99-1.00] | | |
| | English recall (FLORES, n=1004) | 1.000 [1.00-1.00] | 1.000 [1.00-1.00] | 1.000 [1.00-1.00] | | |
| | Spanish recall, not called Chavacano (FLORES, n=1004) | 0.998 [0.99-1.00] | 0.998 [0.99-1.00] | 0.997 [0.99-1.00] | | |
| | Chavacano recall (Tatoeba test, n=231) | 0.840 [0.79-0.88] | 0.970 [0.94-0.99] | 0.974 [0.94-0.99] | | |
| | Bikol recall (story sentences, clean) | 0.894 [0.87-0.91] | 0.920 [0.90-0.94] | 0.961 [0.95-0.97] | | |
| | Aklanon+Surigaonon -> oth (n=110, templated) | 0.300 [0.22-0.39] | 0.327 [0.25-0.42] | 0.345 [0.26-0.44] | | |
| | Short strings 2-5 words, all langs | 0.771 [0.76-0.78] | 0.837 [0.83-0.85] | 0.832 [0.82-0.84] | | |
| | of which 2 words | 0.624 [0.59-0.65] | 0.708 [0.68-0.73] | 0.704 [0.68-0.73] | | |
| | of which 3 words | 0.739 [0.71-0.76] | 0.819 [0.80-0.84] | 0.815 [0.79-0.84] | | |
| | of which 4-5 words | 0.861 [0.85-0.88] | 0.911 [0.90-0.92] | 0.905 [0.89-0.92] | | |
| | Short strings excl. ind/msa | 0.819 [0.81-0.83] | 0.866 [0.85-0.88] | 0.861 [0.85-0.87] | | |
| | Tatoeba <=4 words | 0.866 [0.83-0.89] | 0.858 [0.83-0.89] | 0.856 [0.82-0.88] | | |
| | ALLCAPS/Title/lower variants | 0.975 [0.97-0.98] | 0.999 [1.00-1.00] | 0.999 [1.00-1.00] | | |
| | English recall, ALLCAPS/Title/lower | 0.931 [0.91-0.95] | 1.000 [1.00-1.00] | 1.000 [1.00-1.00] | | |
| | Whole FLORES set (9 langs) | 0.986 [0.98-0.99] | 0.989 [0.99-0.99] | 0.989 [0.99-0.99] | | |
| ### Verdict per weak spot (final model = `c7` + marker rules `c7r`; before = `c3r` = c3 + rules; GlotLID shown as the strongest external baseline) | |
| * **Pangasinan**: not a weak spot on trustworthy data. FLORES-200 recall 0.997-0.999 (n=1,004, 95% CI >= 0.99); GlotLID 1.000. The earlier "weak" reading came from noisy wiki/PLD gold. No change made. Caveat: FLORES is formal translated text; Pangasinan web/social text is untested. | |
| * **English recall / ALLCAPS**: FLORES English 1.000, ALLCAPS/Title/lower variants 1.000 (GlotLID 0.931). On the messy crawl gold English recall is 0.94 (n=114, +-0.04) with GlotLID 0.87; the misses are Taglish frame ambiguities (eng<->tgl), not language-ID errors. No change. | |
| * **Chavacano vs Spanish**: Chavacano recall 0.974 on Tatoeba test (n=231, CI 0.94-0.99; GlotLID 0.84); FLORES Spanish recall 0.997 (n=1,004). cbk-zam Wikipedia gold is Spanish-like and stays flagged as noisy. Extra gate features (count of Chavacano-not-Spanish function words) are in gate v2. Free-text Chavacano (news/social) is not covered by any trustworthy set, and the eBible Chavacano NT could not be downloaded and is NC/ND-licensed anyway. | |
| * **Bikol**: real fix. Rinconada (a Bikol language) had been trained as `oth`, which pulled Central Bikol sentences into `oth` (40/900). Counting it as `bcl`: Bikol recall 0.920 -> 0.961 (paired dev +1.7 [+0.4,+3.0]*, sealed +2.3 [+1.1,+3.6]*); GlotLID 0.894. Cost (honest): BasahaCorpus Minasbate (a Visayan language, deliberately in `oth`) now leaks into bcl/hil: oth recall 266 -> 236 of 400; the basaha set therefore shows a significant -2.9 (dev) / -2.0 (sealed) drop even though Rinconada itself went from 34 to 311 correct of 400 under the new class definition. crawl/goldspan/golddoc/pld show no significant change (all CIs include 0). Only Central Bikol sentences were testable; Legazpi/Miraya/Rinconada free text has no trustworthy eval. | |
| * **Aklanon / Surigaonon**: no open training corpus exists (only ~28k chars of templated Wikimedia Incubator stubs + 28 Tatoeba sentences), so no class could be trained. Instead an explicit *unsupported Philippine language* bucket: `ourlid.rules` (hand-written, unit-tested markers; ALLCAPS acronyms ignored) overturns a Visayan/Tagalog-looking prediction to `oth` and records `sub=akl|sgd`. Recall on the 110-item templated set 0.291 -> 0.345 (it is low because single stub sentences rarely carry two markers; intervals 0.26-0.44). What the rules did on real data: **0 false fires** on 125k supported-class dev sentences and on every trust/gold set; on the real 229k-doc release they flag 173 Aklanon and 30 Surigaonon docs (all rmn_news), of which 39 were previously labelled ceb/bcl/tgl/hil/krj/war and 1 gold-`ceb` golddoc that is in fact Surigaonon (gold error found by the rule). They are not a classifier: recall on free Aklanon/Surigaonon text is unknown. An orthographic Aklanon `e`-for-`l` rule was tried and rejected (fires on Tagalog `babae`, English `health`, Spanish words). | |
| * **Strings under five words**: not fixed. 2 words 0.704, 3 words 0.815, 4-5 words 0.905 (c7r) vs GlotLID 0.624 / 0.739 / 0.861. Raising short-fragment augmentation from 10% to 20% (round W2) gave no gain and cost crawl accuracy (3 tgl->eng flips) so it was reverted. Roughly a third of the remaining 2-5 word errors are Indonesian<->Malay, intrinsically ambiguous. Use the gate: with c3, for 2-3 word strings confidence >= 0.99 and not abstaining keeps 41% of items at 0.914 precision (4-5 words: 60% at 0.961). | |
| **Final artefacts**: model `models/ourlid-c7` (weights gitignored; staged in `release_stage/` for the private release); sidecar `out/ourlid_v0.2.9.5_c7.gate_v2.parquet` with new columns `ourlid_label_final` (rule-adjusted label), `ourlid_oth_sub` (akl|sgd), `gate_v2`, `gate_v2_reason`, `gate_v2_p_correct`. Gate v2 counts on the 229,719 release docs (thresholds and tree head fit on dev halves only; policy tau = same coverage as thresholds): accept 192,193; review 26,535 (low_conf_tgl 22,056, abstain_set 4,119, low_conf_hil 275, meta_low 38, other 47); accept_ph_other 10,990 (incl. `unsupported_ph_akl` 173, `unsupported_ph_sgd` 30); junk 1. Gate v2 sealed precision of accepted at matched coverage 0.986. Known side effect of c7 vs c3: 106 release labels differ, 76 of them tgl->eng on Taglish-frame docs. | |
| **What still needs native review**: Aklanon and Surigaonon (the only open text is ~28k characters of templated Wikimedia Incubator stubs plus 28 Tatoeba sentences; the marker lists are derived from grammar sketches and are *unverified by native speakers*), Bikol varieties beyond Central Bikol (Rinconada/Miraya/Legazpi have no trustworthy eval text), Chavacano vs Spanish on free web text (Tatoeba n=231 is short, simple sentences), and the review bucket via `labeling/` (scripts/ingest_labels.py). | |
| <!-- WEAK:END --> | |
| <!-- S12:BEGIN --> | |
| ### Speed and English/short-string follow-up (2026-10-12; model c7 unchanged) | |
| * **Speed (round S1)**: profiling shows ~75% of CPU in the two tagger GEMMs; ONNX would not help on 1 BLAS thread/worker and was not added. New opt-in cascade `ourlid.r3.analyze_fast` (`OURLID_FAST=100:0.95` for `scripts/run_release.py`): analyse the first 100 words, escalate to the full doc if that analysis abstains or conf < 0.95. 7 worker processes, shared box: full ~291-word news docs 17.5k -> **46.9k docs/min**; ~100-word docs 57k docs/min (exact). Exactness vs full analysis on 2,376 fresh release docs: label equal 99.83%, label+abstain equal 99.3%; crawl/goldspan/PLD identical (short docs). It is NOT label-identical on long docs (0.17% differ) and conf/mixture/cs_share of fast-path docs describe the first 100 words (`ourlid_fast` column). Default remains exact. | |
| * **English recall / short strings (round S2)**: English recall is already 97.7% on 2-5-word fragments, 100% on short Tatoeba/ALLCAPS/FLORES; remaining eng<->tgl crawl "errors" are Taglish-frame ambiguity or name-only strings (gold `nol`). Residual short-string loss is msa<->ind, war->ceb/hil and spa->cbk, not English. Only gate-level change kept: English/name/Tagalog function-word features in the gate tree head (sealed precision at equal coverage +0.1 to +0.3 pt, no regression; dev gain at low coverage does not transfer). 42 of 229,719 v0.2.9.5 docs change gate decision; labels unchanged. A label-agnostic `accept_taglish` rule was tried and reverted (no precision gain). | |
| <!-- S12:END --> | |