jpaulpoliquit commited on
Commit
8f78c5a
·
verified ·
1 Parent(s): 863c520

WikaLID 1.0.1: rename from OurLID, public model card, ourlid alias kept

Browse files
Files changed (42) hide show
  1. .gitattributes +1 -0
  2. CHANGELOG.md +6 -0
  3. MODEL_LICENSE.md +2 -2
  4. README.md +286 -110
  5. dist/{ourlid-1.0.0-py3-none-any.whl → wikalid-1.0.1-py3-none-any.whl} +2 -2
  6. eval/RESULTS_v1.0.1.json +2597 -0
  7. eval/RESULTS_v1.0.1.md +117 -0
  8. eval/evaluate.py +10 -10
  9. eval/parity_himay.py +2 -2
  10. pyproject.toml +6 -5
  11. src/ourlid/__init__.py +9 -48
  12. src/ourlid/__main__.py +1 -1
  13. src/wikalid/__init__.py +50 -0
  14. src/wikalid/__main__.py +2 -0
  15. src/{ourlid → wikalid}/cbk_vs_spa.json +0 -0
  16. src/{ourlid → wikalid}/cli.py +9 -9
  17. src/{ourlid → wikalid}/doc.py +0 -0
  18. src/{ourlid → wikalid}/features.py +0 -0
  19. src/{ourlid → wikalid}/gate.py +1 -1
  20. src/{ourlid → wikalid}/head.py +0 -0
  21. src/{ourlid → wikalid}/infer.py +0 -0
  22. src/{ourlid → wikalid}/lexicon.py +0 -0
  23. src/{ourlid → wikalid}/model/MANIFEST.json +3 -2
  24. src/{ourlid → wikalid}/model/conformal.json +0 -0
  25. src/{ourlid → wikalid}/model/ensemble.txt +0 -0
  26. src/{ourlid → wikalid}/model/gate_head.npz +0 -0
  27. src/{ourlid → wikalid}/model/gate_thresholds.json +0 -0
  28. src/{ourlid → wikalid}/model/head.npz +0 -0
  29. src/{ourlid → wikalid}/model/head_report.json +0 -0
  30. src/{ourlid → wikalid}/model/lexicon.json +0 -0
  31. src/{ourlid → wikalid}/model/model_int8.npz +0 -0
  32. src/{ourlid → wikalid}/model/model_int8_b.npz +0 -0
  33. src/{ourlid → wikalid}/model/r3opts.json +0 -0
  34. src/{ourlid → wikalid}/model/train_cfg.json +0 -0
  35. src/{ourlid → wikalid}/predictor.py +0 -0
  36. src/{ourlid → wikalid}/r3.py +0 -0
  37. src/{ourlid → wikalid}/rules.py +0 -0
  38. src/{ourlid → wikalid}/sidecar.py +3 -3
  39. src/{ourlid → wikalid}/synth.py +0 -0
  40. src/{ourlid → wikalid}/textutil.py +0 -0
  41. tests/test_package.py +24 -15
  42. tests/test_rules.py +1 -1
.gitattributes CHANGED
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  dist/ourlid-1.0.0-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
 
 
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  dist/ourlid-1.0.0-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
37
+ dist/wikalid-1.0.1-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
CHANGELOG.md CHANGED
@@ -1,5 +1,11 @@
1
  # Changelog
2
 
 
 
 
 
 
 
3
  ## 1.0.0 — 2026-11-15
4
  First versioned package. Same weights and decisions as the research model `c7` + gate v2 that labelled Himay
5
  v0.2.9.5 → v0.3.0.4 (labels/gates bit-identical on a 3,000-doc Himay sample and on every eval item).
 
1
  # Changelog
2
 
3
+ ## 1.0.1 — 2026-11-15
4
+ Renamed OurLID → **WikaLID** (*wika* = "language" in Tagalog). Package and CLI are now `wikalid`; `import ourlid`,
5
+ `python -m ourlid` and the `ourlid` command remain as aliases. Weights, outputs and sidecar column names (`ourlid_*`)
6
+ are unchanged: all 17 sidecar columns bit-identical to 1.0.0 on a 3,000-doc Himay sample, and every evaluation number
7
+ identical (`eval/RESULTS_v1.0.1.md`). Public model card rewritten.
8
+
9
  ## 1.0.0 — 2026-11-15
10
  First versioned package. Same weights and decisions as the research model `c7` + gate v2 that labelled Himay
11
  v0.2.9.5 → v0.3.0.4 (labels/gates bit-identical on a 3,000-doc Himay sample and on every eval item).
MODEL_LICENSE.md CHANGED
@@ -1,7 +1,7 @@
1
- # OurLID 1.0.0 — licence status
2
 
3
  * **Code** (`src/`, `eval/*.py`, `tests/`): Apache-2.0 (`LICENSE`).
4
- * **Model weights** (`src/ourlid/model/`): **private preview — no licence granted.** Internal use by sapinsapin only;
5
  do not redistribute, mirror or make public. The public licence is an open decision for the owner.
6
  Intended licence if published: **CC BY-SA 4.0** (share-alike, because Wikipedia CC BY-SA text is the largest training
7
  source), with attribution to Wikipedia contributors, Tatoeba contributors, FineWeb-2 (ODC-By) and BasahaCorpus
 
1
+ # WikaLID 1.0.1 (formerly OurLID) — licence status
2
 
3
  * **Code** (`src/`, `eval/*.py`, `tests/`): Apache-2.0 (`LICENSE`).
4
+ * **Model weights** (`src/wikalid/model/`): **private preview — no licence granted.** Internal use by sapinsapin only;
5
  do not redistribute, mirror or make public. The public licence is an open decision for the owner.
6
  Intended licence if published: **CC BY-SA 4.0** (share-alike, because Wikipedia CC BY-SA text is the largest training
7
  source), with attribution to Wikipedia contributors, Tatoeba contributors, FineWeb-2 (ODC-By) and BasahaCorpus
README.md CHANGED
@@ -1,138 +1,314 @@
1
  ---
2
  license: other
3
- license_name: ourlid-private-preview
4
- license_link: https://huggingface.co/sapinsapin/ourlid/blob/main/MODEL_LICENSE.md
5
- language: [tl, ceb, hil, war, ilo, bcl, pam, pag, cbk, krj, tsg, en, es, id, ms]
6
- library_name: ourlid
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7
  pipeline_tag: text-classification
8
- tags: [language-identification, lid, philippines, taglish, code-switching, filipino, tagalog, cebuano, hiligaynon, numpy]
 
 
 
 
 
 
 
 
 
 
 
9
  ---
10
 
11
- # OurLID 1.0.0 — language identification for Philippine text
12
 
13
- **Status: private preview (sapinsapin). Not a public release.** Weights may not be redistributed until the owner decides
14
- on a public licence (see `MODEL_LICENSE.md`).
15
 
16
- OurLID tells you which language a Philippine text is in — Tagalog/Filipino (including Taglish), Cebuano, Hiligaynon,
17
- Waray, Ilocano, Bikol, Kapampangan, Pangasinan, Chavacano, plus experimental Kinaray-a and Tausug — and separates them
18
- from English, Spanish, Indonesian, Malay, "another language" and "not language at all" (lists, rosters, score tables).
19
- It also says **how mixed** the text is (English share, `cs_level` none/light/heavy) and **how sure it is** (calibrated
20
- confidence, a conformal label set, and a gate decision: accept / review / junk).
21
 
22
- It is the language tool behind the Himay corpus (`lang_tool` column) and runs on CPU with numpy only.
23
 
24
- ## Use
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
25
 
26
  ```bash
27
- pip install ourlid-1.0.0-py3-none-any.whl # numpy only; add pyarrow for parquet batch jobs
28
- ourlid predict --gate "Nag-submit na ako ng report pero hindi pa daw na-approve."
29
- ourlid sidecar documents.parquet labels.parquet --workers 3 # columns document_id, source_name, text
30
- ourlid verify --expect-version 1.0.0 # weight sha256 + version pin
31
  ```
 
 
 
 
 
32
  ```python
33
- import ourlid
34
- ourlid.classify("Indi ko mahibal-an kon ngaa wala sila nagkadto.")
35
- # {'label': 'hil', 'conf': 0.999, 'label_set': ['hil'], 'abstain': False, 'mixture': {'hil': 1.0}, 'cs_level': 'none',
36
- # 'label_final': 'hil', 'oth_sub': None, 'gate': 'accept_ph_other', 'gate_reason': 'accepted', 'gate_p_correct': 0.997, ...}
 
 
 
 
 
 
 
 
 
 
37
  ```
38
 
39
- | output | meaning |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
40
  |---|---|
41
- | `label` | majority language (standard LID convention) |
42
- | `label_final` | `label`, or `oth` when the Aklanon/Surigaonon marker rule fires (`oth_sub` = `akl` / `sgd`) |
43
- | `conf` | temperature-scaled probability of `label` |
44
- | `label_set`, `abstain` | split-conformal set at alpha 0.10; `abstain` = set is not exactly one label |
45
- | `mixture`, `cs_level`, `cs_share` | token-level language shares; Taglish is a statistic (English share), not a class |
46
- | `gate`, `gate_reason`, `gate_p_correct` | gate v2: `accept` (tgl) / `accept_eng` / `accept_ph_other` / `review` / `junk` |
 
 
 
 
 
 
 
47
 
48
- ## Labels (17)
 
 
 
 
 
 
49
 
50
- `tgl ceb hil war ilo bcl pam pag cbk krj tsg eng spa ind msa oth nol`. `krj` (Kinaray-a) and `tsg` (Tausug) are
51
- experimental (web-only training text, no trustworthy test set for free text). Rinconada is counted as Bikol (`bcl`).
52
- `oth` = any other language, including the rule bucket for Aklanon/Surigaonon. `nol` = not a language.
53
 
54
- ## How it works
 
55
 
56
- 1. Two independently seeded token taggers (hashed character n-grams + morphology signatures, int8 weights, 26 MB each)
57
- label every word; posteriors are averaged, smoothed by a sticky HMM over words, with extra "ALLCAPS→Title" and
58
- "accent-stripped" views.
59
- 2. A small softmax head turns the word-level mixture plus closed-class lexicon evidence into a document label.
60
- 3. Split-conformal calibration gives a label set; a 150-tree gradient-boosted gate head (fit on dev halves only, exported
61
- to plain arrays, no pickle) plus named rules decides accept / review / junk.
62
- 4. Hand-written, unit-tested marker rules move Aklanon/Surigaonon-looking text into `oth` (they never create a supported label).
63
 
64
- ## Benchmarks (sealed halves, never used for a decision)
65
 
66
- All numbers are produced by `eval/evaluate.py` (full tables: `eval/RESULTS_v1.0.0.md`). Baselines were run live on the
67
- same items on 2026-11-15 and match the cached research-loop predictions item for item. Accuracy on corrected gold
68
- (known label noise in wiki/hard/PLD/Chavacano fixed by GlotLID/OpenLID/lid176 consensus — a rule that, if anything,
69
- favours those baselines).
 
 
 
 
 
 
70
 
71
- | set (sealed n) | what it is | OurLID 1.0.0 | GlotLID v3 | OpenLID-v3 | fastText lid.176 |
 
 
72
  |---|---|---|---|---|---|
73
- | crawl realism (335) | private: real Filipino news spans incl. Taglish, lists, bylines | **0.949** | 0.875 | 0.785 | 0.734 |
74
- | news gold spans (327) | private: hand-labelled news spans (single annotator) | **0.939** | 0.899 | 0.881 | 0.783 |
75
- | news gold docs (148) | private: hand-labelled whole articles | **0.986** | 0.959 | 0.919 | 0.818 |
76
- | PLD read speech (1,653) | private: UP Philippine Languages Database prompts | **0.917** | 0.834 | 0.417 | 0.341 |
77
- | Wikipedia held-out (2,563) | public, `eval/data` | **0.925** | 0.907 | 0.710 | 0.664 |
78
- | PH-LID-Hard v0 (6,379) | public, `eval/data` (synthetic noise: caps, SMS, typos, Taglish, lists) | **0.901** | 0.829 | 0.641 | 0.599 |
79
- | BasahaCorpus stories (794) | private: children's stories (Kinaray-a, Hiligaynon; Minasbate scored as `oth`, which flatters every tool) | **0.739** | 0.583 | 0.026 | 0.031 |
80
- | Tatoeba test (968) | public, `eval/data` | 0.928 | **0.955** | 0.758 | 0.805 |
81
- | **mean** | | **0.911** | 0.855 | 0.642 | 0.597 |
82
-
83
- Paired bootstrap vs GlotLID (sealed): OurLID is better on 7 of 8 sets with 95% intervals above zero (crawl +7.5 pts,
84
- PLD +8.3, PH-LID-Hard +7.2, Basaha +15.6, gold spans +4.0, gold docs +2.7, Wikipedia +1.8) and **worse on Tatoeba
85
- (−2.7 pts, CI −4.6 to −0.9)** — very short, clean sentences, where GlotLID's breadth wins.
86
-
87
- Human-labelled external sets (all rows): FLORES-200 via SIB-200, 9 languages, n=9,036: 0.989 (GlotLID 0.986, OpenLID
88
- 0.985). ALLCAPS/Title/lower variants: 0.999 (GlotLID 0.975). Central Bikol story sentences: Bikol recall 0.961 (GlotLID
89
- 0.894). 2–5-word fragments: 0.832 (GlotLID 0.771); 2 words alone 0.704.
90
-
91
- **Gate v2.** On sealed halves, rows the gate accepts are 98.5–100% correct while covering 36–78% of these deliberately
92
- hard sets. On real news (Himay parent, 240,386 docs) it accepts 88.6% and routes 11.4% to `review`.
93
-
94
- **Speed.** ~2–7k docs/min per core depending on length (two taggers); 3 workers labelled a 3,000-article sample in ~10 s
95
- end to end on a shared 8-core box.
96
-
97
- ## Known limits (read before trusting a label)
98
-
99
- * **Surigaonon vs Cebuano, Aklanon vs Hiligaynon:** no class exists (no open training text). The marker rule catches only
100
- 34.5% of a templated Aklanon/Surigaonon set (CI 0.26–0.44); everything else is called `ceb`/`hil`. Recall on free text is unknown.
101
- * **Hiligaynon vs Kinaray-a vs Cebuano:** improved (Basaha +15.6 pts vs GlotLID) but still the main confusion family;
102
- `krj` is experimental.
103
- * **`krj` / `pag` labels on news:** on Himay, sampled `krj` (3/3) and `pag` (2/2) rows were lists, Aklanon or Hiligaynon.
104
- Himay forces them to `review`.
105
- * **Tagalog-dominant Taglish labelled `eng`:** English-heavy showbiz/tabloid Taglish with ALLCAPS leads can come out
106
- `eng`; the conformal set is then usually empty and the gate sends it to `review` (when OurLID abstains on `eng`, the
107
- top label is right only ~22% of the time). Use `gate != "review"` as the clean default.
108
- * **Short strings:** 2 words 0.70, 3 words 0.82; a third of the errors are Indonesian↔Malay.
109
- * **Tatoeba-style short clean sentences:** GlotLID is better (see above).
110
- * **Gold quality:** the news gold sets have a single, non-native annotator; native-speaker labels are pending.
111
- * **Calibration:** cross-domain conformal coverage at alpha 0.10 measured 0.82–0.92 against a 0.90 target.
112
-
113
- ## Training data (none of it ships in this repo)
114
-
115
- Wikipedia dumps (CC BY-SA 4.0), Tatoeba (CC BY 2.0 FR), FineWeb-2 web text (ODC-By 1.0; page copyright stays with
116
- publishers; used for training only, never redistributed), BasahaCorpus stories (CC BY 4.0 per its authors, Imperial &
117
- Kochmar 2023 / Let's Read Asia). **No scraped news and no Himay text is in the training data.** GlotLID/OpenLID/lid176
118
- were used only as label-noise detectors; no third-party weights are inside OurLID. Details: `docs/DATA_CARD.md`.
119
-
120
- What ships: code (Apache-2.0), weights, and the three public evaluation sets derived from Wikipedia/Tatoeba
121
- (`eval/data`, CC BY-SA 4.0 / CC BY 2.0 FR). Private evaluation sets (news gold, crawl realism, PLD, Basaha, trust sets)
122
- and the native-speaker labelling sheets are **not** included.
123
-
124
- ## Reproduce
125
 
126
- ```bash
127
- pip install dist/ourlid-1.0.0-py3-none-any.whl pyarrow pytest
128
- pytest -q tests # package tests
129
- python eval/run_baselines.py # (python with fasttext) GlotLID / OpenLID / lid176, live
130
- python eval/evaluate.py --workers 4 # -> eval/RESULTS_v1.0.0.{md,json}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
131
  ```
132
- Private sets are read from `$OURLID_PRIVATE_EVAL` and skipped when absent; the public three always run.
133
 
134
- ## Citation / provenance
 
 
 
 
 
135
 
136
- Built 2026-10-10 → 2026-10-12 (rounds r1–r3, c1–c7, weak-spot loop W1–W3, gate v2), frozen as 1.0.0 on 2026-11-15.
137
- Weights = research model `c7` (taggers `tok_c7a` + `tok_c7b`) with gate v2 (English/name features). Labels are
138
- bit-identical to the research pipeline that labelled Himay v0.3.0.4 (`eval/PARITY_HIMAY_v1.0.0.json`).
 
1
  ---
2
  license: other
3
+ license_name: wikalid-preview
4
+ license_link: https://huggingface.co/sapinsapin/wikalid/blob/main/MODEL_LICENSE.md
5
+ language:
6
+ - tl
7
+ - ceb
8
+ - hil
9
+ - war
10
+ - ilo
11
+ - bcl
12
+ - pam
13
+ - pag
14
+ - cbk
15
+ - krj
16
+ - tsg
17
+ - en
18
+ - es
19
+ - id
20
+ - ms
21
+ - multilingual
22
+ library_name: wikalid
23
  pipeline_tag: text-classification
24
+ tags:
25
+ - language-identification
26
+ - lid
27
+ - philippines
28
+ - filipino
29
+ - tagalog
30
+ - taglish
31
+ - code-switching
32
+ - cebuano
33
+ - hiligaynon
34
+ - low-resource
35
+ - cpu
36
  ---
37
 
38
+ # WikaLID
39
 
40
+ **Language identification for Philippine languages, with code-switching measurement and calibrated abstention.**
 
41
 
42
+ *Wika* is Tagalog for "language". WikaLID was released as OurLID 1.0.0; 1.0.1 has the same weights under the new name.
 
 
 
 
43
 
44
+ ## Overview
45
 
46
+ General-purpose language identifiers such as GlotLID, OpenLID and fastText `lid.176` cover hundreds of languages, but
47
+ they struggle with Philippine web and news text. They mix up the Visayan languages, fail on ALLCAPS headlines and
48
+ tabloid style, can't separate Chavacano from Spanish, and treat Tagalog–English code-switching (Taglish) as noise.
49
+ WikaLID is a small, CPU-only model built for that setting:
50
+
51
+ * **17 labels**: 11 Philippine languages, 4 contact languages (English, Spanish, Indonesian, Malay), "other language"
52
+ and "not a language".
53
+ * **Token-level tagging.** Every word gets a language, so the output includes the language mixture and the English
54
+ share. Taglish is reported as a measurement, not a separate class.
55
+ * **Calibrated uncertainty.** It returns a temperature-scaled confidence and a conformal label set, plus a gate that
56
+ sorts each text into *accept*, *review* or *junk*.
57
+ * **Small and deterministic.** Two 26 MB int8 taggers, numpy-only inference, no GPU, no PyTorch, no fastText.
58
+
59
+ It was built to label a Filipino text corpus for language-model pretraining. The gate exists so that uncertain rows
60
+ can be held back instead of being silently mislabelled.
61
+
62
+ ## Supported labels
63
+
64
+ | code | language | notes |
65
+ |---|---|---|
66
+ | `tgl` | Tagalog / Filipino | Includes Taglish. Code-switching is reported in `cs_level` / `cs_share` |
67
+ | `ceb` | Cebuano | |
68
+ | `hil` | Hiligaynon | Most frequent confusions are with `ceb` and `krj` |
69
+ | `war` | Waray | |
70
+ | `ilo` | Ilocano | |
71
+ | `bcl` | Bikol | Central Bikol; Rinconada is also labelled `bcl` |
72
+ | `pam` | Kapampangan | |
73
+ | `pag` | Pangasinan | Strong on formal text; rare in news, so treat news predictions with care |
74
+ | `cbk` | Chavacano | Separated from Spanish |
75
+ | `krj` | Kinaray-a | **Experimental** |
76
+ | `tsg` | Tausug | **Experimental**: no human-labelled free-text test set |
77
+ | `eng` | English | |
78
+ | `spa` | Spanish | |
79
+ | `ind` | Indonesian | Often confused with `msa` on very short strings |
80
+ | `msa` | Malay | |
81
+ | `oth` | any other language | Includes Aklanon / Surigaonon detected by marker rules (`oth_sub` = `akl` / `sgd`) |
82
+ | `nol` | not a language | Numbered lists, name rosters, score tables (≈ ISO `zxx`) |
83
+
84
+ ## Install
85
+
86
+ WikaLID is distributed as a wheel with the weights bundled (about 52 MB). It needs Python ≥ 3.10 and numpy.
87
 
88
  ```bash
89
+ pip install wikalid-1.0.1-py3-none-any.whl # add pyarrow for parquet batch jobs
90
+ wikalid verify --expect-version 1.0.1 # checks the version and the sha256 of every weight file
 
 
91
  ```
92
+
93
+ ## Quickstart
94
+
95
+ ### Python
96
+
97
  ```python
98
+ import wikalid
99
+
100
+ lid = wikalid.load() # bundled model; deterministic
101
+ lid.analyze("Nag-submit na ako ng report pero hindi pa daw na-approve.")
102
+ # {'label': 'tgl', 'conf': 0.9968, 'label_set': ['tgl'], 'abstain': False,
103
+ # 'mixture': {'tgl': 0.9998}, 'cs_level': 'none', 'cs_share': 2.3e-06, ...}
104
+
105
+ lid.analyze("Grabe yung traffic kanina sa EDSA, super late na ako sa meeting with the client.")
106
+ # {'label': 'tgl', 'conf': 0.9344, 'label_set': ['tgl'], 'mixture': {'tgl': 0.7477, 'eng': 0.252},
107
+ # 'cs_level': 'heavy', 'cs_share': 0.252, ...}
108
+
109
+ wikalid.classify("Indi ko mahibal-an kon ngaa wala sila nagkadto.") # analysis + gate decision
110
+ # {..., 'label': 'hil', 'conf': 0.9993, 'label_final': 'hil', 'oth_sub': None,
111
+ # 'gate': 'accept_ph_other', 'gate_reason': 'accepted', 'gate_p_correct': 0.997}
112
  ```
113
 
114
+ ### Command line
115
+
116
+ ```bash
117
+ $ wikalid predict "Wala ko kahibalo nganong wala sila miadto."
118
+ {"text": "Wala ko kahibalo nganong wala sila miadto.", "label": "ceb", "conf": 0.9895, "abstain": false,
119
+ "label_set": ["ceb"], "mixture": {"ceb": 1.0}, "cs_level": "none", ...}
120
+
121
+ $ echo "Ano ang pangalan mo?" | wikalid predict --gate
122
+ {"text": "Ano ang pangalan mo?", "label": "tgl", "conf": 0.9967, ..., "gate": "review", "gate_reason": "meta_low", "gate_p_correct": 0.504}
123
+
124
+ $ wikalid sidecar docs.parquet labels.parquet --workers 4 # input columns: document_id, source_name, text
125
+ ```
126
+
127
+ The outputs above are abridged; values are rounded from real runs of 1.0.1.
128
+
129
+ `wikalid sidecar` writes one row per document with the model columns (`ourlid_label`, `ourlid_conf`,
130
+ `ourlid_label_set`, `ourlid_mixture`, `ourlid_cs_share`, …) and the gate columns (`ourlid_label_final`, `ourlid_oth_sub`,
131
+ `gate_v2`, `gate_v2_reason`, `gate_v2_p_correct`). The `ourlid_` prefix is kept for compatibility with existing
132
+ consumers. `wikalid score` and `wikalid gate` run the two steps separately. `wikalid info` prints the version, labels and
133
+ weight hashes.
134
+
135
+ ### Output fields
136
+
137
+ | field | meaning |
138
  |---|---|
139
+ | `label` | Majority language of the text (standard LID convention) |
140
+ | `conf` | Temperature-scaled probability of `label` |
141
+ | `label_set`, `abstain` | Split-conformal prediction set at α = 0.10; `abstain` is true when the set is not exactly one label |
142
+ | `mixture` | Share of word tokens per language (languages ≥ 1%) |
143
+ | `cs_level`, `cs_share` | English share for non-English texts (`none` < 0.03 ≤ `light` < 0.25 ≤ `heavy`) |
144
+ | `label_final`, `oth_sub` | `label`, or `oth` + `akl`/`sgd` when the Aklanon/Surigaonon marker rule fires |
145
+ | `gate`, `gate_reason`, `gate_p_correct` | Gate decision (below) |
146
+
147
+ ## The gate: accept, review, junk
148
+
149
+ A label is only useful if you know when to trust it. The gate combines named rules with a small gradient-boosted
150
+ "is this label correct?" model. That model was fit only on development data, so its scores are not tuned to the test
151
+ sets reported below.
152
 
153
+ | gate | meaning | example (real outputs) |
154
+ |---|---|---|
155
+ | `accept` | confident Tagalog | "Nag-submit na ako ng report…" → `tgl`, p_correct 0.995 |
156
+ | `accept_ph_other` | confident other Philippine language, or a marker-rule hit | "Indi ko mahibal-an…" → `hil`; an Aklanon sentence → `oth`/`akl`, reason `unsupported_ph_akl` |
157
+ | `accept_eng` | confident English | |
158
+ | `review` | the model abstained, the text is very short, a sibling language is close, or confidence is below that label's threshold | heavy Taglish → `tgl` but reason `low_conf_tgl`; "Ano ang pangalan mo?" → reason `meta_low` |
159
+ | `junk` | not language | a numbered name roster → `nol`, reason `junk_nol` |
160
 
161
+ The gate is deliberately strict. The English threshold is very high, so many clean English sentences land in `review`.
162
+ That suits a corpus where English is opt-in, but you may want your own thresholds for other uses.
 
163
 
164
+ Rows the gate accepts were 97.2–100% correct on every sealed evaluation set, and at least 98.5% correct on every set except BasahaCorpus (table below). On about 240k Filipino
165
+ news articles, it accepted 88.6% and sent 11.4% to review.
166
 
167
+ ## Evaluation
 
 
 
 
 
 
168
 
169
+ ### Method
170
 
171
+ * Each evaluation set is split 50/50 by a hash of a group key, so related items stay on the same side. Every modelling
172
+ and threshold decision was made on the *dev* half. The numbers below are from the *sealed* half, which was never
173
+ used for any decision.
174
+ * All baselines were run on exactly the same items with their published models.
175
+ * "Corrected gold": some gold labels in the Wikipedia-derived sets are known to be wrong (English sentences inside
176
+ regional-language wikis, Spanish inside Chavacano). Where at least two of GlotLID, OpenLID and lid.176 agree on
177
+ English or Spanish with probability ≥ 0.6, the gold is corrected. This rule relies on the baselines, so it can only
178
+ favour them.
179
+ * Three sets are public and ship with the source repository: Wikipedia held-out, PH-LID-Hard v0 and the Tatoeba test
180
+ split. The others are private, because they contain copyrighted news text or come from restricted sources.
181
 
182
+ ### Results (sealed halves; accuracy on corrected gold)
183
+
184
+ | set (n) | description | WikaLID | GlotLID v3 | OpenLID-v3 | fastText lid.176 |
185
  |---|---|---|---|---|---|
186
+ | Crawl realism (335) | Real Filipino news spans: Taglish, headlines, lists, bylines | **0.949** | 0.875 | 0.785 | 0.734 |
187
+ | News gold spans (327) | Hand-labelled news spans | **0.939** | 0.899 | 0.881 | 0.783 |
188
+ | News gold docs (148) | Hand-labelled whole articles | **0.986** | 0.959 | 0.919 | 0.818 |
189
+ | PLD read speech (1,653) | Prompts from the Philippine Languages Database, 10 languages | **0.917** | 0.834 | 0.417 | 0.341 |
190
+ | Wikipedia held-out (2,563) | Public | **0.925** | 0.907 | 0.710 | 0.664 |
191
+ | PH-LID-Hard v0 (6,379) | Public; synthetic noise: caps, SMS spelling, typos, Taglish, lists | **0.901** | 0.829 | 0.641 | 0.599 |
192
+ | BasahaCorpus (794) | Children's stories (Kinaray-a, Hiligaynon, …) | **0.739** | 0.583 | 0.026 | 0.031 |
193
+ | Tatoeba test (968) | Public; short, clean sentences | 0.928 | **0.955** | 0.758 | 0.805 |
194
+ | **Mean** | | **0.911** | 0.855 | 0.642 | 0.597 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
195
 
196
+ Paired bootstrap against GlotLID (2,000 resamples, 95% CI) on the sealed halves:
197
+
198
+ * WikaLID is better on seven sets: crawl +7.5 pts [+4.5, +11.0], gold spans +4.0 [+0.9, +7.3], gold docs +2.7
199
+ [+0.7, +5.4], PLD +8.3 [+6.7, +9.9], Wikipedia +1.8 [+1.0, +2.7], PH-LID-Hard +7.2 [+6.3, +8.0], Basaha +15.6
200
+ [+12.3, +18.9].
201
+ * **It is worse on Tatoeba: −2.7 pts [−4.6, −0.9].** On short, clean, textbook-style sentences, GlotLID's breadth wins.
202
+
203
+ ### Human-labelled external sets (all items)
204
+
205
+ | set | WikaLID | GlotLID v3 | OpenLID-v3 | lid.176 |
206
+ |---|---|---|---|---|
207
+ | FLORES-200 via SIB-200, 9 languages (9,036) | **0.989** | 0.986 | 0.985 | 0.774 |
208
+ | Same sentences in UPPER / Title / lower case (5,388) | **0.999** | 0.975 | 0.575 | 0.522 |
209
+ | Central Bikol story sentences: Bikol recall (900) | **0.961** | 0.894 | 0.000 | 0.003 |
210
+ | 2–5 word fragments of FLORES sentences (4,320) | **0.832** | 0.771 | 0.773 | 0.590 |
211
+ | … of which 2 words (1,080) | **0.704** | 0.624 | 0.621 | 0.514 |
212
+ | Short Tatoeba sentences (543) | 0.856 | **0.866** | 0.632 | 0.685 |
213
+ | Aklanon / Surigaonon sentences → `oth` (110) | **0.345** | 0.300 | 0.082 | 0.055 |
214
+
215
+ ### Gate (sealed halves)
216
+
217
+ | set | share accepted | accuracy of accepted rows |
218
+ |---|---|---|
219
+ | Crawl realism | 0.510 | 1.000 |
220
+ | News gold spans | 0.514 | 1.000 |
221
+ | News gold docs | 0.784 | 0.991 |
222
+ | PLD read speech | 0.658 | 0.985 |
223
+ | Wikipedia held-out | 0.566 | 0.989 |
224
+ | PH-LID-Hard v0 | 0.510 | 0.994 |
225
+ | BasahaCorpus | 0.358 | 0.972 |
226
+ | Tatoeba test | 0.395 | 0.995 |
227
+
228
+ These sets are deliberately hard and short, so acceptance rates are much lower than on ordinary news.
229
+
230
+ ### Speed
231
+
232
+ Measured on a shared 8-core CPU, one BLAS thread per worker: about 7,000 documents per minute per core for 100-word
233
+ documents, and about 2,200 per minute per core for full news articles (~400 tokens). Throughput scales roughly
234
+ linearly with workers.
235
+
236
+ ## Limitations and bias
237
+
238
+ * **Languages without a class.** Aklanon, Surigaonon and many smaller Philippine languages have no open training text.
239
+ Hand-written marker rules move only about a third of Aklanon/Surigaonon sentences to `oth`. The rest are labelled
240
+ `hil` or `ceb`. Do not use WikaLID to count speakers of unsupported languages.
241
+ * **The Visayan family.** Hiligaynon, Kinaray-a and Cebuano remain the main source of errors. `krj` and `tsg` are
242
+ experimental.
243
+ * **English-heavy Taglish.** Tabloid and showbiz text with ALLCAPS leads and English quotes can be labelled `eng`.
244
+ Usually the label set comes back empty and the gate sends the text to `review`. When WikaLID abstains on `eng`, the top
245
+ label is right only about 22% of the time on our gold data.
246
+ * **Short strings.** Two-word inputs are 70% accurate, and a third of the remaining errors are Indonesian vs Malay.
247
+ * **Rare labels on news.** In a manual sample of news predictions, every `krj` (3/3) and `pag` (2/2) prediction was
248
+ wrong: lists, Aklanon or Hiligaynon text. Route these to review.
249
+ * **Calibration across domains.** The conformal sets target 90% coverage at α = 0.10, but that target holds only near
250
+ the calibration data. On the sealed halves, the gold label was inside the set for 82–91% of items on the news, speech,
251
+ Wikipedia, PH-LID-Hard and Tatoeba sets, and for 56% on BasahaCorpus. Between 8% and 29% of sets come back empty, which
252
+ means "no label is safe"; treat an empty set as a signal to review, not as a prediction.
253
+ * **Evaluation caveats.**
254
+ * The news gold sets were labelled by a single, non-native annotator. A native-speaker re-annotation is planned.
255
+ * PH-LID-Hard is synthetic and shares design ideas with the training-data augmentation.
256
+ * BasahaCorpus accuracy counts Minasbate as `oth`, which flatters every system.
257
+ * Real-news results should carry the most weight.
258
+ * **Register and region.** The training text is mostly encyclopedic, web and story text in standard orthographies.
259
+ Older spellings, regional varieties and social-media slang are less well covered.
260
+
261
+ ## Training data
262
+
263
+ No training text is distributed with the model. The taggers were trained on sentences from:
264
+
265
+ * Wikipedia (Tagalog, Cebuano, Waray, Ilocano, Bikol, Kapampangan, Pangasinan, Chavacano, English, Spanish, Indonesian,
266
+ Malay and about 30 other languages): CC BY-SA 4.0.
267
+ * Tatoeba: CC BY 2.0 FR.
268
+ * FineWeb-2 web text for the lower-resource languages, filtered for lexical purity: ODC-By 1.0. Page copyright remains
269
+ with the publishers, and this text is used for training only.
270
+ * BasahaCorpus children's stories (Imperial & Kochmar, 2023) for Kinaray-a and Hiligaynon: CC BY 4.0 as stated by its
271
+ authors.
272
+
273
+ Training sentences are augmented with synthetic code-switching, casing, SMS-style spelling and accent variants. No
274
+ scraped news is in the training data. Existing LID tools (GlotLID, OpenLID, lid.176) were used only to detect
275
+ label noise. No third-party model weights are included.
276
+
277
+ ## Versioning
278
+
279
+ * Releases follow semantic versioning. A version always pins exact weights. `wikalid verify --expect-version X`
280
+ checks the version and the sha256 of every bundled file against `MANIFEST.json`.
281
+ * **1.0.1** (2026-11-15): renamed from OurLID to WikaLID. Weights and outputs are identical to 1.0.0. `import ourlid`
282
+ and the `ourlid` command remain as aliases.
283
+ * **1.0.0** (2026-11-15): first versioned release, as OurLID.
284
+ * A retrained model will be released as a new minor version with a fresh evaluation. Weights are never swapped silently.
285
+
286
+ ## License
287
+
288
+ * **Code:** Apache-2.0.
289
+ * **Model weights:** preview release, all rights reserved; redistribution is not permitted. A public licence is under
290
+ consideration. Because Wikipedia is the largest training source, CC BY-SA 4.0 is the likely choice. See
291
+ `MODEL_LICENSE.md`.
292
+ * **Public evaluation sets:** CC BY-SA 4.0 (Wikipedia-derived) and CC BY 2.0 FR (Tatoeba).
293
+
294
+ ## Citation
295
+
296
+ ```bibtex
297
+ @software{wikalid_2026,
298
+ title = {WikaLID: Language Identification for Philippine Languages},
299
+ author = {{sapinsapin}},
300
+ year = {2026},
301
+ version = {1.0.1},
302
+ url = {https://huggingface.co/sapinsapin/wikalid}
303
+ }
304
  ```
 
305
 
306
+ ## Acknowledgements and contact
307
+
308
+ WikaLID builds on public resources from Wikipedia and Tatoeba contributors, FineWeb-2, BasahaCorpus (Imperial &
309
+ Kochmar, 2023; stories from Let's Read Asia / The Asia Foundation), FLORES-200 / SIB-200 (used for evaluation only) and
310
+ the Philippine Languages Database (used for evaluation only). Baselines: GlotLID (Kargaran et al., 2023), OpenLID
311
+ (Burchell et al., 2023) and fastText lid.176 (Joulin et al., 2016).
312
 
313
+ Questions, error reports and offers to help with native-speaker review are welcome in the Community tab of this
314
+ repository.
 
dist/{ourlid-1.0.0-py3-none-any.whl → wikalid-1.0.1-py3-none-any.whl} RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:a4682f767bd8f1875495d04ffa34ba6523e9cf15e267ee115bf04d4961bd4eda
3
- size 52466537
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6815be3abfcad2f538096489d32e8e68c0ee1e46c5aa533787644fe6be5c249d
3
+ size 52469878
eval/RESULTS_v1.0.1.json ADDED
@@ -0,0 +1,2597 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "version": "wikalid-1.0.1",
3
+ "speed_items_per_s": {
4
+ "crawl": 260.0556615657041,
5
+ "goldspan": 279.75437264136514,
6
+ "golddoc": 82.7144606130041,
7
+ "pld": 361.29866678479993,
8
+ "wiki": 412.01515781078336,
9
+ "hard": 417.6514915312866,
10
+ "basaha": 340.77461396884735,
11
+ "tato": 416.0644453947067,
12
+ "t_sib": 433.517085761704,
13
+ "t_frag": 413.52074818541155,
14
+ "t_caps": 394.02282398376394,
15
+ "t_bikol": 390.347293016329,
16
+ "t_aklsgd": 90.24064579563718,
17
+ "t_tshort": 312.07255571899447
18
+ },
19
+ "core": {
20
+ "sealed": {
21
+ "crawl": {
22
+ "n": 335,
23
+ "public": false,
24
+ "wikalid-1.0.1": {
25
+ "acc_raw": 0.9492537313432836,
26
+ "acc_corrected": 0.9492537313432836,
27
+ "macro_f1_corrected": 0.9452212148681051
28
+ },
29
+ "glotlid": {
30
+ "acc_raw": 0.8746268656716418,
31
+ "acc_corrected": 0.8746268656716418,
32
+ "macro_f1_corrected": 0.9258273703041144
33
+ },
34
+ "openlid3": {
35
+ "acc_raw": 0.7850746268656716,
36
+ "acc_corrected": 0.7850746268656716,
37
+ "macro_f1_corrected": 0.6166170580237387
38
+ },
39
+ "lid176": {
40
+ "acc_raw": 0.7343283582089553,
41
+ "acc_corrected": 0.7343283582089553,
42
+ "macro_f1_corrected": 0.5681250135015445
43
+ }
44
+ },
45
+ "goldspan": {
46
+ "n": 327,
47
+ "public": false,
48
+ "wikalid-1.0.1": {
49
+ "acc_raw": 0.9388379204892966,
50
+ "acc_corrected": 0.9388379204892966,
51
+ "macro_f1_corrected": 0.9339768728004022
52
+ },
53
+ "glotlid": {
54
+ "acc_raw": 0.8990825688073395,
55
+ "acc_corrected": 0.8990825688073395,
56
+ "macro_f1_corrected": 0.9213529123029144
57
+ },
58
+ "openlid3": {
59
+ "acc_raw": 0.8807339449541285,
60
+ "acc_corrected": 0.8807339449541285,
61
+ "macro_f1_corrected": 0.8725169369237166
62
+ },
63
+ "lid176": {
64
+ "acc_raw": 0.7828746177370031,
65
+ "acc_corrected": 0.7828746177370031,
66
+ "macro_f1_corrected": 0.7374188657622348
67
+ }
68
+ },
69
+ "golddoc": {
70
+ "n": 148,
71
+ "public": false,
72
+ "wikalid-1.0.1": {
73
+ "acc_raw": 0.9864864864864865,
74
+ "acc_corrected": 0.9864864864864865,
75
+ "macro_f1_corrected": 0.99581589958159
76
+ },
77
+ "glotlid": {
78
+ "acc_raw": 0.9594594594594594,
79
+ "acc_corrected": 0.9594594594594594,
80
+ "macro_f1_corrected": 0.99581589958159
81
+ },
82
+ "openlid3": {
83
+ "acc_raw": 0.918918918918919,
84
+ "acc_corrected": 0.918918918918919,
85
+ "macro_f1_corrected": 0.9916666666666667
86
+ },
87
+ "lid176": {
88
+ "acc_raw": 0.8175675675675675,
89
+ "acc_corrected": 0.8175675675675675,
90
+ "macro_f1_corrected": 0.9008264462809917
91
+ }
92
+ },
93
+ "pld": {
94
+ "n": 1653,
95
+ "public": false,
96
+ "wikalid-1.0.1": {
97
+ "acc_raw": 0.9159104658197217,
98
+ "acc_corrected": 0.9171203871748337,
99
+ "macro_f1_corrected": 0.9256639461728243
100
+ },
101
+ "glotlid": {
102
+ "acc_raw": 0.8300060496067756,
103
+ "acc_corrected": 0.8342407743496673,
104
+ "macro_f1_corrected": 0.8384979959772872
105
+ },
106
+ "openlid3": {
107
+ "acc_raw": 0.41681790683605563,
108
+ "acc_corrected": 0.41742286751361163,
109
+ "macro_f1_corrected": 0.39562927937811565
110
+ },
111
+ "lid176": {
112
+ "acc_raw": 0.3369630973986691,
113
+ "acc_corrected": 0.3411978221415608,
114
+ "macro_f1_corrected": 0.28759118174467524
115
+ }
116
+ },
117
+ "wiki": {
118
+ "n": 2563,
119
+ "public": true,
120
+ "wikalid-1.0.1": {
121
+ "acc_raw": 0.8985563792430745,
122
+ "acc_corrected": 0.9250877877487319,
123
+ "macro_f1_corrected": 0.9284581306444153
124
+ },
125
+ "glotlid": {
126
+ "acc_raw": 0.8685134607881388,
127
+ "acc_corrected": 0.906749902458057,
128
+ "macro_f1_corrected": 0.9068774909539535
129
+ },
130
+ "openlid3": {
131
+ "acc_raw": 0.6761607491221225,
132
+ "acc_corrected": 0.7104955130706204,
133
+ "macro_f1_corrected": 0.6253482661073548
134
+ },
135
+ "lid176": {
136
+ "acc_raw": 0.6195864221615295,
137
+ "acc_corrected": 0.6640655481857198,
138
+ "macro_f1_corrected": 0.5537504126384613
139
+ }
140
+ },
141
+ "hard": {
142
+ "n": 6379,
143
+ "public": true,
144
+ "wikalid-1.0.1": {
145
+ "acc_raw": 0.8789778962219784,
146
+ "acc_corrected": 0.9007681454773475,
147
+ "macro_f1_corrected": 0.8800734956692698
148
+ },
149
+ "glotlid": {
150
+ "acc_raw": 0.7990280608245807,
151
+ "acc_corrected": 0.8291268223859539,
152
+ "macro_f1_corrected": 0.7603844713921172
153
+ },
154
+ "openlid3": {
155
+ "acc_raw": 0.619376077755134,
156
+ "acc_corrected": 0.6411663270105032,
157
+ "macro_f1_corrected": 0.5499190269122138
158
+ },
159
+ "lid176": {
160
+ "acc_raw": 0.5674870669383916,
161
+ "acc_corrected": 0.5989967079479542,
162
+ "macro_f1_corrected": 0.4522667230292048
163
+ }
164
+ },
165
+ "basaha": {
166
+ "n": 794,
167
+ "public": false,
168
+ "wikalid-1.0.1": {
169
+ "acc_raw": 0.739294710327456,
170
+ "acc_corrected": 0.739294710327456,
171
+ "macro_f1_corrected": 0.7949411406077508
172
+ },
173
+ "glotlid": {
174
+ "acc_raw": 0.5831234256926953,
175
+ "acc_corrected": 0.5831234256926953,
176
+ "macro_f1_corrected": 0.6413626117787863
177
+ },
178
+ "openlid3": {
179
+ "acc_raw": 0.02644836272040302,
180
+ "acc_corrected": 0.02644836272040302,
181
+ "macro_f1_corrected": 0.01195219123505976
182
+ },
183
+ "lid176": {
184
+ "acc_raw": 0.031486146095717885,
185
+ "acc_corrected": 0.031486146095717885,
186
+ "macro_f1_corrected": 0.027777777777777776
187
+ }
188
+ },
189
+ "tato": {
190
+ "n": 968,
191
+ "public": true,
192
+ "wikalid-1.0.1": {
193
+ "acc_raw": 0.9307851239669421,
194
+ "acc_corrected": 0.9276859504132231,
195
+ "macro_f1_corrected": 0.9397934525618807
196
+ },
197
+ "glotlid": {
198
+ "acc_raw": 0.9493801652892562,
199
+ "acc_corrected": 0.9545454545454546,
200
+ "macro_f1_corrected": 0.9674877115849037
201
+ },
202
+ "openlid3": {
203
+ "acc_raw": 0.7541322314049587,
204
+ "acc_corrected": 0.7582644628099173,
205
+ "macro_f1_corrected": 0.6814969661391874
206
+ },
207
+ "lid176": {
208
+ "acc_raw": 0.8006198347107438,
209
+ "acc_corrected": 0.8047520661157025,
210
+ "macro_f1_corrected": 0.8037869284335353
211
+ }
212
+ }
213
+ },
214
+ "dev": {
215
+ "crawl": {
216
+ "n": 326,
217
+ "public": false,
218
+ "wikalid-1.0.1": {
219
+ "acc_raw": 0.9631901840490797,
220
+ "acc_corrected": 0.9601226993865031,
221
+ "macro_f1_corrected": 0.9625247361477572
222
+ },
223
+ "glotlid": {
224
+ "acc_raw": 0.8926380368098159,
225
+ "acc_corrected": 0.8895705521472392,
226
+ "macro_f1_corrected": 0.9019766397124888
227
+ },
228
+ "openlid3": {
229
+ "acc_raw": 0.7699386503067485,
230
+ "acc_corrected": 0.7730061349693251,
231
+ "macro_f1_corrected": 0.5809186296991175
232
+ },
233
+ "lid176": {
234
+ "acc_raw": 0.7331288343558282,
235
+ "acc_corrected": 0.7361963190184049,
236
+ "macro_f1_corrected": 0.5275206062736826
237
+ }
238
+ },
239
+ "goldspan": {
240
+ "n": 266,
241
+ "public": false,
242
+ "wikalid-1.0.1": {
243
+ "acc_raw": 0.9624060150375939,
244
+ "acc_corrected": 0.9624060150375939,
245
+ "macro_f1_corrected": 0.9714543812104788
246
+ },
247
+ "glotlid": {
248
+ "acc_raw": 0.8984962406015038,
249
+ "acc_corrected": 0.8984962406015038,
250
+ "macro_f1_corrected": 0.9387562793957143
251
+ },
252
+ "openlid3": {
253
+ "acc_raw": 0.8458646616541353,
254
+ "acc_corrected": 0.8458646616541353,
255
+ "macro_f1_corrected": 0.6227779940314
256
+ },
257
+ "lid176": {
258
+ "acc_raw": 0.7706766917293233,
259
+ "acc_corrected": 0.7706766917293233,
260
+ "macro_f1_corrected": 0.5042274392781381
261
+ }
262
+ },
263
+ "golddoc": {
264
+ "n": 146,
265
+ "public": false,
266
+ "wikalid-1.0.1": {
267
+ "acc_raw": 1.0,
268
+ "acc_corrected": 1.0,
269
+ "macro_f1_corrected": 1.0
270
+ },
271
+ "glotlid": {
272
+ "acc_raw": 0.9383561643835616,
273
+ "acc_corrected": 0.9383561643835616,
274
+ "macro_f1_corrected": 0.9773755656108597
275
+ },
276
+ "openlid3": {
277
+ "acc_raw": 0.8698630136986302,
278
+ "acc_corrected": 0.8698630136986302,
279
+ "macro_f1_corrected": 0.9819819819819819
280
+ },
281
+ "lid176": {
282
+ "acc_raw": 0.773972602739726,
283
+ "acc_corrected": 0.773972602739726,
284
+ "macro_f1_corrected": 0.8715596330275229
285
+ }
286
+ },
287
+ "pld": {
288
+ "n": 1569,
289
+ "public": false,
290
+ "wikalid-1.0.1": {
291
+ "acc_raw": 0.9056724028043339,
292
+ "acc_corrected": 0.9056724028043339,
293
+ "macro_f1_corrected": 0.9169691092184136
294
+ },
295
+ "glotlid": {
296
+ "acc_raw": 0.8527724665391969,
297
+ "acc_corrected": 0.8553218610579987,
298
+ "macro_f1_corrected": 0.8679972593410478
299
+ },
300
+ "openlid3": {
301
+ "acc_raw": 0.4569789674952199,
302
+ "acc_corrected": 0.45825366475462076,
303
+ "macro_f1_corrected": 0.4136458247569358
304
+ },
305
+ "lid176": {
306
+ "acc_raw": 0.3753983428935628,
307
+ "acc_corrected": 0.37794773741236454,
308
+ "macro_f1_corrected": 0.29795429135587
309
+ }
310
+ },
311
+ "wiki": {
312
+ "n": 2474,
313
+ "public": true,
314
+ "wikalid-1.0.1": {
315
+ "acc_raw": 0.898949070331447,
316
+ "acc_corrected": 0.9248181083265966,
317
+ "macro_f1_corrected": 0.9275778391492562
318
+ },
319
+ "glotlid": {
320
+ "acc_raw": 0.872271624898949,
321
+ "acc_corrected": 0.9078415521422797,
322
+ "macro_f1_corrected": 0.9072448684695931
323
+ },
324
+ "openlid3": {
325
+ "acc_raw": 0.6750202101859337,
326
+ "acc_corrected": 0.7012934518997574,
327
+ "macro_f1_corrected": 0.613474975198989
328
+ },
329
+ "lid176": {
330
+ "acc_raw": 0.6261115602263541,
331
+ "acc_corrected": 0.6649151172190784,
332
+ "macro_f1_corrected": 0.559276551222542
333
+ }
334
+ },
335
+ "hard": {
336
+ "n": 6141,
337
+ "public": true,
338
+ "wikalid-1.0.1": {
339
+ "acc_raw": 0.882266731802638,
340
+ "acc_corrected": 0.9057156814851002,
341
+ "macro_f1_corrected": 0.8873738321187085
342
+ },
343
+ "glotlid": {
344
+ "acc_raw": 0.7951473701351571,
345
+ "acc_corrected": 0.82641263637844,
346
+ "macro_f1_corrected": 0.7611605654646197
347
+ },
348
+ "openlid3": {
349
+ "acc_raw": 0.6143950496661782,
350
+ "acc_corrected": 0.6363784399934864,
351
+ "macro_f1_corrected": 0.55206952431819
352
+ },
353
+ "lid176": {
354
+ "acc_raw": 0.5684741898713565,
355
+ "acc_corrected": 0.6021820550398957,
356
+ "macro_f1_corrected": 0.45472278889467443
357
+ }
358
+ },
359
+ "basaha": {
360
+ "n": 806,
361
+ "public": false,
362
+ "wikalid-1.0.1": {
363
+ "acc_raw": 0.7531017369727047,
364
+ "acc_corrected": 0.7531017369727047,
365
+ "macro_f1_corrected": 0.7817121529694779
366
+ },
367
+ "glotlid": {
368
+ "acc_raw": 0.5496277915632755,
369
+ "acc_corrected": 0.5496277915632755,
370
+ "macro_f1_corrected": 0.5981772873473876
371
+ },
372
+ "openlid3": {
373
+ "acc_raw": 0.016129032258064516,
374
+ "acc_corrected": 0.016129032258064516,
375
+ "macro_f1_corrected": 0.015765765765765764
376
+ },
377
+ "lid176": {
378
+ "acc_raw": 0.031017369727047148,
379
+ "acc_corrected": 0.031017369727047148,
380
+ "macro_f1_corrected": 0.03524229074889868
381
+ }
382
+ },
383
+ "tato": {
384
+ "n": 879,
385
+ "public": true,
386
+ "wikalid-1.0.1": {
387
+ "acc_raw": 0.9351535836177475,
388
+ "acc_corrected": 0.9271899886234357,
389
+ "macro_f1_corrected": 0.9424981517429267
390
+ },
391
+ "glotlid": {
392
+ "acc_raw": 0.9408418657565415,
393
+ "acc_corrected": 0.9431171786120591,
394
+ "macro_f1_corrected": 0.9537518269150536
395
+ },
396
+ "openlid3": {
397
+ "acc_raw": 0.7667804323094426,
398
+ "acc_corrected": 0.7736063708759955,
399
+ "macro_f1_corrected": 0.6932009448006917
400
+ },
401
+ "lid176": {
402
+ "acc_raw": 0.7986348122866894,
403
+ "acc_corrected": 0.8065984072810012,
404
+ "macro_f1_corrected": 0.79135387842101
405
+ }
406
+ }
407
+ },
408
+ "all": {
409
+ "crawl": {
410
+ "n": 661,
411
+ "public": false,
412
+ "wikalid-1.0.1": {
413
+ "acc_raw": 0.9561270801815431,
414
+ "acc_corrected": 0.9546142208774584,
415
+ "macro_f1_corrected": 0.9237178861239013
416
+ },
417
+ "glotlid": {
418
+ "acc_raw": 0.8835098335854765,
419
+ "acc_corrected": 0.8819969742813918,
420
+ "macro_f1_corrected": 0.8054684637926802
421
+ },
422
+ "openlid3": {
423
+ "acc_raw": 0.7776096822995462,
424
+ "acc_corrected": 0.7791225416036308,
425
+ "macro_f1_corrected": 0.5167738175035881
426
+ },
427
+ "lid176": {
428
+ "acc_raw": 0.7337367624810892,
429
+ "acc_corrected": 0.735249621785174,
430
+ "macro_f1_corrected": 0.41546818702156924
431
+ }
432
+ },
433
+ "goldspan": {
434
+ "n": 593,
435
+ "public": false,
436
+ "wikalid-1.0.1": {
437
+ "acc_raw": 0.9494097807757167,
438
+ "acc_corrected": 0.9494097807757167,
439
+ "macro_f1_corrected": 0.8577709147562972
440
+ },
441
+ "glotlid": {
442
+ "acc_raw": 0.8988195615514334,
443
+ "acc_corrected": 0.8988195615514334,
444
+ "macro_f1_corrected": 0.6611717645352578
445
+ },
446
+ "openlid3": {
447
+ "acc_raw": 0.8650927487352446,
448
+ "acc_corrected": 0.8650927487352446,
449
+ "macro_f1_corrected": 0.5199837315153472
450
+ },
451
+ "lid176": {
452
+ "acc_raw": 0.7774030354131535,
453
+ "acc_corrected": 0.7774030354131535,
454
+ "macro_f1_corrected": 0.3729074002101596
455
+ }
456
+ },
457
+ "golddoc": {
458
+ "n": 294,
459
+ "public": false,
460
+ "wikalid-1.0.1": {
461
+ "acc_raw": 0.9931972789115646,
462
+ "acc_corrected": 0.9931972789115646,
463
+ "macro_f1_corrected": 0.9810489661935692
464
+ },
465
+ "glotlid": {
466
+ "acc_raw": 0.9489795918367347,
467
+ "acc_corrected": 0.9489795918367347,
468
+ "macro_f1_corrected": 0.9262168849655593
469
+ },
470
+ "openlid3": {
471
+ "acc_raw": 0.8945578231292517,
472
+ "acc_corrected": 0.8945578231292517,
473
+ "macro_f1_corrected": 0.5512265512265512
474
+ },
475
+ "lid176": {
476
+ "acc_raw": 0.7959183673469388,
477
+ "acc_corrected": 0.7959183673469388,
478
+ "macro_f1_corrected": 0.5659224441833137
479
+ }
480
+ },
481
+ "pld": {
482
+ "n": 3222,
483
+ "public": false,
484
+ "wikalid-1.0.1": {
485
+ "acc_raw": 0.9109248913718188,
486
+ "acc_corrected": 0.9115456238361266,
487
+ "macro_f1_corrected": 0.9226356531452579
488
+ },
489
+ "glotlid": {
490
+ "acc_raw": 0.8410924891371818,
491
+ "acc_corrected": 0.8445065176908753,
492
+ "macro_f1_corrected": 0.8531363677352409
493
+ },
494
+ "openlid3": {
495
+ "acc_raw": 0.43637492240844195,
496
+ "acc_corrected": 0.4373060211049038,
497
+ "macro_f1_corrected": 0.40474253590796233
498
+ },
499
+ "lid176": {
500
+ "acc_raw": 0.35567970204841715,
501
+ "acc_corrected": 0.3590937306021105,
502
+ "macro_f1_corrected": 0.2924533326385652
503
+ }
504
+ },
505
+ "wiki": {
506
+ "n": 5037,
507
+ "public": true,
508
+ "wikalid-1.0.1": {
509
+ "acc_raw": 0.8987492555092317,
510
+ "acc_corrected": 0.9249553305539011,
511
+ "macro_f1_corrected": 0.9281129552088062
512
+ },
513
+ "glotlid": {
514
+ "acc_raw": 0.8703593408775064,
515
+ "acc_corrected": 0.9072860829859043,
516
+ "macro_f1_corrected": 0.9072521485914344
517
+ },
518
+ "openlid3": {
519
+ "acc_raw": 0.6756005558864403,
520
+ "acc_corrected": 0.7059757792336708,
521
+ "macro_f1_corrected": 0.6196306323515995
522
+ },
523
+ "lid176": {
524
+ "acc_raw": 0.6227913440540004,
525
+ "acc_corrected": 0.6644828270796109,
526
+ "macro_f1_corrected": 0.5564368479383145
527
+ }
528
+ },
529
+ "hard": {
530
+ "n": 12520,
531
+ "public": true,
532
+ "wikalid-1.0.1": {
533
+ "acc_raw": 0.880591054313099,
534
+ "acc_corrected": 0.9031948881789137,
535
+ "macro_f1_corrected": 0.8837680817481542
536
+ },
537
+ "glotlid": {
538
+ "acc_raw": 0.7971246006389776,
539
+ "acc_corrected": 0.8277955271565495,
540
+ "macro_f1_corrected": 0.7608953515211622
541
+ },
542
+ "openlid3": {
543
+ "acc_raw": 0.6169329073482428,
544
+ "acc_corrected": 0.638817891373802,
545
+ "macro_f1_corrected": 0.5510186484085835
546
+ },
547
+ "lid176": {
548
+ "acc_raw": 0.5679712460063898,
549
+ "acc_corrected": 0.6005591054313099,
550
+ "macro_f1_corrected": 0.45366292115150086
551
+ }
552
+ },
553
+ "basaha": {
554
+ "n": 1600,
555
+ "public": false,
556
+ "wikalid-1.0.1": {
557
+ "acc_raw": 0.74625,
558
+ "acc_corrected": 0.74625,
559
+ "macro_f1_corrected": 0.7884680885328015
560
+ },
561
+ "glotlid": {
562
+ "acc_raw": 0.56625,
563
+ "acc_corrected": 0.56625,
564
+ "macro_f1_corrected": 0.6200922101491063
565
+ },
566
+ "openlid3": {
567
+ "acc_raw": 0.02125,
568
+ "acc_corrected": 0.02125,
569
+ "macro_f1_corrected": 0.013742071881606765
570
+ },
571
+ "lid176": {
572
+ "acc_raw": 0.03125,
573
+ "acc_corrected": 0.03125,
574
+ "macro_f1_corrected": 0.031315240083507306
575
+ }
576
+ },
577
+ "tato": {
578
+ "n": 1847,
579
+ "public": true,
580
+ "wikalid-1.0.1": {
581
+ "acc_raw": 0.9328641039523552,
582
+ "acc_corrected": 0.9274499187872225,
583
+ "macro_f1_corrected": 0.9407261199876595
584
+ },
585
+ "glotlid": {
586
+ "acc_raw": 0.9453167298321603,
587
+ "acc_corrected": 0.9491066594477531,
588
+ "macro_f1_corrected": 0.9610631108461403
589
+ },
590
+ "openlid3": {
591
+ "acc_raw": 0.7601515971846237,
592
+ "acc_corrected": 0.7655657823497564,
593
+ "macro_f1_corrected": 0.6865861879529651
594
+ },
595
+ "lid176": {
596
+ "acc_raw": 0.7996751488900921,
597
+ "acc_corrected": 0.805630752571738,
598
+ "macro_f1_corrected": 0.7984173253607114
599
+ }
600
+ }
601
+ }
602
+ },
603
+ "trust": {
604
+ "t_sib": {
605
+ "n": 9036,
606
+ "wikalid-1.0.1": {
607
+ "acc": 0.9892651615759186,
608
+ "ci95": [
609
+ 0.9869227230925369,
610
+ 0.9911917606833734
611
+ ],
612
+ "by": {
613
+ "ceb": {
614
+ "acc": 0.9970119521912351,
615
+ "n": 1004,
616
+ "ci95": [
617
+ 0.9912516700049157,
618
+ 0.9989833034274174
619
+ ]
620
+ },
621
+ "eng": {
622
+ "acc": 1.0,
623
+ "n": 1004,
624
+ "ci95": [
625
+ 0.9961882899058743,
626
+ 1.0000000000000002
627
+ ]
628
+ },
629
+ "ilo": {
630
+ "acc": 1.0,
631
+ "n": 1004,
632
+ "ci95": [
633
+ 0.9961882899058743,
634
+ 1.0000000000000002
635
+ ]
636
+ },
637
+ "ind": {
638
+ "acc": 0.9591633466135459,
639
+ "n": 1004,
640
+ "ci95": [
641
+ 0.9450695618604726,
642
+ 0.9697567362403402
643
+ ]
644
+ },
645
+ "msa": {
646
+ "acc": 0.953187250996016,
647
+ "n": 1004,
648
+ "ci95": [
649
+ 0.9383043206814081,
650
+ 0.9646153444723224
651
+ ]
652
+ },
653
+ "pag": {
654
+ "acc": 0.9970119521912351,
655
+ "n": 1004,
656
+ "ci95": [
657
+ 0.9912516700049157,
658
+ 0.9989833034274174
659
+ ]
660
+ },
661
+ "sib200": {
662
+ "acc": 0.9892651615759186,
663
+ "n": 9036,
664
+ "ci95": [
665
+ 0.9869227230925369,
666
+ 0.9911917606833734
667
+ ]
668
+ },
669
+ "spa": {
670
+ "acc": 0.9970119521912351,
671
+ "n": 1004,
672
+ "ci95": [
673
+ 0.9912516700049157,
674
+ 0.9989833034274174
675
+ ]
676
+ },
677
+ "tgl": {
678
+ "acc": 1.0,
679
+ "n": 1004,
680
+ "ci95": [
681
+ 0.9961882899058743,
682
+ 1.0000000000000002
683
+ ]
684
+ },
685
+ "war": {
686
+ "acc": 1.0,
687
+ "n": 1004,
688
+ "ci95": [
689
+ 0.9961882899058743,
690
+ 1.0000000000000002
691
+ ]
692
+ }
693
+ }
694
+ },
695
+ "glotlid": {
696
+ "acc": 0.9863877822045153,
697
+ "ci95": [
698
+ 0.9837834463036013,
699
+ 0.9885787242901842
700
+ ],
701
+ "by": {
702
+ "ceb": {
703
+ "acc": 0.9910358565737052,
704
+ "n": 1004,
705
+ "ci95": [
706
+ 0.9830514267968079,
707
+ 0.9952769136884431
708
+ ]
709
+ },
710
+ "eng": {
711
+ "acc": 1.0,
712
+ "n": 1004,
713
+ "ci95": [
714
+ 0.9961882899058743,
715
+ 1.0000000000000002
716
+ ]
717
+ },
718
+ "ilo": {
719
+ "acc": 0.999003984063745,
720
+ "n": 1004,
721
+ "ci95": [
722
+ 0.9943796894699304,
723
+ 0.9998241616114303
724
+ ]
725
+ },
726
+ "ind": {
727
+ "acc": 0.9950199203187251,
728
+ "n": 1004,
729
+ "ci95": [
730
+ 0.9883950765339303,
731
+ 0.9978710192493755
732
+ ]
733
+ },
734
+ "msa": {
735
+ "acc": 0.8944223107569721,
736
+ "n": 1004,
737
+ "ci95": [
738
+ 0.873887228535692,
739
+ 0.9119505459717305
740
+ ]
741
+ },
742
+ "pag": {
743
+ "acc": 1.0,
744
+ "n": 1004,
745
+ "ci95": [
746
+ 0.9961882899058743,
747
+ 1.0000000000000002
748
+ ]
749
+ },
750
+ "sib200": {
751
+ "acc": 0.9863877822045153,
752
+ "n": 9036,
753
+ "ci95": [
754
+ 0.9837834463036013,
755
+ 0.9885787242901842
756
+ ]
757
+ },
758
+ "spa": {
759
+ "acc": 0.99800796812749,
760
+ "n": 1004,
761
+ "ci95": [
762
+ 0.9927658578932684,
763
+ 0.9994535543635784
764
+ ]
765
+ },
766
+ "tgl": {
767
+ "acc": 1.0,
768
+ "n": 1004,
769
+ "ci95": [
770
+ 0.9961882899058743,
771
+ 1.0000000000000002
772
+ ]
773
+ },
774
+ "war": {
775
+ "acc": 1.0,
776
+ "n": 1004,
777
+ "ci95": [
778
+ 0.9961882899058743,
779
+ 1.0000000000000002
780
+ ]
781
+ }
782
+ }
783
+ },
784
+ "openlid3": {
785
+ "acc": 0.9846170872067287,
786
+ "ci95": [
787
+ 0.9818657500423222,
788
+ 0.9869565355163097
789
+ ],
790
+ "by": {
791
+ "ceb": {
792
+ "acc": 1.0,
793
+ "n": 1004,
794
+ "ci95": [
795
+ 0.9961882899058743,
796
+ 1.0000000000000002
797
+ ]
798
+ },
799
+ "eng": {
800
+ "acc": 0.99800796812749,
801
+ "n": 1004,
802
+ "ci95": [
803
+ 0.9927658578932684,
804
+ 0.9994535543635784
805
+ ]
806
+ },
807
+ "ilo": {
808
+ "acc": 1.0,
809
+ "n": 1004,
810
+ "ci95": [
811
+ 0.9961882899058743,
812
+ 1.0000000000000002
813
+ ]
814
+ },
815
+ "ind": {
816
+ "acc": 0.9183266932270916,
817
+ "n": 1004,
818
+ "ci95": [
819
+ 0.8997488848216905,
820
+ 0.9337154214740607
821
+ ]
822
+ },
823
+ "msa": {
824
+ "acc": 0.9631474103585658,
825
+ "n": 1004,
826
+ "ci95": [
827
+ 0.9496171963573046,
828
+ 0.9731468570415629
829
+ ]
830
+ },
831
+ "pag": {
832
+ "acc": 0.9960159362549801,
833
+ "n": 1004,
834
+ "ci95": [
835
+ 0.9898008918922921,
836
+ 0.9984496427155273
837
+ ]
838
+ },
839
+ "sib200": {
840
+ "acc": 0.9846170872067287,
841
+ "n": 9036,
842
+ "ci95": [
843
+ 0.9818657500423222,
844
+ 0.9869565355163097
845
+ ]
846
+ },
847
+ "spa": {
848
+ "acc": 0.9860557768924303,
849
+ "n": 1004,
850
+ "ci95": [
851
+ 0.9767302763202146,
852
+ 0.9916758700424679
853
+ ]
854
+ },
855
+ "tgl": {
856
+ "acc": 1.0,
857
+ "n": 1004,
858
+ "ci95": [
859
+ 0.9961882899058743,
860
+ 1.0000000000000002
861
+ ]
862
+ },
863
+ "war": {
864
+ "acc": 1.0,
865
+ "n": 1004,
866
+ "ci95": [
867
+ 0.9961882899058743,
868
+ 1.0000000000000002
869
+ ]
870
+ }
871
+ }
872
+ },
873
+ "lid176": {
874
+ "acc": 0.7737937140327579,
875
+ "ci95": [
876
+ 0.7650519445652785,
877
+ 0.7823027789953644
878
+ ],
879
+ "by": {
880
+ "ceb": {
881
+ "acc": 0.9223107569721115,
882
+ "n": 1004,
883
+ "ci95": [
884
+ 0.9040963910139844,
885
+ 0.9373056705798216
886
+ ]
887
+ },
888
+ "eng": {
889
+ "acc": 1.0,
890
+ "n": 1004,
891
+ "ci95": [
892
+ 0.9961882899058743,
893
+ 1.0000000000000002
894
+ ]
895
+ },
896
+ "ilo": {
897
+ "acc": 0.9850597609561753,
898
+ "n": 1004,
899
+ "ci95": [
900
+ 0.975496224606418,
901
+ 0.9909254829317506
902
+ ]
903
+ },
904
+ "ind": {
905
+ "acc": 0.9691235059760956,
906
+ "n": 1004,
907
+ "ci95": [
908
+ 0.9565068693060876,
909
+ 0.9781638170398621
910
+ ]
911
+ },
912
+ "msa": {
913
+ "acc": 0.449203187250996,
914
+ "n": 1004,
915
+ "ci95": [
916
+ 0.4186863948679612,
917
+ 0.48010722508184045
918
+ ]
919
+ },
920
+ "pag": {
921
+ "acc": 0.0,
922
+ "n": 1004,
923
+ "ci95": [
924
+ 0.0,
925
+ 0.0038117100941259022
926
+ ]
927
+ },
928
+ "sib200": {
929
+ "acc": 0.7737937140327579,
930
+ "n": 9036,
931
+ "ci95": [
932
+ 0.7650519445652785,
933
+ 0.7823027789953644
934
+ ]
935
+ },
936
+ "spa": {
937
+ "acc": 1.0,
938
+ "n": 1004,
939
+ "ci95": [
940
+ 0.9961882899058743,
941
+ 1.0000000000000002
942
+ ]
943
+ },
944
+ "tgl": {
945
+ "acc": 0.99800796812749,
946
+ "n": 1004,
947
+ "ci95": [
948
+ 0.9927658578932684,
949
+ 0.9994535543635784
950
+ ]
951
+ },
952
+ "war": {
953
+ "acc": 0.6404382470119522,
954
+ "n": 1004,
955
+ "ci95": [
956
+ 0.6102712632213478,
957
+ 0.6695346110350828
958
+ ]
959
+ }
960
+ }
961
+ }
962
+ },
963
+ "t_frag": {
964
+ "n": 4320,
965
+ "wikalid-1.0.1": {
966
+ "acc": 0.8319444444444445,
967
+ "ci95": [
968
+ 0.8205002431714228,
969
+ 0.8427988009008671
970
+ ],
971
+ "by": {
972
+ "ceb": {
973
+ "acc": 0.8145833333333333,
974
+ "n": 480,
975
+ "ci95": [
976
+ 0.7773661170146048,
977
+ 0.8468050993504205
978
+ ]
979
+ },
980
+ "eng": {
981
+ "acc": 0.9770833333333333,
982
+ "n": 480,
983
+ "ci95": [
984
+ 0.9594342159747407,
985
+ 0.9871565691086419
986
+ ]
987
+ },
988
+ "ilo": {
989
+ "acc": 0.9416666666666667,
990
+ "n": 480,
991
+ "ci95": [
992
+ 0.9169836611947353,
993
+ 0.9593361922449029
994
+ ]
995
+ },
996
+ "ind": {
997
+ "acc": 0.8208333333333333,
998
+ "n": 480,
999
+ "ci95": [
1000
+ 0.7840199414043012,
1001
+ 0.8525520276037377
1002
+ ]
1003
+ },
1004
+ "len2": {
1005
+ "acc": 0.7037037037037037,
1006
+ "n": 1080,
1007
+ "ci95": [
1008
+ 0.6757869878102885,
1009
+ 0.7301763928165486
1010
+ ]
1011
+ },
1012
+ "len3": {
1013
+ "acc": 0.8148148148148148,
1014
+ "n": 1080,
1015
+ "ci95": [
1016
+ 0.7905457940854126,
1017
+ 0.8368521577924264
1018
+ ]
1019
+ },
1020
+ "len4": {
1021
+ "acc": 0.8777777777777778,
1022
+ "n": 1080,
1023
+ "ci95": [
1024
+ 0.8568925867327652,
1025
+ 0.8959849555206416
1026
+ ]
1027
+ },
1028
+ "len5": {
1029
+ "acc": 0.9314814814814815,
1030
+ "n": 1080,
1031
+ "ci95": [
1032
+ 0.9148340030838412,
1033
+ 0.9450702486075498
1034
+ ]
1035
+ },
1036
+ "msa": {
1037
+ "acc": 0.6395833333333333,
1038
+ "n": 480,
1039
+ "ci95": [
1040
+ 0.5956792716294282,
1041
+ 0.681270870731212
1042
+ ]
1043
+ },
1044
+ "pag": {
1045
+ "acc": 0.8729166666666667,
1046
+ "n": 480,
1047
+ "ci95": [
1048
+ 0.8401304181213104,
1049
+ 0.8997811562451764
1050
+ ]
1051
+ },
1052
+ "spa": {
1053
+ "acc": 0.8708333333333333,
1054
+ "n": 480,
1055
+ "ci95": [
1056
+ 0.8378597177438099,
1057
+ 0.8979182724083392
1058
+ ]
1059
+ },
1060
+ "tgl": {
1061
+ "acc": 0.84375,
1062
+ "n": 480,
1063
+ "ci95": [
1064
+ 0.8085522736473225,
1065
+ 0.8734891217184335
1066
+ ]
1067
+ },
1068
+ "war": {
1069
+ "acc": 0.70625,
1070
+ "n": 480,
1071
+ "ci95": [
1072
+ 0.663993732629122,
1073
+ 0.7452311045903318
1074
+ ]
1075
+ }
1076
+ }
1077
+ },
1078
+ "glotlid": {
1079
+ "acc": 0.7712962962962963,
1080
+ "ci95": [
1081
+ 0.7585339732255689,
1082
+ 0.7835765425713093
1083
+ ],
1084
+ "by": {
1085
+ "ceb": {
1086
+ "acc": 0.7604166666666666,
1087
+ "n": 480,
1088
+ "ci95": [
1089
+ 0.7202600260476222,
1090
+ 0.7964380007446172
1091
+ ]
1092
+ },
1093
+ "eng": {
1094
+ "acc": 0.9083333333333333,
1095
+ "n": 480,
1096
+ "ci95": [
1097
+ 0.8791758375690445,
1098
+ 0.931006668441187
1099
+ ]
1100
+ },
1101
+ "ilo": {
1102
+ "acc": 0.8916666666666667,
1103
+ "n": 480,
1104
+ "ci95": [
1105
+ 0.8606887969411983,
1106
+ 0.91642503535433
1107
+ ]
1108
+ },
1109
+ "ind": {
1110
+ "acc": 0.8666666666666667,
1111
+ "n": 480,
1112
+ "ci95": [
1113
+ 0.8333257856869319,
1114
+ 0.8941850360365412
1115
+ ]
1116
+ },
1117
+ "len2": {
1118
+ "acc": 0.6240740740740741,
1119
+ "n": 1080,
1120
+ "ci95": [
1121
+ 0.5947944606747045,
1122
+ 0.6524741438889144
1123
+ ]
1124
+ },
1125
+ "len3": {
1126
+ "acc": 0.7388888888888889,
1127
+ "n": 1080,
1128
+ "ci95": [
1129
+ 0.7118782358681339,
1130
+ 0.7642060923215204
1131
+ ]
1132
+ },
1133
+ "len4": {
1134
+ "acc": 0.8351851851851851,
1135
+ "n": 1080,
1136
+ "ci95": [
1137
+ 0.8118768985265994,
1138
+ 0.8561173914139233
1139
+ ]
1140
+ },
1141
+ "len5": {
1142
+ "acc": 0.8870370370370371,
1143
+ "n": 1080,
1144
+ "ci95": [
1145
+ 0.8667696567362586,
1146
+ 0.9045607664547317
1147
+ ]
1148
+ },
1149
+ "msa": {
1150
+ "acc": 0.3416666666666667,
1151
+ "n": 480,
1152
+ "ci95": [
1153
+ 0.3006451830920638,
1154
+ 0.3852024166182546
1155
+ ]
1156
+ },
1157
+ "pag": {
1158
+ "acc": 0.8229166666666666,
1159
+ "n": 480,
1160
+ "ci95": [
1161
+ 0.7862412266386819,
1162
+ 0.854464326583695
1163
+ ]
1164
+ },
1165
+ "spa": {
1166
+ "acc": 0.85,
1167
+ "n": 480,
1168
+ "ci95": [
1169
+ 0.8152829232916574,
1170
+ 0.8791592247171123
1171
+ ]
1172
+ },
1173
+ "tgl": {
1174
+ "acc": 0.8666666666666667,
1175
+ "n": 480,
1176
+ "ci95": [
1177
+ 0.8333257856869319,
1178
+ 0.8941850360365412
1179
+ ]
1180
+ },
1181
+ "war": {
1182
+ "acc": 0.6333333333333333,
1183
+ "n": 480,
1184
+ "ci95": [
1185
+ 0.5893222088010526,
1186
+ 0.675227180916574
1187
+ ]
1188
+ }
1189
+ }
1190
+ },
1191
+ "openlid3": {
1192
+ "acc": 0.7733796296296296,
1193
+ "ci95": [
1194
+ 0.7606557433895734,
1195
+ 0.7856177371190557
1196
+ ],
1197
+ "by": {
1198
+ "ceb": {
1199
+ "acc": 0.8416666666666667,
1200
+ "n": 480,
1201
+ "ci95": [
1202
+ 0.8063127326027417,
1203
+ 0.8715950785486763
1204
+ ]
1205
+ },
1206
+ "eng": {
1207
+ "acc": 0.7166666666666667,
1208
+ "n": 480,
1209
+ "ci95": [
1210
+ 0.6747571073684372,
1211
+ 0.755135650922706
1212
+ ]
1213
+ },
1214
+ "ilo": {
1215
+ "acc": 0.9333333333333333,
1216
+ "n": 480,
1217
+ "ci95": [
1218
+ 0.9074012300031347,
1219
+ 0.9523842865791519
1220
+ ]
1221
+ },
1222
+ "ind": {
1223
+ "acc": 0.8104166666666667,
1224
+ "n": 480,
1225
+ "ci95": [
1226
+ 0.7729383775947093,
1227
+ 0.84296567034164
1228
+ ]
1229
+ },
1230
+ "len2": {
1231
+ "acc": 0.6212962962962963,
1232
+ "n": 1080,
1233
+ "ci95": [
1234
+ 0.5919848236309152,
1235
+ 0.6497479166514287
1236
+ ]
1237
+ },
1238
+ "len3": {
1239
+ "acc": 0.75,
1240
+ "n": 1080,
1241
+ "ci95": [
1242
+ 0.7233192092286217,
1243
+ 0.7749085760861328
1244
+ ]
1245
+ },
1246
+ "len4": {
1247
+ "acc": 0.8342592592592593,
1248
+ "n": 1080,
1249
+ "ci95": [
1250
+ 0.8109048245345469,
1251
+ 0.8552441773122175
1252
+ ]
1253
+ },
1254
+ "len5": {
1255
+ "acc": 0.887962962962963,
1256
+ "n": 1080,
1257
+ "ci95": [
1258
+ 0.8677594847017363,
1259
+ 0.9054162265830124
1260
+ ]
1261
+ },
1262
+ "msa": {
1263
+ "acc": 0.55,
1264
+ "n": 480,
1265
+ "ci95": [
1266
+ 0.5052718108484551,
1267
+ 0.5939342102956551
1268
+ ]
1269
+ },
1270
+ "pag": {
1271
+ "acc": 0.7895833333333333,
1272
+ "n": 480,
1273
+ "ci95": [
1274
+ 0.7508915928659468,
1275
+ 0.8236766129270234
1276
+ ]
1277
+ },
1278
+ "spa": {
1279
+ "acc": 0.6916666666666667,
1280
+ "n": 480,
1281
+ "ci95": [
1282
+ 0.6489673795407492,
1283
+ 0.7313223681783391
1284
+ ]
1285
+ },
1286
+ "tgl": {
1287
+ "acc": 0.8875,
1288
+ "n": 480,
1289
+ "ci95": [
1290
+ 0.8561001232465634,
1291
+ 0.9127465406202887
1292
+ ]
1293
+ },
1294
+ "war": {
1295
+ "acc": 0.7395833333333334,
1296
+ "n": 480,
1297
+ "ci95": [
1298
+ 0.6985298417841903,
1299
+ 0.77683234286467
1300
+ ]
1301
+ }
1302
+ }
1303
+ },
1304
+ "lid176": {
1305
+ "acc": 0.5900462962962963,
1306
+ "ci95": [
1307
+ 0.5753061534658956,
1308
+ 0.6046264324086656
1309
+ ],
1310
+ "by": {
1311
+ "ceb": {
1312
+ "acc": 0.68125,
1313
+ "n": 480,
1314
+ "ci95": [
1315
+ 0.6382635469798807,
1316
+ 0.721358279667518
1317
+ ]
1318
+ },
1319
+ "eng": {
1320
+ "acc": 0.9916666666666667,
1321
+ "n": 480,
1322
+ "ci95": [
1323
+ 0.9787711280476984,
1324
+ 0.9967547465360497
1325
+ ]
1326
+ },
1327
+ "ilo": {
1328
+ "acc": 0.5458333333333333,
1329
+ "n": 480,
1330
+ "ci95": [
1331
+ 0.5011027665973042,
1332
+ 0.5898360861181298
1333
+ ]
1334
+ },
1335
+ "ind": {
1336
+ "acc": 0.75,
1337
+ "n": 480,
1338
+ "ci95": [
1339
+ 0.709380219226088,
1340
+ 0.7866498864944619
1341
+ ]
1342
+ },
1343
+ "len2": {
1344
+ "acc": 0.5138888888888888,
1345
+ "n": 1080,
1346
+ "ci95": [
1347
+ 0.4840835513937429,
1348
+ 0.5435957700126323
1349
+ ]
1350
+ },
1351
+ "len3": {
1352
+ "acc": 0.5472222222222223,
1353
+ "n": 1080,
1354
+ "ci95": [
1355
+ 0.517419875518785,
1356
+ 0.576689817262891
1357
+ ]
1358
+ },
1359
+ "len4": {
1360
+ "acc": 0.6277777777777778,
1361
+ "n": 1080,
1362
+ "ci95": [
1363
+ 0.5985422101910842,
1364
+ 0.6561075467475681
1365
+ ]
1366
+ },
1367
+ "len5": {
1368
+ "acc": 0.6712962962962963,
1369
+ "n": 1080,
1370
+ "ci95": [
1371
+ 0.6427164149552664,
1372
+ 0.6986618823900284
1373
+ ]
1374
+ },
1375
+ "msa": {
1376
+ "acc": 0.3541666666666667,
1377
+ "n": 480,
1378
+ "ci95": [
1379
+ 0.3126932319826662,
1380
+ 0.39795587301367974
1381
+ ]
1382
+ },
1383
+ "pag": {
1384
+ "acc": 0.0,
1385
+ "n": 480,
1386
+ "ci95": [
1387
+ -8.673617379884035e-19,
1388
+ 0.00793978855890027
1389
+ ]
1390
+ },
1391
+ "spa": {
1392
+ "acc": 0.9333333333333333,
1393
+ "n": 480,
1394
+ "ci95": [
1395
+ 0.9074012300031347,
1396
+ 0.9523842865791519
1397
+ ]
1398
+ },
1399
+ "tgl": {
1400
+ "acc": 0.7291666666666666,
1401
+ "n": 480,
1402
+ "ci95": [
1403
+ 0.687707743498603,
1404
+ 0.7669865200785677
1405
+ ]
1406
+ },
1407
+ "war": {
1408
+ "acc": 0.325,
1409
+ "n": 480,
1410
+ "ci95": [
1411
+ 0.28463156886351215,
1412
+ 0.368147357132103
1413
+ ]
1414
+ }
1415
+ }
1416
+ }
1417
+ },
1418
+ "t_caps": {
1419
+ "n": 5388,
1420
+ "wikalid-1.0.1": {
1421
+ "acc": 0.9988864142538976,
1422
+ "ci95": [
1423
+ 0.9975723876556302,
1424
+ 0.9994895438373128
1425
+ ],
1426
+ "by": {
1427
+ "ceb": {
1428
+ "acc": 0.9948914431673053,
1429
+ "n": 783,
1430
+ "ci95": [
1431
+ 0.986938790959379,
1432
+ 0.9980116742930938
1433
+ ]
1434
+ },
1435
+ "eng": {
1436
+ "acc": 1.0,
1437
+ "n": 765,
1438
+ "ci95": [
1439
+ 0.9950033921161395,
1440
+ 1.0000000000000002
1441
+ ]
1442
+ },
1443
+ "ilo": {
1444
+ "acc": 1.0,
1445
+ "n": 699,
1446
+ "ci95": [
1447
+ 0.9945341880731021,
1448
+ 1.0
1449
+ ]
1450
+ },
1451
+ "lower": {
1452
+ "acc": 0.9983296213808464,
1453
+ "n": 1796,
1454
+ "ci95": [
1455
+ 0.9951001914542256,
1456
+ 0.9994317718029856
1457
+ ]
1458
+ },
1459
+ "pag": {
1460
+ "acc": 0.9987080103359173,
1461
+ "n": 774,
1462
+ "ci95": [
1463
+ 0.9927180861293878,
1464
+ 0.9997719015493376
1465
+ ]
1466
+ },
1467
+ "spa": {
1468
+ "acc": 1.0,
1469
+ "n": 786,
1470
+ "ci95": [
1471
+ 0.9951362399752052,
1472
+ 1.0
1473
+ ]
1474
+ },
1475
+ "tgl": {
1476
+ "acc": 0.99867197875166,
1477
+ "n": 753,
1478
+ "ci95": [
1479
+ 0.9925160690054392,
1480
+ 0.9997655391936874
1481
+ ]
1482
+ },
1483
+ "title": {
1484
+ "acc": 0.9988864142538976,
1485
+ "n": 1796,
1486
+ "ci95": [
1487
+ 0.9959486042757345,
1488
+ 0.9996945678789707
1489
+ ]
1490
+ },
1491
+ "upper": {
1492
+ "acc": 0.9994432071269488,
1493
+ "n": 1796,
1494
+ "ci95": [
1495
+ 0.9968526729003299,
1496
+ 0.9999017081518694
1497
+ ]
1498
+ },
1499
+ "war": {
1500
+ "acc": 1.0,
1501
+ "n": 828,
1502
+ "ci95": [
1503
+ 0.9953818130759511,
1504
+ 1.0
1505
+ ]
1506
+ }
1507
+ }
1508
+ },
1509
+ "glotlid": {
1510
+ "acc": 0.9751299183370453,
1511
+ "ci95": [
1512
+ 0.9706208513243705,
1513
+ 0.9789619405736706
1514
+ ],
1515
+ "by": {
1516
+ "ceb": {
1517
+ "acc": 0.9591315453384419,
1518
+ "n": 783,
1519
+ "ci95": [
1520
+ 0.9428755574692792,
1521
+ 0.970904293519789
1522
+ ]
1523
+ },
1524
+ "eng": {
1525
+ "acc": 0.930718954248366,
1526
+ "n": 765,
1527
+ "ci95": [
1528
+ 0.9104886883171216,
1529
+ 0.9466449527345594
1530
+ ]
1531
+ },
1532
+ "ilo": {
1533
+ "acc": 0.9985693848354793,
1534
+ "n": 699,
1535
+ "ci95": [
1536
+ 0.9919411735524428,
1537
+ 0.999747423138476
1538
+ ]
1539
+ },
1540
+ "lower": {
1541
+ "acc": 0.9983296213808464,
1542
+ "n": 1796,
1543
+ "ci95": [
1544
+ 0.9951001914542256,
1545
+ 0.9994317718029856
1546
+ ]
1547
+ },
1548
+ "pag": {
1549
+ "acc": 0.9935400516795866,
1550
+ "n": 774,
1551
+ "ci95": [
1552
+ 0.9849674441294312,
1553
+ 0.9972376732106547
1554
+ ]
1555
+ },
1556
+ "spa": {
1557
+ "acc": 0.9491094147582697,
1558
+ "n": 786,
1559
+ "ci95": [
1560
+ 0.9314429805910065,
1561
+ 0.9624071280890125
1562
+ ]
1563
+ },
1564
+ "tgl": {
1565
+ "acc": 0.9960159362549801,
1566
+ "n": 753,
1567
+ "ci95": [
1568
+ 0.9883523084517515,
1569
+ 0.9986441779993106
1570
+ ]
1571
+ },
1572
+ "title": {
1573
+ "acc": 0.9899777282850779,
1574
+ "n": 1796,
1575
+ "ci95": [
1576
+ 0.9842127020060506,
1577
+ 0.9936511277887491
1578
+ ]
1579
+ },
1580
+ "upper": {
1581
+ "acc": 0.9370824053452116,
1582
+ "n": 1796,
1583
+ "ci95": [
1584
+ 0.9248928077217875,
1585
+ 0.9474061768110735
1586
+ ]
1587
+ },
1588
+ "war": {
1589
+ "acc": 1.0,
1590
+ "n": 828,
1591
+ "ci95": [
1592
+ 0.9953818130759511,
1593
+ 1.0
1594
+ ]
1595
+ }
1596
+ }
1597
+ },
1598
+ "openlid3": {
1599
+ "acc": 0.5749814402375649,
1600
+ "ci95": [
1601
+ 0.5617326363392661,
1602
+ 0.5881233980071434
1603
+ ],
1604
+ "by": {
1605
+ "ceb": {
1606
+ "acc": 0.6462324393358876,
1607
+ "n": 783,
1608
+ "ci95": [
1609
+ 0.612101682005147,
1610
+ 0.678935294179132
1611
+ ]
1612
+ },
1613
+ "eng": {
1614
+ "acc": 0.4549019607843137,
1615
+ "n": 765,
1616
+ "ci95": [
1617
+ 0.4199272911685178,
1618
+ 0.4903273048366933
1619
+ ]
1620
+ },
1621
+ "ilo": {
1622
+ "acc": 0.6609442060085837,
1623
+ "n": 699,
1624
+ "ci95": [
1625
+ 0.6250552920839941,
1626
+ 0.6950737384116398
1627
+ ]
1628
+ },
1629
+ "lower": {
1630
+ "acc": 0.9988864142538976,
1631
+ "n": 1796,
1632
+ "ci95": [
1633
+ 0.9959486042757345,
1634
+ 0.9996945678789707
1635
+ ]
1636
+ },
1637
+ "pag": {
1638
+ "acc": 0.6085271317829457,
1639
+ "n": 774,
1640
+ "ci95": [
1641
+ 0.573686384289186,
1642
+ 0.6422958928222464
1643
+ ]
1644
+ },
1645
+ "spa": {
1646
+ "acc": 0.5458015267175572,
1647
+ "n": 786,
1648
+ "ci95": [
1649
+ 0.5108543219550099,
1650
+ 0.5803031962106577
1651
+ ]
1652
+ },
1653
+ "tgl": {
1654
+ "acc": 0.6308100929614874,
1655
+ "n": 753,
1656
+ "ci95": [
1657
+ 0.5957579765214207,
1658
+ 0.6645342695707603
1659
+ ]
1660
+ },
1661
+ "title": {
1662
+ "acc": 0.7260579064587973,
1663
+ "n": 1796,
1664
+ "ci95": [
1665
+ 0.704965643486489,
1666
+ 0.7461851688961116
1667
+ ]
1668
+ },
1669
+ "upper": {
1670
+ "acc": 0.0,
1671
+ "n": 1796,
1672
+ "ci95": [
1673
+ 0.0,
1674
+ 0.002134410050306649
1675
+ ]
1676
+ },
1677
+ "war": {
1678
+ "acc": 0.4915458937198068,
1679
+ "n": 828,
1680
+ "ci95": [
1681
+ 0.4576111496341992,
1682
+ 0.5255587230915697
1683
+ ]
1684
+ }
1685
+ }
1686
+ },
1687
+ "lid176": {
1688
+ "acc": 0.5215293244246474,
1689
+ "ci95": [
1690
+ 0.5081801578367945,
1691
+ 0.5348478124210855
1692
+ ],
1693
+ "by": {
1694
+ "ceb": {
1695
+ "acc": 0.4661558109833972,
1696
+ "n": 783,
1697
+ "ci95": [
1698
+ 0.4314640406362553,
1699
+ 0.501178056578737
1700
+ ]
1701
+ },
1702
+ "eng": {
1703
+ "acc": 0.9647058823529412,
1704
+ "n": 765,
1705
+ "ci95": [
1706
+ 0.9491356016719222,
1707
+ 0.9756322568830781
1708
+ ]
1709
+ },
1710
+ "ilo": {
1711
+ "acc": 0.43061516452074394,
1712
+ "n": 699,
1713
+ "ci95": [
1714
+ 0.3943845333535883,
1715
+ 0.46760428461051645
1716
+ ]
1717
+ },
1718
+ "lower": {
1719
+ "acc": 0.7878619153674833,
1720
+ "n": 1796,
1721
+ "ci95": [
1722
+ 0.7683500600977665,
1723
+ 0.8061449399066783
1724
+ ]
1725
+ },
1726
+ "pag": {
1727
+ "acc": 0.0,
1728
+ "n": 774,
1729
+ "ci95": [
1730
+ 0.0,
1731
+ 0.004938794736614755
1732
+ ]
1733
+ },
1734
+ "spa": {
1735
+ "acc": 0.9936386768447837,
1736
+ "n": 786,
1737
+ "ci95": [
1738
+ 0.9851955812525237,
1739
+ 0.9972798923107831
1740
+ ]
1741
+ },
1742
+ "tgl": {
1743
+ "acc": 0.5816733067729084,
1744
+ "n": 753,
1745
+ "ci95": [
1746
+ 0.5461123229613584,
1747
+ 0.6164051707916277
1748
+ ]
1749
+ },
1750
+ "title": {
1751
+ "acc": 0.5066815144766147,
1752
+ "n": 1796,
1753
+ "ci95": [
1754
+ 0.48356947361579444,
1755
+ 0.5297650331541347
1756
+ ]
1757
+ },
1758
+ "upper": {
1759
+ "acc": 0.2700445434298441,
1760
+ "n": 1796,
1761
+ "ci95": [
1762
+ 0.2500176421747124,
1763
+ 0.2910530831602282
1764
+ ]
1765
+ },
1766
+ "war": {
1767
+ "acc": 0.22584541062801933,
1768
+ "n": 828,
1769
+ "ci95": [
1770
+ 0.19866781968925123,
1771
+ 0.25555519584639885
1772
+ ]
1773
+ }
1774
+ }
1775
+ }
1776
+ },
1777
+ "t_bikol": {
1778
+ "n": 1800,
1779
+ "wikalid-1.0.1": {
1780
+ "acc": 0.94,
1781
+ "ci95": [
1782
+ 0.9280633120585668,
1783
+ 0.9500625721654137
1784
+ ],
1785
+ "by": {
1786
+ "bcl": {
1787
+ "acc": 0.9611111111111111,
1788
+ "n": 900,
1789
+ "ci95": [
1790
+ 0.9463957540769419,
1791
+ 0.9719067449450107
1792
+ ]
1793
+ },
1794
+ "ceb": {
1795
+ "acc": 0.9188888888888889,
1796
+ "n": 900,
1797
+ "ci95": [
1798
+ 0.8992212323860316,
1799
+ 0.934995736604995
1800
+ ]
1801
+ },
1802
+ "story": {
1803
+ "acc": 0.94,
1804
+ "n": 1800,
1805
+ "ci95": [
1806
+ 0.9280633120585668,
1807
+ 0.9500625721654137
1808
+ ]
1809
+ }
1810
+ }
1811
+ },
1812
+ "glotlid": {
1813
+ "acc": 0.8933333333333333,
1814
+ "ci95": [
1815
+ 0.878225542898005,
1816
+ 0.9067657778476745
1817
+ ],
1818
+ "by": {
1819
+ "bcl": {
1820
+ "acc": 0.8944444444444445,
1821
+ "n": 900,
1822
+ "ci95": [
1823
+ 0.8726657739209606,
1824
+ 0.9128700987363724
1825
+ ]
1826
+ },
1827
+ "ceb": {
1828
+ "acc": 0.8922222222222222,
1829
+ "n": 900,
1830
+ "ci95": [
1831
+ 0.8702698070914948,
1832
+ 0.910840511353684
1833
+ ]
1834
+ },
1835
+ "story": {
1836
+ "acc": 0.8933333333333333,
1837
+ "n": 1800,
1838
+ "ci95": [
1839
+ 0.878225542898005,
1840
+ 0.9067657778476745
1841
+ ]
1842
+ }
1843
+ }
1844
+ },
1845
+ "openlid3": {
1846
+ "acc": 0.4666666666666667,
1847
+ "ci95": [
1848
+ 0.4437146676882942,
1849
+ 0.48976064411291936
1850
+ ],
1851
+ "by": {
1852
+ "bcl": {
1853
+ "acc": 0.0,
1854
+ "n": 900,
1855
+ "ci95": [
1856
+ 4.336808689942018e-19,
1857
+ 0.004250302265352689
1858
+ ]
1859
+ },
1860
+ "ceb": {
1861
+ "acc": 0.9333333333333333,
1862
+ "n": 900,
1863
+ "ci95": [
1864
+ 0.9151252450043573,
1865
+ 0.9478578263656704
1866
+ ]
1867
+ },
1868
+ "story": {
1869
+ "acc": 0.4666666666666667,
1870
+ "n": 1800,
1871
+ "ci95": [
1872
+ 0.4437146676882942,
1873
+ 0.48976064411291936
1874
+ ]
1875
+ }
1876
+ }
1877
+ },
1878
+ "lid176": {
1879
+ "acc": 0.2911111111111111,
1880
+ "ci95": [
1881
+ 0.27058719962290767,
1882
+ 0.3125247543313642
1883
+ ],
1884
+ "by": {
1885
+ "bcl": {
1886
+ "acc": 0.0033333333333333335,
1887
+ "n": 900,
1888
+ "ci95": [
1889
+ 0.001134247210102744,
1890
+ 0.009754386373480927
1891
+ ]
1892
+ },
1893
+ "ceb": {
1894
+ "acc": 0.5788888888888889,
1895
+ "n": 900,
1896
+ "ci95": [
1897
+ 0.5463629609744777,
1898
+ 0.6107442135569888
1899
+ ]
1900
+ },
1901
+ "story": {
1902
+ "acc": 0.2911111111111111,
1903
+ "n": 1800,
1904
+ "ci95": [
1905
+ 0.27058719962290767,
1906
+ 0.3125247543313642
1907
+ ]
1908
+ }
1909
+ }
1910
+ }
1911
+ },
1912
+ "t_aklsgd": {
1913
+ "n": 110,
1914
+ "wikalid-1.0.1": {
1915
+ "acc": 0.34545454545454546,
1916
+ "ci95": [
1917
+ 0.26316254590275023,
1918
+ 0.4381768590247983
1919
+ ],
1920
+ "by": {
1921
+ "incubator": {
1922
+ "acc": 0.3695652173913043,
1923
+ "n": 92,
1924
+ "ci95": [
1925
+ 0.27801481252642296,
1926
+ 0.4715720057236897
1927
+ ]
1928
+ },
1929
+ "oth": {
1930
+ "acc": 0.34545454545454546,
1931
+ "n": 110,
1932
+ "ci95": [
1933
+ 0.26316254590275023,
1934
+ 0.4381768590247983
1935
+ ]
1936
+ },
1937
+ "tatoeba": {
1938
+ "acc": 0.2222222222222222,
1939
+ "n": 18,
1940
+ "ci95": [
1941
+ 0.09000781913382,
1942
+ 0.4521502645230549
1943
+ ]
1944
+ }
1945
+ }
1946
+ },
1947
+ "glotlid": {
1948
+ "acc": 0.3,
1949
+ "ci95": [
1950
+ 0.22229769508028016,
1951
+ 0.3912003583553708
1952
+ ],
1953
+ "by": {
1954
+ "incubator": {
1955
+ "acc": 0.2717391304347826,
1956
+ "n": 92,
1957
+ "ci95": [
1958
+ 0.19135648353357726,
1959
+ 0.3704204484041199
1960
+ ]
1961
+ },
1962
+ "oth": {
1963
+ "acc": 0.3,
1964
+ "n": 110,
1965
+ "ci95": [
1966
+ 0.22229769508028016,
1967
+ 0.3912003583553708
1968
+ ]
1969
+ },
1970
+ "tatoeba": {
1971
+ "acc": 0.4444444444444444,
1972
+ "n": 18,
1973
+ "ci95": [
1974
+ 0.24559231101389126,
1975
+ 0.6628393057174837
1976
+ ]
1977
+ }
1978
+ }
1979
+ },
1980
+ "openlid3": {
1981
+ "acc": 0.08181818181818182,
1982
+ "ci95": [
1983
+ 0.04364015135431728,
1984
+ 0.14821941492022558
1985
+ ],
1986
+ "by": {
1987
+ "incubator": {
1988
+ "acc": 0.043478260869565216,
1989
+ "n": 92,
1990
+ "ci95": [
1991
+ 0.01703544061158062,
1992
+ 0.10651842326381375
1993
+ ]
1994
+ },
1995
+ "oth": {
1996
+ "acc": 0.08181818181818182,
1997
+ "n": 110,
1998
+ "ci95": [
1999
+ 0.04364015135431728,
2000
+ 0.14821941492022558
2001
+ ]
2002
+ },
2003
+ "tatoeba": {
2004
+ "acc": 0.2777777777777778,
2005
+ "n": 18,
2006
+ "ci95": [
2007
+ 0.12499569146355474,
2008
+ 0.5087307754619452
2009
+ ]
2010
+ }
2011
+ }
2012
+ },
2013
+ "lid176": {
2014
+ "acc": 0.05454545454545454,
2015
+ "ci95": [
2016
+ 0.025235483062071407,
2017
+ 0.11391927231733298
2018
+ ],
2019
+ "by": {
2020
+ "incubator": {
2021
+ "acc": 0.010869565217391304,
2022
+ "n": 92,
2023
+ "ci95": [
2024
+ 0.0019212795763616508,
2025
+ 0.05902928886156091
2026
+ ]
2027
+ },
2028
+ "oth": {
2029
+ "acc": 0.05454545454545454,
2030
+ "n": 110,
2031
+ "ci95": [
2032
+ 0.025235483062071407,
2033
+ 0.11391927231733298
2034
+ ]
2035
+ },
2036
+ "tatoeba": {
2037
+ "acc": 0.2777777777777778,
2038
+ "n": 18,
2039
+ "ci95": [
2040
+ 0.12499569146355474,
2041
+ 0.5087307754619452
2042
+ ]
2043
+ }
2044
+ }
2045
+ }
2046
+ },
2047
+ "t_tshort": {
2048
+ "n": 543,
2049
+ "wikalid-1.0.1": {
2050
+ "acc": 0.856353591160221,
2051
+ "ci95": [
2052
+ 0.824347035657435,
2053
+ 0.8833533294940092
2054
+ ],
2055
+ "by": {
2056
+ "cbk": {
2057
+ "acc": 0.9367088607594937,
2058
+ "n": 79,
2059
+ "ci95": [
2060
+ 0.8602493258443956,
2061
+ 0.9726655381852706
2062
+ ]
2063
+ },
2064
+ "ceb": {
2065
+ "acc": 0.6666666666666666,
2066
+ "n": 39,
2067
+ "ci95": [
2068
+ 0.5097710432024052,
2069
+ 0.7936723389308484
2070
+ ]
2071
+ },
2072
+ "eng": {
2073
+ "acc": 1.0,
2074
+ "n": 58,
2075
+ "ci95": [
2076
+ 0.9378800031047062,
2077
+ 0.9999999999999999
2078
+ ]
2079
+ },
2080
+ "hil": {
2081
+ "acc": 1.0,
2082
+ "n": 1,
2083
+ "ci95": [
2084
+ 0.20654329147389294,
2085
+ 1.0
2086
+ ]
2087
+ },
2088
+ "ilo": {
2089
+ "acc": 0.8222222222222222,
2090
+ "n": 45,
2091
+ "ci95": [
2092
+ 0.6866989315634587,
2093
+ 0.9070572107414617
2094
+ ]
2095
+ },
2096
+ "ind": {
2097
+ "acc": 0.8410596026490066,
2098
+ "n": 151,
2099
+ "ci95": [
2100
+ 0.7743902105949425,
2101
+ 0.8908057057479524
2102
+ ]
2103
+ },
2104
+ "pag": {
2105
+ "acc": 1.0,
2106
+ "n": 1,
2107
+ "ci95": [
2108
+ 0.20654329147389294,
2109
+ 1.0
2110
+ ]
2111
+ },
2112
+ "pam": {
2113
+ "acc": 0.782608695652174,
2114
+ "n": 23,
2115
+ "ci95": [
2116
+ 0.5809611981726508,
2117
+ 0.9033616439828094
2118
+ ]
2119
+ },
2120
+ "tatoeba": {
2121
+ "acc": 0.856353591160221,
2122
+ "n": 543,
2123
+ "ci95": [
2124
+ 0.824347035657435,
2125
+ 0.8833533294940092
2126
+ ]
2127
+ },
2128
+ "tgl": {
2129
+ "acc": 0.9705882352941176,
2130
+ "n": 102,
2131
+ "ci95": [
2132
+ 0.9170680330254163,
2133
+ 0.9899477339319994
2134
+ ]
2135
+ },
2136
+ "war": {
2137
+ "acc": 0.5454545454545454,
2138
+ "n": 44,
2139
+ "ci95": [
2140
+ 0.4006593217065349,
2141
+ 0.6829499221314639
2142
+ ]
2143
+ }
2144
+ }
2145
+ },
2146
+ "glotlid": {
2147
+ "acc": 0.8655616942909761,
2148
+ "ci95": [
2149
+ 0.8342870692237219,
2150
+ 0.8917001270685865
2151
+ ],
2152
+ "by": {
2153
+ "cbk": {
2154
+ "acc": 0.6455696202531646,
2155
+ "n": 79,
2156
+ "ci95": [
2157
+ 0.5355906892335188,
2158
+ 0.7420475987763698
2159
+ ]
2160
+ },
2161
+ "ceb": {
2162
+ "acc": 0.7948717948717948,
2163
+ "n": 39,
2164
+ "ci95": [
2165
+ 0.6446572703687138,
2166
+ 0.8922040980208888
2167
+ ]
2168
+ },
2169
+ "eng": {
2170
+ "acc": 0.9827586206896551,
2171
+ "n": 58,
2172
+ "ci95": [
2173
+ 0.9085892732270167,
2174
+ 0.9969500401154581
2175
+ ]
2176
+ },
2177
+ "hil": {
2178
+ "acc": 0.0,
2179
+ "n": 1,
2180
+ "ci95": [
2181
+ 0.0,
2182
+ 0.7934567085261071
2183
+ ]
2184
+ },
2185
+ "ilo": {
2186
+ "acc": 0.8888888888888888,
2187
+ "n": 45,
2188
+ "ci95": [
2189
+ 0.7650062450910905,
2190
+ 0.9515959956217445
2191
+ ]
2192
+ },
2193
+ "ind": {
2194
+ "acc": 0.9139072847682119,
2195
+ "n": 151,
2196
+ "ci95": [
2197
+ 0.8582784981751292,
2198
+ 0.948998099328384
2199
+ ]
2200
+ },
2201
+ "pag": {
2202
+ "acc": 1.0,
2203
+ "n": 1,
2204
+ "ci95": [
2205
+ 0.20654329147389294,
2206
+ 1.0
2207
+ ]
2208
+ },
2209
+ "pam": {
2210
+ "acc": 0.8260869565217391,
2211
+ "n": 23,
2212
+ "ci95": [
2213
+ 0.6286194521470433,
2214
+ 0.9302145964938723
2215
+ ]
2216
+ },
2217
+ "tatoeba": {
2218
+ "acc": 0.8655616942909761,
2219
+ "n": 543,
2220
+ "ci95": [
2221
+ 0.8342870692237219,
2222
+ 0.8917001270685865
2223
+ ]
2224
+ },
2225
+ "tgl": {
2226
+ "acc": 0.9901960784313726,
2227
+ "n": 102,
2228
+ "ci95": [
2229
+ 0.9465407803733753,
2230
+ 0.9982673102072662
2231
+ ]
2232
+ },
2233
+ "war": {
2234
+ "acc": 0.7272727272727273,
2235
+ "n": 44,
2236
+ "ci95": [
2237
+ 0.581508453144907,
2238
+ 0.8365377660450867
2239
+ ]
2240
+ }
2241
+ }
2242
+ },
2243
+ "openlid3": {
2244
+ "acc": 0.6316758747697975,
2245
+ "ci95": [
2246
+ 0.5903117707766475,
2247
+ 0.6711899145377104
2248
+ ],
2249
+ "by": {
2250
+ "cbk": {
2251
+ "acc": 0.0,
2252
+ "n": 79,
2253
+ "ci95": [
2254
+ -3.469446951953614e-18,
2255
+ 0.04637283683559951
2256
+ ]
2257
+ },
2258
+ "ceb": {
2259
+ "acc": 0.8461538461538461,
2260
+ "n": 39,
2261
+ "ci95": [
2262
+ 0.7027023013426569,
2263
+ 0.9275262615494854
2264
+ ]
2265
+ },
2266
+ "eng": {
2267
+ "acc": 0.6551724137931034,
2268
+ "n": 58,
2269
+ "ci95": [
2270
+ 0.5266753715786767,
2271
+ 0.7643908362814045
2272
+ ]
2273
+ },
2274
+ "hil": {
2275
+ "acc": 0.0,
2276
+ "n": 1,
2277
+ "ci95": [
2278
+ 0.0,
2279
+ 0.7934567085261071
2280
+ ]
2281
+ },
2282
+ "ilo": {
2283
+ "acc": 0.8444444444444444,
2284
+ "n": 45,
2285
+ "ci95": [
2286
+ 0.712158094887512,
2287
+ 0.9225467468867133
2288
+ ]
2289
+ },
2290
+ "ind": {
2291
+ "acc": 0.8609271523178808,
2292
+ "n": 151,
2293
+ "ci95": [
2294
+ 0.7967393701331015,
2295
+ 0.9072058228899619
2296
+ ]
2297
+ },
2298
+ "pag": {
2299
+ "acc": 1.0,
2300
+ "n": 1,
2301
+ "ci95": [
2302
+ 0.20654329147389294,
2303
+ 1.0
2304
+ ]
2305
+ },
2306
+ "pam": {
2307
+ "acc": 0.0,
2308
+ "n": 23,
2309
+ "ci95": [
2310
+ 0.0,
2311
+ 0.14312112541726274
2312
+ ]
2313
+ },
2314
+ "tatoeba": {
2315
+ "acc": 0.6316758747697975,
2316
+ "n": 543,
2317
+ "ci95": [
2318
+ 0.5903117707766475,
2319
+ 0.6711899145377104
2320
+ ]
2321
+ },
2322
+ "tgl": {
2323
+ "acc": 0.7941176470588235,
2324
+ "n": 102,
2325
+ "ci95": [
2326
+ 0.7056727244152231,
2327
+ 0.8612121299331615
2328
+ ]
2329
+ },
2330
+ "war": {
2331
+ "acc": 0.5,
2332
+ "n": 44,
2333
+ "ci95": [
2334
+ 0.35831521095931473,
2335
+ 0.6416847890406853
2336
+ ]
2337
+ }
2338
+ }
2339
+ },
2340
+ "lid176": {
2341
+ "acc": 0.6850828729281768,
2342
+ "ci95": [
2343
+ 0.6448299972330791,
2344
+ 0.7227353086982913
2345
+ ],
2346
+ "by": {
2347
+ "cbk": {
2348
+ "acc": 0.20253164556962025,
2349
+ "n": 79,
2350
+ "ci95": [
2351
+ 0.12869001243979505,
2352
+ 0.303962181626954
2353
+ ]
2354
+ },
2355
+ "ceb": {
2356
+ "acc": 0.5897435897435898,
2357
+ "n": 39,
2358
+ "ci95": [
2359
+ 0.43418348473344903,
2360
+ 0.7292091056459952
2361
+ ]
2362
+ },
2363
+ "eng": {
2364
+ "acc": 1.0,
2365
+ "n": 58,
2366
+ "ci95": [
2367
+ 0.9378800031047062,
2368
+ 0.9999999999999999
2369
+ ]
2370
+ },
2371
+ "hil": {
2372
+ "acc": 0.0,
2373
+ "n": 1,
2374
+ "ci95": [
2375
+ 0.0,
2376
+ 0.7934567085261071
2377
+ ]
2378
+ },
2379
+ "ilo": {
2380
+ "acc": 0.5555555555555556,
2381
+ "n": 45,
2382
+ "ci95": [
2383
+ 0.4117588341636533,
2384
+ 0.6906129145096089
2385
+ ]
2386
+ },
2387
+ "ind": {
2388
+ "acc": 0.8874172185430463,
2389
+ "n": 151,
2390
+ "ci95": [
2391
+ 0.827099549136263,
2392
+ 0.9285113461270252
2393
+ ]
2394
+ },
2395
+ "pag": {
2396
+ "acc": 0.0,
2397
+ "n": 1,
2398
+ "ci95": [
2399
+ 0.0,
2400
+ 0.7934567085261071
2401
+ ]
2402
+ },
2403
+ "pam": {
2404
+ "acc": 0.6521739130434783,
2405
+ "n": 23,
2406
+ "ci95": [
2407
+ 0.4488997690689164,
2408
+ 0.8118894536301776
2409
+ ]
2410
+ },
2411
+ "tatoeba": {
2412
+ "acc": 0.6850828729281768,
2413
+ "n": 543,
2414
+ "ci95": [
2415
+ 0.6448299972330791,
2416
+ 0.7227353086982913
2417
+ ]
2418
+ },
2419
+ "tgl": {
2420
+ "acc": 0.8235294117647058,
2421
+ "n": 102,
2422
+ "ci95": [
2423
+ 0.7382155715042175,
2424
+ 0.8853577682790058
2425
+ ]
2426
+ },
2427
+ "war": {
2428
+ "acc": 0.38636363636363635,
2429
+ "n": 44,
2430
+ "ci95": [
2431
+ 0.25720998886701985,
2432
+ 0.5337669015379832
2433
+ ]
2434
+ }
2435
+ }
2436
+ }
2437
+ }
2438
+ },
2439
+ "paired_vs_glotlid": {
2440
+ "crawl": {
2441
+ "delta": 0.07462686567164178,
2442
+ "ci95": [
2443
+ 0.04470149253731343,
2444
+ 0.11044776119402985
2445
+ ]
2446
+ },
2447
+ "goldspan": {
2448
+ "delta": 0.039755351681957186,
2449
+ "ci95": [
2450
+ 0.009174311926605505,
2451
+ 0.07339449541284404
2452
+ ]
2453
+ },
2454
+ "golddoc": {
2455
+ "delta": 0.02702702702702703,
2456
+ "ci95": [
2457
+ 0.006756756756756757,
2458
+ 0.05405405405405406
2459
+ ]
2460
+ },
2461
+ "pld": {
2462
+ "delta": 0.08287961282516637,
2463
+ "ci95": [
2464
+ 0.06654567453115548,
2465
+ 0.09921355111917725
2466
+ ]
2467
+ },
2468
+ "wiki": {
2469
+ "delta": 0.01833788529067499,
2470
+ "ci95": [
2471
+ 0.009754194303550527,
2472
+ 0.027311744049941473
2473
+ ]
2474
+ },
2475
+ "hard": {
2476
+ "delta": 0.07164132309139364,
2477
+ "ci95": [
2478
+ 0.06317604640225741,
2479
+ 0.08026336416366202
2480
+ ]
2481
+ },
2482
+ "basaha": {
2483
+ "delta": 0.1561712846347607,
2484
+ "ci95": [
2485
+ 0.12342569269521411,
2486
+ 0.1889168765743073
2487
+ ]
2488
+ },
2489
+ "tato": {
2490
+ "delta": -0.026859504132231406,
2491
+ "ci95": [
2492
+ -0.04648760330578512,
2493
+ -0.009297520661157025
2494
+ ]
2495
+ }
2496
+ },
2497
+ "gate_sealed": {
2498
+ "crawl": {
2499
+ "coverage": 0.5104477611940299,
2500
+ "precision_accepted_corrected": 1.0,
2501
+ "n": 335
2502
+ },
2503
+ "goldspan": {
2504
+ "coverage": 0.5137614678899083,
2505
+ "precision_accepted_corrected": 1.0,
2506
+ "n": 327
2507
+ },
2508
+ "golddoc": {
2509
+ "coverage": 0.7837837837837838,
2510
+ "precision_accepted_corrected": 0.9913793103448276,
2511
+ "n": 148
2512
+ },
2513
+ "pld": {
2514
+ "coverage": 0.6581972171808832,
2515
+ "precision_accepted_corrected": 0.9852941176470589,
2516
+ "n": 1653
2517
+ },
2518
+ "wiki": {
2519
+ "coverage": 0.5657432696059306,
2520
+ "precision_accepted_corrected": 0.9889655172413793,
2521
+ "n": 2563
2522
+ },
2523
+ "hard": {
2524
+ "coverage": 0.5101113027120239,
2525
+ "precision_accepted_corrected": 0.994161032575292,
2526
+ "n": 6379
2527
+ },
2528
+ "basaha": {
2529
+ "coverage": 0.35768261964735515,
2530
+ "precision_accepted_corrected": 0.971830985915493,
2531
+ "n": 794
2532
+ },
2533
+ "tato": {
2534
+ "coverage": 0.39462809917355374,
2535
+ "precision_accepted_corrected": 0.9947643979057592,
2536
+ "n": 968
2537
+ }
2538
+ },
2539
+ "consistency_vs_research_c7r": {
2540
+ "crawl": {
2541
+ "n": 661,
2542
+ "same_label": 661
2543
+ },
2544
+ "goldspan": {
2545
+ "n": 593,
2546
+ "same_label": 593
2547
+ },
2548
+ "golddoc": {
2549
+ "n": 294,
2550
+ "same_label": 294
2551
+ },
2552
+ "pld": {
2553
+ "n": 3222,
2554
+ "same_label": 3222
2555
+ },
2556
+ "wiki": {
2557
+ "n": 5037,
2558
+ "same_label": 5037
2559
+ },
2560
+ "hard": {
2561
+ "n": 12520,
2562
+ "same_label": 12520
2563
+ },
2564
+ "basaha": {
2565
+ "n": 1600,
2566
+ "same_label": 1600
2567
+ },
2568
+ "tato": {
2569
+ "n": 1847,
2570
+ "same_label": 1847
2571
+ },
2572
+ "t_sib": {
2573
+ "n": 9036,
2574
+ "same_label": 9036
2575
+ },
2576
+ "t_frag": {
2577
+ "n": 4320,
2578
+ "same_label": 4320
2579
+ },
2580
+ "t_caps": {
2581
+ "n": 5388,
2582
+ "same_label": 5388
2583
+ },
2584
+ "t_bikol": {
2585
+ "n": 1799,
2586
+ "same_label": 1799
2587
+ },
2588
+ "t_aklsgd": {
2589
+ "n": 110,
2590
+ "same_label": 110
2591
+ },
2592
+ "t_tshort": {
2593
+ "n": 543,
2594
+ "same_label": 543
2595
+ }
2596
+ }
2597
+ }
eval/RESULTS_v1.0.1.md ADDED
@@ -0,0 +1,117 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # WikaLID 1.0.1 evaluation (generated by eval/evaluate.py; do not edit by hand)
2
+
3
+ ## Sealed halves (never used for any decision): accuracy raw / corrected gold (macro-F1 corrected)
4
+
5
+ | system | crawl (n=335) | goldspan (n=327) | golddoc (n=148) | pld (n=1653) | wiki* (n=2563) | hard* (n=6379) | basaha (n=794) | tato* (n=968) | mean corrected |
6
+ |---|---|---|---|---|---|---|---|---|---|
7
+ | wikalid-1.0.1 | 0.949 / 0.949 (0.95) | 0.939 / 0.939 (0.93) | 0.986 / 0.986 (1.00) | 0.916 / 0.917 (0.93) | 0.899 / 0.925 (0.93) | 0.879 / 0.901 (0.88) | 0.739 / 0.739 (0.79) | 0.931 / 0.928 (0.94) | **0.911** |
8
+ | glotlid | 0.875 / 0.875 (0.93) | 0.899 / 0.899 (0.92) | 0.959 / 0.959 (1.00) | 0.830 / 0.834 (0.84) | 0.869 / 0.907 (0.91) | 0.799 / 0.829 (0.76) | 0.583 / 0.583 (0.64) | 0.949 / 0.955 (0.97) | **0.855** |
9
+ | openlid3 | 0.785 / 0.785 (0.62) | 0.881 / 0.881 (0.87) | 0.919 / 0.919 (0.99) | 0.417 / 0.417 (0.40) | 0.676 / 0.710 (0.63) | 0.619 / 0.641 (0.55) | 0.026 / 0.026 (0.01) | 0.754 / 0.758 (0.68) | **0.642** |
10
+ | lid176 | 0.734 / 0.734 (0.57) | 0.783 / 0.783 (0.74) | 0.818 / 0.818 (0.90) | 0.337 / 0.341 (0.29) | 0.620 / 0.664 (0.55) | 0.567 / 0.599 (0.45) | 0.031 / 0.031 (0.03) | 0.801 / 0.805 (0.80) | **0.597** |
11
+
12
+ \* public set (ships in eval/data, anyone can re-run). Others are private (news gold, crawl realism, PLD read speech, BasahaCorpus stories).
13
+
14
+ ## WikaLID 1.0.1 minus GlotLID v3, sealed, corrected gold (paired bootstrap, 2,000 resamples)
15
+
16
+ | set | delta | 95% CI |
17
+ |---|---|---|
18
+ | crawl | +0.075 | [+0.045, +0.110] |
19
+ | goldspan | +0.040 | [+0.009, +0.073] |
20
+ | golddoc | +0.027 | [+0.007, +0.054] |
21
+ | pld | +0.083 | [+0.067, +0.099] |
22
+ | wiki | +0.018 | [+0.010, +0.027] |
23
+ | hard | +0.072 | [+0.063, +0.080] |
24
+ | basaha | +0.156 | [+0.123, +0.189] |
25
+ | tato | -0.027 | [-0.046, -0.009] |
26
+
27
+ ## Gate v2 on sealed halves (accept = accept / accept_eng / accept_ph_other)
28
+
29
+ | set | coverage | precision of accepted (corrected gold) |
30
+ |---|---|---|
31
+ | crawl | 0.510 | 1.000 |
32
+ | goldspan | 0.514 | 1.000 |
33
+ | golddoc | 0.784 | 0.991 |
34
+ | pld | 0.658 | 0.985 |
35
+ | wiki | 0.566 | 0.989 |
36
+ | hard | 0.510 | 0.994 |
37
+ | basaha | 0.358 | 0.972 |
38
+ | tato | 0.395 | 0.995 |
39
+
40
+ ## Human-labelled external trust sets (all rows; accuracy with 95% Wilson interval)
41
+
42
+ ### t_sib (n=9036)
43
+
44
+ | system | overall | ceb | eng | ilo | ind | msa | pag | sib200 | spa | tgl | war |
45
+ |---|---|---|---|---|---|---|---|---|---|---|---|
46
+ | wikalid-1.0.1 | 0.989 [0.99-0.99] | 0.997 | 1.000 | 1.000 | 0.959 | 0.953 | 0.997 | 0.989 | 0.997 | 1.000 | 1.000 |
47
+ | glotlid | 0.986 [0.98-0.99] | 0.991 | 1.000 | 0.999 | 0.995 | 0.894 | 1.000 | 0.986 | 0.998 | 1.000 | 1.000 |
48
+ | openlid3 | 0.985 [0.98-0.99] | 1.000 | 0.998 | 1.000 | 0.918 | 0.963 | 0.996 | 0.985 | 0.986 | 1.000 | 1.000 |
49
+ | lid176 | 0.774 [0.77-0.78] | 0.922 | 1.000 | 0.985 | 0.969 | 0.449 | 0.000 | 0.774 | 1.000 | 0.998 | 0.640 |
50
+
51
+ ### t_frag (n=4320)
52
+
53
+ | system | overall | ceb | eng | ilo | ind | len2 | len3 | len4 | len5 | msa | pag | spa | tgl | war |
54
+ |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
55
+ | wikalid-1.0.1 | 0.832 [0.82-0.84] | 0.815 | 0.977 | 0.942 | 0.821 | 0.704 | 0.815 | 0.878 | 0.931 | 0.640 | 0.873 | 0.871 | 0.844 | 0.706 |
56
+ | glotlid | 0.771 [0.76-0.78] | 0.760 | 0.908 | 0.892 | 0.867 | 0.624 | 0.739 | 0.835 | 0.887 | 0.342 | 0.823 | 0.850 | 0.867 | 0.633 |
57
+ | openlid3 | 0.773 [0.76-0.79] | 0.842 | 0.717 | 0.933 | 0.810 | 0.621 | 0.750 | 0.834 | 0.888 | 0.550 | 0.790 | 0.692 | 0.887 | 0.740 |
58
+ | lid176 | 0.590 [0.58-0.60] | 0.681 | 0.992 | 0.546 | 0.750 | 0.514 | 0.547 | 0.628 | 0.671 | 0.354 | 0.000 | 0.933 | 0.729 | 0.325 |
59
+
60
+ ### t_caps (n=5388)
61
+
62
+ | system | overall | ceb | eng | ilo | lower | pag | spa | tgl | title | upper | war |
63
+ |---|---|---|---|---|---|---|---|---|---|---|---|
64
+ | wikalid-1.0.1 | 0.999 [1.00-1.00] | 0.995 | 1.000 | 1.000 | 0.998 | 0.999 | 1.000 | 0.999 | 0.999 | 0.999 | 1.000 |
65
+ | glotlid | 0.975 [0.97-0.98] | 0.959 | 0.931 | 0.999 | 0.998 | 0.994 | 0.949 | 0.996 | 0.990 | 0.937 | 1.000 |
66
+ | openlid3 | 0.575 [0.56-0.59] | 0.646 | 0.455 | 0.661 | 0.999 | 0.609 | 0.546 | 0.631 | 0.726 | 0.000 | 0.492 |
67
+ | lid176 | 0.522 [0.51-0.53] | 0.466 | 0.965 | 0.431 | 0.788 | 0.000 | 0.994 | 0.582 | 0.507 | 0.270 | 0.226 |
68
+
69
+ ### t_bikol (n=1800)
70
+
71
+ | system | overall | bcl | ceb | story |
72
+ |---|---|---|---|---|
73
+ | wikalid-1.0.1 | 0.940 [0.93-0.95] | 0.961 | 0.919 | 0.940 |
74
+ | glotlid | 0.893 [0.88-0.91] | 0.894 | 0.892 | 0.893 |
75
+ | openlid3 | 0.467 [0.44-0.49] | 0.000 | 0.933 | 0.467 |
76
+ | lid176 | 0.291 [0.27-0.31] | 0.003 | 0.579 | 0.291 |
77
+
78
+ ### t_aklsgd (n=110)
79
+
80
+ | system | overall | incubator | oth | tatoeba |
81
+ |---|---|---|---|---|
82
+ | wikalid-1.0.1 | 0.345 [0.26-0.44] | 0.370 | 0.345 | 0.222 |
83
+ | glotlid | 0.300 [0.22-0.39] | 0.272 | 0.300 | 0.444 |
84
+ | openlid3 | 0.082 [0.04-0.15] | 0.043 | 0.082 | 0.278 |
85
+ | lid176 | 0.055 [0.03-0.11] | 0.011 | 0.055 | 0.278 |
86
+
87
+ ### t_tshort (n=543)
88
+
89
+ | system | overall | cbk | ceb | eng | hil | ilo | ind | pag | pam | tatoeba | tgl | war |
90
+ |---|---|---|---|---|---|---|---|---|---|---|---|---|
91
+ | wikalid-1.0.1 | 0.856 [0.82-0.88] | 0.937 | 0.667 | 1.000 | 1.000 | 0.822 | 0.841 | 1.000 | 0.783 | 0.856 | 0.971 | 0.545 |
92
+ | glotlid | 0.866 [0.83-0.89] | 0.646 | 0.795 | 0.983 | 0.000 | 0.889 | 0.914 | 1.000 | 0.826 | 0.866 | 0.990 | 0.727 |
93
+ | openlid3 | 0.632 [0.59-0.67] | 0.000 | 0.846 | 0.655 | 0.000 | 0.844 | 0.861 | 1.000 | 0.000 | 0.632 | 0.794 | 0.500 |
94
+ | lid176 | 0.685 [0.64-0.72] | 0.203 | 0.590 | 1.000 | 0.000 | 0.556 | 0.887 | 0.000 | 0.652 | 0.685 | 0.824 | 0.386 |
95
+
96
+ ## Consistency with the research model that labels Himay (c7 + rules, cached predictions)
97
+
98
+ | set | n | same label |
99
+ |---|---|---|
100
+ | crawl | 661 | 661 |
101
+ | goldspan | 593 | 593 |
102
+ | golddoc | 294 | 294 |
103
+ | pld | 3222 | 3222 |
104
+ | wiki | 5037 | 5037 |
105
+ | hard | 12520 | 12520 |
106
+ | basaha | 1600 | 1600 |
107
+ | tato | 1847 | 1847 |
108
+ | t_sib | 9036 | 9036 |
109
+ | t_frag | 4320 | 4320 |
110
+ | t_caps | 5388 | 5388 |
111
+ | t_bikol | 1799 | 1799 |
112
+ | t_aklsgd | 110 | 110 |
113
+ | t_tshort | 543 | 543 |
114
+
115
+ ## Throughput during this run (items/s, all workers, shared box)
116
+
117
+ crawl: 260 | goldspan: 280 | golddoc: 83 | pld: 361 | wiki: 412 | hard: 418 | basaha: 341 | tato: 416 | t_sib: 434 | t_frag: 414 | t_caps: 394 | t_bikol: 390 | t_aklsgd: 90 | t_tshort: 312
eval/evaluate.py CHANGED
@@ -1,8 +1,8 @@
1
- """Reproducible evaluation of the installed `ourlid` package (1.0.0) against GlotLID v3, OpenLID-v3 and fastText lid.176.
2
 
3
  usage: python eval/evaluate.py [--workers 4] [--skip-predict]
4
  needs: eval/out/preds/{glotlid,openlid3,lid176}__<set>.json from eval/run_baselines.py (live fastText runs)
5
- writes: eval/out/preds/ourlid-1.0.0__<set>.json, eval/RESULTS_v1.0.0.json, eval/RESULTS_v1.0.0.md
6
 
7
  Protocol (unchanged from the research loop): every set is split 50/50 by md5 of a group key into dev (all tuning
8
  decisions were taken there) and sealed (never used for a decision). Headline = sealed halves. "corrected" gold fixes known
@@ -15,15 +15,15 @@ from multiprocessing import get_context
15
  import numpy as np
16
  HERE = os.path.dirname(os.path.abspath(__file__)); sys.path.insert(0, HERE)
17
  from sets_v1 import CORE, TRUST, PUBLIC, available, rows, ok, part_of
18
- PRED = os.path.join(HERE, "out", "preds"); TAG = "ourlid-1.0.0"; BASE = ["glotlid", "openlid3", "lid176"]
19
  CBKM = set(json.load(open(os.path.join(HERE, "data", "cbk_markers.json"))))
20
  _P = _G = None
21
 
22
 
23
  def _init():
24
  global _P, _G
25
- import ourlid
26
- from ourlid.gate import Gate
27
  _P = ourlid.load(rules=True); _G = Gate(ourlid.model_dir())
28
 
29
 
@@ -36,7 +36,7 @@ def _work(batch):
36
 
37
 
38
  def predict(workers, resume=False):
39
- import ourlid
40
  os.makedirs(PRED, exist_ok=True); speed = {}
41
  for n in available():
42
  if resume and os.path.exists(f"{PRED}/{TAG}__{n}.json"): continue
@@ -118,12 +118,12 @@ def main():
118
  p = f"/workspace/ourlid/eval/preds/c7r__{n}.json"
119
  A, B = load(TAG, n), (json.load(open(p))["pred"] if os.path.exists(p) else None)
120
  if A and B: res["consistency_vs_research_c7r"][n] = {"n": len(A), "same_label": sum(A[i][0] == B[i][0] for i in A if i in B)}
121
- json.dump(res, open(os.path.join(HERE, "RESULTS_v1.0.0.json"), "w"), indent=1, default=float)
122
  write_md(res, tools)
123
 
124
 
125
  def write_md(res, tools):
126
- L = ["# OurLID 1.0.0 evaluation (generated by eval/evaluate.py; do not edit by hand)", ""]
127
  S = res["core"]["sealed"]; sets = list(S)
128
  L += ["## Sealed halves (never used for any decision): accuracy raw / corrected gold (macro-F1 corrected)", "",
129
  "| system | " + " | ".join(f"{n}{'*' if S[n]['public'] else ''} (n={S[n]['n']})" for n in sets) + " | mean corrected |", "|---|" + "---|" * (len(sets) + 1)]
@@ -135,7 +135,7 @@ def write_md(res, tools):
135
  cells.append(f"{c['acc_raw']:.3f} / {c['acc_corrected']:.3f} ({c['macro_f1_corrected']:.2f})"); ms.append(c["acc_corrected"])
136
  L.append(f"| {t} | " + " | ".join(cells) + f" | **{np.mean(ms):.3f}** |")
137
  L += ["", "\\* public set (ships in eval/data, anyone can re-run). Others are private (news gold, crawl realism, PLD read speech, BasahaCorpus stories).", "",
138
- "## OurLID 1.0.0 minus GlotLID v3, sealed, corrected gold (paired bootstrap, 2,000 resamples)", "", "| set | delta | 95% CI |", "|---|---|---|"]
139
  for n, d in res["paired_vs_glotlid"].items(): L.append(f"| {n} | {d['delta']:+.3f} | [{d['ci95'][0]:+.3f}, {d['ci95'][1]:+.3f}] |")
140
  L += ["", "## Gate v2 on sealed halves (accept = accept / accept_eng / accept_ph_other)", "", "| set | coverage | precision of accepted (corrected gold) |", "|---|---|---|"]
141
  for n, d in res["gate_sealed"].items(): L.append(f"| {n} | {d['coverage']:.3f} | {d['precision_accepted_corrected']:.3f} |")
@@ -150,7 +150,7 @@ def write_md(res, tools):
150
  for n, d in res["consistency_vs_research_c7r"].items(): L.append(f"| {n} | {d['n']} | {d['same_label']} |")
151
  if res["speed_items_per_s"]:
152
  L += ["", "## Throughput during this run (items/s, all workers, shared box)", "", " | ".join(f"{n}: {v:.0f}" for n, v in res["speed_items_per_s"].items())]
153
- open(os.path.join(HERE, "RESULTS_v1.0.0.md"), "w").write("\n".join(L) + "\n")
154
 
155
 
156
  if __name__ == "__main__":
 
1
+ """Reproducible evaluation of the installed `wikalid` package (1.0.1) against GlotLID v3, OpenLID-v3 and fastText lid.176.
2
 
3
  usage: python eval/evaluate.py [--workers 4] [--skip-predict]
4
  needs: eval/out/preds/{glotlid,openlid3,lid176}__<set>.json from eval/run_baselines.py (live fastText runs)
5
+ writes: eval/out/preds/wikalid-1.0.1__<set>.json, eval/RESULTS_v1.0.1.json, eval/RESULTS_v1.0.1.md
6
 
7
  Protocol (unchanged from the research loop): every set is split 50/50 by md5 of a group key into dev (all tuning
8
  decisions were taken there) and sealed (never used for a decision). Headline = sealed halves. "corrected" gold fixes known
 
15
  import numpy as np
16
  HERE = os.path.dirname(os.path.abspath(__file__)); sys.path.insert(0, HERE)
17
  from sets_v1 import CORE, TRUST, PUBLIC, available, rows, ok, part_of
18
+ PRED = os.path.join(HERE, "out", "preds"); TAG = "wikalid-1.0.1"; BASE = ["glotlid", "openlid3", "lid176"]
19
  CBKM = set(json.load(open(os.path.join(HERE, "data", "cbk_markers.json"))))
20
  _P = _G = None
21
 
22
 
23
  def _init():
24
  global _P, _G
25
+ import wikalid as ourlid
26
+ from wikalid.gate import Gate
27
  _P = ourlid.load(rules=True); _G = Gate(ourlid.model_dir())
28
 
29
 
 
36
 
37
 
38
  def predict(workers, resume=False):
39
+ import wikalid as ourlid
40
  os.makedirs(PRED, exist_ok=True); speed = {}
41
  for n in available():
42
  if resume and os.path.exists(f"{PRED}/{TAG}__{n}.json"): continue
 
118
  p = f"/workspace/ourlid/eval/preds/c7r__{n}.json"
119
  A, B = load(TAG, n), (json.load(open(p))["pred"] if os.path.exists(p) else None)
120
  if A and B: res["consistency_vs_research_c7r"][n] = {"n": len(A), "same_label": sum(A[i][0] == B[i][0] for i in A if i in B)}
121
+ json.dump(res, open(os.path.join(HERE, "RESULTS_v1.0.1.json"), "w"), indent=1, default=float)
122
  write_md(res, tools)
123
 
124
 
125
  def write_md(res, tools):
126
+ L = ["# WikaLID 1.0.1 evaluation (generated by eval/evaluate.py; do not edit by hand)", ""]
127
  S = res["core"]["sealed"]; sets = list(S)
128
  L += ["## Sealed halves (never used for any decision): accuracy raw / corrected gold (macro-F1 corrected)", "",
129
  "| system | " + " | ".join(f"{n}{'*' if S[n]['public'] else ''} (n={S[n]['n']})" for n in sets) + " | mean corrected |", "|---|" + "---|" * (len(sets) + 1)]
 
135
  cells.append(f"{c['acc_raw']:.3f} / {c['acc_corrected']:.3f} ({c['macro_f1_corrected']:.2f})"); ms.append(c["acc_corrected"])
136
  L.append(f"| {t} | " + " | ".join(cells) + f" | **{np.mean(ms):.3f}** |")
137
  L += ["", "\\* public set (ships in eval/data, anyone can re-run). Others are private (news gold, crawl realism, PLD read speech, BasahaCorpus stories).", "",
138
+ "## WikaLID 1.0.1 minus GlotLID v3, sealed, corrected gold (paired bootstrap, 2,000 resamples)", "", "| set | delta | 95% CI |", "|---|---|---|"]
139
  for n, d in res["paired_vs_glotlid"].items(): L.append(f"| {n} | {d['delta']:+.3f} | [{d['ci95'][0]:+.3f}, {d['ci95'][1]:+.3f}] |")
140
  L += ["", "## Gate v2 on sealed halves (accept = accept / accept_eng / accept_ph_other)", "", "| set | coverage | precision of accepted (corrected gold) |", "|---|---|---|"]
141
  for n, d in res["gate_sealed"].items(): L.append(f"| {n} | {d['coverage']:.3f} | {d['precision_accepted_corrected']:.3f} |")
 
150
  for n, d in res["consistency_vs_research_c7r"].items(): L.append(f"| {n} | {d['n']} | {d['same_label']} |")
151
  if res["speed_items_per_s"]:
152
  L += ["", "## Throughput during this run (items/s, all workers, shared box)", "", " | ".join(f"{n}: {v:.0f}" for n, v in res["speed_items_per_s"].items())]
153
+ open(os.path.join(HERE, "RESULTS_v1.0.1.md"), "w").write("\n".join(L) + "\n")
154
 
155
 
156
  if __name__ == "__main__":
eval/parity_himay.py CHANGED
@@ -1,4 +1,4 @@
1
- """Parity check: ourlid 1.0.0 (installed package) vs the sidecar that labels Himay (built by the research-repo c7 scripts).
2
  usage: parity_himay.py <documents.parquet> <shipped sidecar.parquet> <workdir> [n=3000] [workers=3]
3
  Samples n docs (seed 20261115; every non-tgl label is over-sampled), runs `ourlid sidecar`, compares every column."""
4
  import json, os, subprocess, sys
@@ -16,7 +16,7 @@ for i in range(f.num_row_groups):
16
  sub = pa.concat_tables(parts); src = os.path.join(wd, "parity_docs.parquet"); pq.write_table(sub, src, row_group_size=250)
17
  out = os.path.join(wd, "parity_sidecar.parquet")
18
  if os.path.exists(out): os.remove(out)
19
- exe = os.path.join(os.path.dirname(sys.executable), "ourlid")
20
  subprocess.run([exe, "sidecar", src, out, "--workers", workers], check=True, stdout=subprocess.DEVNULL)
21
  N = pq.read_table(out).to_pandas().set_index("document_id"); O = S.filter(pc.is_in(S.column("document_id"), pa.array(sorted(pick)))).to_pandas().set_index("document_id").loc[N.index]
22
  rep = {"n_docs": len(N), "labels_sampled": O.ourlid_label_final.value_counts().to_dict(), "columns": {}}
 
1
+ """Parity check: wikalid (installed package; formerly ourlid) vs the sidecar that labels Himay (built by the research-repo c7 scripts).
2
  usage: parity_himay.py <documents.parquet> <shipped sidecar.parquet> <workdir> [n=3000] [workers=3]
3
  Samples n docs (seed 20261115; every non-tgl label is over-sampled), runs `ourlid sidecar`, compares every column."""
4
  import json, os, subprocess, sys
 
16
  sub = pa.concat_tables(parts); src = os.path.join(wd, "parity_docs.parquet"); pq.write_table(sub, src, row_group_size=250)
17
  out = os.path.join(wd, "parity_sidecar.parquet")
18
  if os.path.exists(out): os.remove(out)
19
+ exe = os.path.join(os.path.dirname(sys.executable), "wikalid")
20
  subprocess.run([exe, "sidecar", src, out, "--workers", workers], check=True, stdout=subprocess.DEVNULL)
21
  N = pq.read_table(out).to_pandas().set_index("document_id"); O = S.filter(pc.is_in(S.column("document_id"), pa.array(sorted(pick)))).to_pandas().set_index("document_id").loc[N.index]
22
  rep = {"n_docs": len(N), "labels_sampled": O.ourlid_label_final.value_counts().to_dict(), "columns": {}}
pyproject.toml CHANGED
@@ -3,9 +3,9 @@ requires = ["setuptools>=77", "wheel"]
3
  build-backend = "setuptools.build_meta"
4
 
5
  [project]
6
- name = "ourlid"
7
- version = "1.0.0"
8
- description = "OurLID: mixture-aware, calibrated language identification for Philippine languages (Tagalog/Taglish, Cebuano, Hiligaynon, Waray, Ilocano, Bikol, Kapampangan, Pangasinan, Chavacano, ...)"
9
  readme = "README.md"
10
  requires-python = ">=3.10"
11
  license = "Apache-2.0"
@@ -17,10 +17,11 @@ parquet = ["pyarrow>=15"]
17
  test = ["pytest", "pyarrow>=15"]
18
 
19
  [project.scripts]
20
- ourlid = "ourlid.cli:main"
 
21
 
22
  [tool.setuptools.packages.find]
23
  where = ["src"]
24
 
25
  [tool.setuptools.package-data]
26
- ourlid = ["cbk_vs_spa.json", "model/*"]
 
3
  build-backend = "setuptools.build_meta"
4
 
5
  [project]
6
+ name = "wikalid"
7
+ version = "1.0.1"
8
+ description = "WikaLID: mixture-aware, calibrated language identification for Philippine languages (Tagalog/Taglish, Cebuano, Hiligaynon, Waray, Ilocano, Bikol, Kapampangan, Pangasinan, Chavacano, ...)"
9
  readme = "README.md"
10
  requires-python = ">=3.10"
11
  license = "Apache-2.0"
 
17
  test = ["pytest", "pyarrow>=15"]
18
 
19
  [project.scripts]
20
+ wikalid = "wikalid.cli:main"
21
+ ourlid = "wikalid.cli:main" # former name, kept as an alias
22
 
23
  [tool.setuptools.packages.find]
24
  where = ["src"]
25
 
26
  [tool.setuptools.package-data]
27
+ wikalid = ["cbk_vs_spa.json", "model/*"]
src/ourlid/__init__.py CHANGED
@@ -1,50 +1,11 @@
1
- """OurLID 1.0: open, mixture-aware, calibrated language identification for Philippine text.
 
 
2
 
3
- import ourlid
4
- lid = ourlid.load() # bundled model ourlid-1.0.0 (c7 two-tagger ensemble + head + rules)
5
- lid.analyze("Nag-submit na ako ng report pero hindi pa daw na-approve.")
6
- ourlid.classify("Indi ko mahibal-an kon ngaa wala sila nagkadto.") # analysis + gate decision
7
 
8
- Numpy-only inference, deterministic, CPU. See MODEL_CARD.md for labels, benchmarks and known limits.
9
- """
10
- from __future__ import annotations
11
- import os
12
-
13
- __version__ = "1.0.0"
14
- MODEL_ID = "ourlid-1.0.0"
15
- GATE_ID = "gate_v2"
16
- TOOL_ID = f"{MODEL_ID}+{GATE_ID}" # value written to Himay's `lang_tool` column
17
- LABELS = ["tgl", "ceb", "hil", "war", "ilo", "bcl", "pam", "pag", "cbk", "krj", "tsg", "eng", "spa", "ind", "msa", "oth", "nol"]
18
-
19
-
20
- def model_dir() -> str:
21
- """Directory of the bundled ourlid-1.0.0 weights (override with env OURLID_MODEL)."""
22
- return os.environ.get("OURLID_MODEL") or os.path.join(os.path.dirname(os.path.abspath(__file__)), "model")
23
-
24
-
25
- def load(model: str | None = None, alpha: str = "0.1", rules: bool = True):
26
- """-> predictor with .analyze(text, max_tokens=None) -> dict.
27
-
28
- alpha: conformal level for `label_set`/`abstain` ("0.01" | "0.05" | "0.1"; 0.1 is the shipped default).
29
- rules: apply the unsupported-Philippine-language marker bucket (Aklanon/Surigaonon -> label 'oth', key 'rule')."""
30
- import json
31
- from .r3 import PredictorR3
32
- md = model or model_dir()
33
- opts = json.load(open(os.path.join(md, "r3opts.json")))
34
- opts.pop("alpha", None)
35
- return PredictorR3(md, alpha, rules=3.0 if rules else None, **opts)
36
-
37
-
38
- _DEFAULT = {}
39
-
40
-
41
- def classify(text: str, model: str | None = None) -> dict:
42
- """Analysis + gate v2 decision for one text: adds gate ('accept'|'accept_eng'|'accept_ph_other'|'review'|'junk'),
43
- gate_reason, gate_p_correct, label_final, oth_sub. Same decision as the batch sidecar."""
44
- from .gate import Gate
45
- key = model or model_dir()
46
- if key not in _DEFAULT:
47
- _DEFAULT[key] = (load(key, rules=False), Gate(key))
48
- P, G = _DEFAULT[key]
49
- a = P.analyze(text)
50
- return dict(a, **G.decide_one(text, a))
 
1
+ """Compatibility alias: `ourlid` is the former name of WikaLID. `import ourlid` and `ourlid.<submodule>` resolve to wikalid."""
2
+ import importlib as _il
3
+ import sys as _sys
4
 
5
+ import wikalid as _w
6
+ from wikalid import * # noqa: F401,F403
7
+ from wikalid import __version__, MODEL_ID, GATE_ID, TOOL_ID, LABELS, model_dir, load, classify # noqa: F401
 
8
 
9
+ for _m in ("cli", "doc", "features", "gate", "head", "infer", "lexicon", "predictor", "r3", "rules", "sidecar", "synth", "textutil"):
10
+ _sys.modules[f"{__name__}.{_m}"] = _il.import_module(f"wikalid.{_m}")
11
+ globals()[_m] = _sys.modules[f"{__name__}.{_m}"]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
src/ourlid/__main__.py CHANGED
@@ -1,2 +1,2 @@
1
- from .cli import main
2
  main()
 
1
+ from wikalid.cli import main
2
  main()
src/wikalid/__init__.py ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """WikaLID: mixture-aware, calibrated language identification for Philippine languages ("wika" = language).
2
+
3
+ import wikalid
4
+ lid = wikalid.load() # bundled model (two-tagger ensemble + head + marker rules)
5
+ lid.analyze("Nag-submit na ako ng report pero hindi pa daw na-approve.")
6
+ wikalid.classify("Indi ko mahibal-an kon ngaa wala sila nagkadto.") # analysis + gate decision
7
+
8
+ Numpy-only inference, deterministic, CPU. Formerly released as OurLID 1.0.0 (same weights); `import ourlid` still works.
9
+ """
10
+ from __future__ import annotations
11
+ import os
12
+
13
+ __version__ = "1.0.1"
14
+ MODEL_ID = "wikalid-1.0.1"
15
+ GATE_ID = "gate_v2"
16
+ TOOL_ID = f"{MODEL_ID}+{GATE_ID}" # value written to Himay's `lang_tool` column
17
+ LABELS = ["tgl", "ceb", "hil", "war", "ilo", "bcl", "pam", "pag", "cbk", "krj", "tsg", "eng", "spa", "ind", "msa", "oth", "nol"]
18
+
19
+
20
+ def model_dir() -> str:
21
+ """Directory of the bundled weights (override with env WIKALID_MODEL; OURLID_MODEL is still honoured)."""
22
+ return os.environ.get("WIKALID_MODEL") or os.environ.get("OURLID_MODEL") or os.path.join(os.path.dirname(os.path.abspath(__file__)), "model")
23
+
24
+
25
+ def load(model: str | None = None, alpha: str = "0.1", rules: bool = True):
26
+ """-> predictor with .analyze(text, max_tokens=None) -> dict.
27
+
28
+ alpha: conformal level for `label_set`/`abstain` ("0.01" | "0.05" | "0.1"; 0.1 is the shipped default).
29
+ rules: apply the unsupported-Philippine-language marker bucket (Aklanon/Surigaonon -> label 'oth', key 'rule')."""
30
+ import json
31
+ from .r3 import PredictorR3
32
+ md = model or model_dir()
33
+ opts = json.load(open(os.path.join(md, "r3opts.json")))
34
+ opts.pop("alpha", None)
35
+ return PredictorR3(md, alpha, rules=3.0 if rules else None, **opts)
36
+
37
+
38
+ _DEFAULT = {}
39
+
40
+
41
+ def classify(text: str, model: str | None = None) -> dict:
42
+ """Analysis + gate v2 decision for one text: adds gate ('accept'|'accept_eng'|'accept_ph_other'|'review'|'junk'),
43
+ gate_reason, gate_p_correct, label_final, oth_sub. Same decision as the batch sidecar."""
44
+ from .gate import Gate
45
+ key = model or model_dir()
46
+ if key not in _DEFAULT:
47
+ _DEFAULT[key] = (load(key, rules=False), Gate(key))
48
+ P, G = _DEFAULT[key]
49
+ a = P.analyze(text)
50
+ return dict(a, **G.decide_one(text, a))
src/wikalid/__main__.py ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ from .cli import main
2
+ main()
src/{ourlid → wikalid}/cbk_vs_spa.json RENAMED
File without changes
src/{ourlid → wikalid}/cli.py RENAMED
@@ -1,10 +1,10 @@
1
- """`ourlid` command line.
2
 
3
- ourlid predict "text" ... [--jsonl FILE|-] [--alpha 0.1] [--no-rules] [--gate] -> one JSON line per text
4
- ourlid score DOCS.parquet RAW.parquet [--workers N] [--max-tokens K] -> model columns per document
5
- ourlid gate RAW.parquet DOCS.parquet OUT.parquet -> + gate v2 columns
6
- ourlid sidecar DOCS.parquet OUT.parquet [--workers N] -> score + gate
7
- ourlid info | ourlid verify -> version, labels, weight hashes
8
  """
9
  from __future__ import annotations
10
  import argparse, hashlib, json, os, sys
@@ -20,9 +20,9 @@ def _sha(p):
20
 
21
  def main(argv=None):
22
  from . import __version__, MODEL_ID, TOOL_ID, LABELS, model_dir
23
- ap = argparse.ArgumentParser(prog="ourlid", description=f"OurLID {__version__}: Philippine-language identification")
24
- ap.add_argument("--version", action="version", version=f"ourlid {__version__} ({MODEL_ID})")
25
- ap.add_argument("--model", default=None, help="model directory (default: bundled ourlid-1.0.0 or $OURLID_MODEL)")
26
  sub = ap.add_subparsers(dest="cmd", required=True)
27
  p = sub.add_parser("predict", help="identify the language of texts (args, --jsonl file, or stdin lines)")
28
  p.add_argument("text", nargs="*"); p.add_argument("--jsonl"); p.add_argument("--alpha", default="0.1", choices=["0.01", "0.05", "0.1"])
 
1
+ """`wikalid` command line (`ourlid` is kept as an alias).
2
 
3
+ wikalid predict "text" ... [--jsonl FILE|-] [--alpha 0.1] [--no-rules] [--gate] -> one JSON line per text
4
+ wikalid score DOCS.parquet RAW.parquet [--workers N] [--max-tokens K] -> model columns per document
5
+ wikalid gate RAW.parquet DOCS.parquet OUT.parquet -> + gate v2 columns
6
+ wikalid sidecar DOCS.parquet OUT.parquet [--workers N] -> score + gate
7
+ wikalid info | wikalid verify -> version, labels, weight hashes
8
  """
9
  from __future__ import annotations
10
  import argparse, hashlib, json, os, sys
 
20
 
21
  def main(argv=None):
22
  from . import __version__, MODEL_ID, TOOL_ID, LABELS, model_dir
23
+ ap = argparse.ArgumentParser(prog=os.path.basename(sys.argv[0]) if os.path.basename(sys.argv[0]) in ("wikalid", "ourlid") else "wikalid", description=f"WikaLID {__version__}: Philippine-language identification")
24
+ ap.add_argument("--version", action="version", version=f"wikalid {__version__} ({MODEL_ID})")
25
+ ap.add_argument("--model", default=None, help="model directory (default: bundled weights, or $WIKALID_MODEL)")
26
  sub = ap.add_subparsers(dest="cmd", required=True)
27
  p = sub.add_parser("predict", help="identify the language of texts (args, --jsonl file, or stdin lines)")
28
  p.add_argument("text", nargs="*"); p.add_argument("--jsonl"); p.add_argument("--alpha", default="0.1", choices=["0.01", "0.05", "0.1"])
src/{ourlid → wikalid}/doc.py RENAMED
File without changes
src/{ourlid → wikalid}/features.py RENAMED
File without changes
src/{ourlid → wikalid}/gate.py RENAMED
@@ -1,4 +1,4 @@
1
- """Gate v2: named rules + a small gradient-boosted tree head over OurLID outputs -> accept / review / junk.
2
 
3
  The tree head was fit with scikit-learn (HistGradientBoostingClassifier, dev halves only) and exported to plain arrays
4
  (`model/gate_head.npz`), so inference needs numpy only and never unpickles anything. `tests/test_gate_parity.py` checks the
 
1
+ """Gate v2: named rules + a small gradient-boosted tree head over WikaLID outputs -> accept / review / junk.
2
 
3
  The tree head was fit with scikit-learn (HistGradientBoostingClassifier, dev halves only) and exported to plain arrays
4
  (`model/gate_head.npz`), so inference needs numpy only and never unpickles anything. `tests/test_gate_parity.py` checks the
src/{ourlid → wikalid}/head.py RENAMED
File without changes
src/{ourlid → wikalid}/infer.py RENAMED
File without changes
src/{ourlid → wikalid}/lexicon.py RENAMED
File without changes
src/{ourlid → wikalid}/model/MANIFEST.json RENAMED
@@ -1,5 +1,5 @@
1
  {
2
- "model": "ourlid-1.0.0",
3
  "built_from": "research repo /workspace/ourlid @ 680798d, models/ourlid-c7 (tok_c7a + tok_c7b) + eval/gate_v2_head.pkl exported to gate_head.npz",
4
  "files": {
5
  "conformal.json": "c389f66562e1e74bf6675fc3f3b81a08fa85040a1fdff4f3ac1a561b140f2bb0",
@@ -13,5 +13,6 @@
13
  "model_int8_b.npz": "d3bb7d051f9576266b00b24551d99698f9e1c5d0bd497d415a9f0a3666c0b2d0",
14
  "r3opts.json": "c7ea93a8c6e1fef34be26f812367ea02773fe22f97d57d743de114222a980b7d",
15
  "train_cfg.json": "6442e9af7754497da62cec822c5884993eba601ec2110a519e7cf43b61a2b613"
16
- }
 
17
  }
 
1
  {
2
+ "model": "wikalid-1.0.1",
3
  "built_from": "research repo /workspace/ourlid @ 680798d, models/ourlid-c7 (tok_c7a + tok_c7b) + eval/gate_v2_head.pkl exported to gate_head.npz",
4
  "files": {
5
  "conformal.json": "c389f66562e1e74bf6675fc3f3b81a08fa85040a1fdff4f3ac1a561b140f2bb0",
 
13
  "model_int8_b.npz": "d3bb7d051f9576266b00b24551d99698f9e1c5d0bd497d415a9f0a3666c0b2d0",
14
  "r3opts.json": "c7ea93a8c6e1fef34be26f812367ea02773fe22f97d57d743de114222a980b7d",
15
  "train_cfg.json": "6442e9af7754497da62cec822c5884993eba601ec2110a519e7cf43b61a2b613"
16
+ },
17
+ "note": "Weights identical to OurLID 1.0.0 (renamed to WikaLID in 1.0.1)."
18
  }
src/{ourlid → wikalid}/model/conformal.json RENAMED
File without changes
src/{ourlid → wikalid}/model/ensemble.txt RENAMED
File without changes
src/{ourlid → wikalid}/model/gate_head.npz RENAMED
File without changes
src/{ourlid → wikalid}/model/gate_thresholds.json RENAMED
File without changes
src/{ourlid → wikalid}/model/head.npz RENAMED
File without changes
src/{ourlid → wikalid}/model/head_report.json RENAMED
File without changes
src/{ourlid → wikalid}/model/lexicon.json RENAMED
File without changes
src/{ourlid → wikalid}/model/model_int8.npz RENAMED
File without changes
src/{ourlid → wikalid}/model/model_int8_b.npz RENAMED
File without changes
src/{ourlid → wikalid}/model/r3opts.json RENAMED
File without changes
src/{ourlid → wikalid}/model/train_cfg.json RENAMED
File without changes
src/{ourlid → wikalid}/predictor.py RENAMED
File without changes
src/{ourlid → wikalid}/r3.py RENAMED
File without changes
src/{ourlid → wikalid}/rules.py RENAMED
File without changes
src/{ourlid → wikalid}/sidecar.py RENAMED
@@ -1,8 +1,8 @@
1
  """Batch sidecar for datasets: score a documents parquet (columns document_id, source_name, text) and apply gate v2.
2
 
3
- ourlid score docs.parquet raw.parquet --workers 3 # model columns (ourlid_label, ourlid_conf, ...), resumable
4
- ourlid gate raw.parquet docs.parquet gated.parquet # + ourlid_label_final, ourlid_oth_sub, gate_v2, gate_v2_reason, gate_v2_p_correct
5
- ourlid sidecar docs.parquet gated.parquet # both
6
 
7
  Column names and values are identical to the research-repo scripts (scripts/run_release.py + scripts/gate_v2_apply.py)
8
  that built every Himay sidecar since v0.2.9.5; `phc release lid` validates the result against the release."""
 
1
  """Batch sidecar for datasets: score a documents parquet (columns document_id, source_name, text) and apply gate v2.
2
 
3
+ wikalid score docs.parquet raw.parquet --workers 3 # model columns (ourlid_label, ourlid_conf, ...), resumable
4
+ wikalid gate raw.parquet docs.parquet gated.parquet # + ourlid_label_final, ourlid_oth_sub, gate_v2, gate_v2_reason, gate_v2_p_correct
5
+ wikalid sidecar docs.parquet gated.parquet # both
6
 
7
  Column names and values are identical to the research-repo scripts (scripts/run_release.py + scripts/gate_v2_apply.py)
8
  that built every Himay sidecar since v0.2.9.5; `phc release lid` validates the result against the release."""
src/{ourlid → wikalid}/synth.py RENAMED
File without changes
src/{ourlid → wikalid}/textutil.py RENAMED
File without changes
tests/test_package.py CHANGED
@@ -1,9 +1,9 @@
1
- """OurLID 1.0.0 package tests: version pin, weight manifest, labels on known texts, gate decisions, determinism, sidecar."""
2
  import json, os, subprocess, sys
3
  import numpy as np
4
  import pytest
5
- import ourlid
6
- from ourlid.gate import Gate, TreeHead
7
 
8
  EX = {"tgl": "Nag-submit na ako ng report pero hindi pa daw na-approve ng boss namin kahapon.",
9
  "hil": "Indi ko mahibal-an kon ngaa wala sila nagkadto sa balay ni Lola sang Domingo.",
@@ -14,18 +14,18 @@ EX = {"tgl": "Nag-submit na ako ng report pero hindi pa daw na-approve ng boss n
14
 
15
  @pytest.fixture(scope="module")
16
  def P():
17
- return ourlid.load()
18
 
19
 
20
  def test_version_and_labels():
21
- assert ourlid.__version__ == "1.0.0" and ourlid.TOOL_ID == "ourlid-1.0.0+gate_v2"
22
- assert len(ourlid.LABELS) == 17 and "nol" in ourlid.LABELS
23
 
24
 
25
  def test_manifest_matches_bundled_weights():
26
- r = subprocess.run([sys.executable, "-m", "ourlid", "verify", "--expect-version", "1.0.0"], capture_output=True, text=True)
27
  assert r.returncode == 0, r.stdout + r.stderr
28
- bad = subprocess.run([sys.executable, "-m", "ourlid", "verify", "--expect-version", "9.9.9"], capture_output=True, text=True)
29
  assert bad.returncode == 1
30
 
31
 
@@ -41,19 +41,19 @@ def test_taglish_is_a_statistic_not_a_class(P):
41
 
42
 
43
  def test_deterministic(P):
44
- a, b = P.analyze(EX["tgl"]), ourlid.load().analyze(EX["tgl"])
45
  assert a == b
46
 
47
 
48
  def test_classify_gate_fields():
49
- r = ourlid.classify(EX["tgl"])
50
  assert r["gate"] == "accept" and r["gate_reason"] == "accepted" and 0 < r["gate_p_correct"] <= 1 and r["label_final"] == "tgl"
51
- j = ourlid.classify("1. Juan Dela Cruz 2. Pedro Penduko 3. Maria Clara 4. Jose Rizal")
52
  assert j["gate"] in ("junk", "review")
53
 
54
 
55
  def test_aklanon_rule_bucket():
56
- r = ourlid.classify("Ro Altavas, a opisyal nga Banwa it Altavas, hay sangka banwa sa probinsya it Aklan. Eon ro mayad-ayad nga baeay.")
57
  assert r["label_final"] == "oth" and r["oth_sub"] == "akl" and r["gate"] == "accept_ph_other"
58
 
59
 
@@ -63,19 +63,28 @@ def test_tree_head_matches_sklearn_when_available():
63
  if not os.path.exists(pkl):
64
  pytest.skip("research gate pickle not on this machine")
65
  import pickle
66
- h = pickle.load(open(pkl, "rb")); T = TreeHead(os.path.join(ourlid.model_dir(), "gate_head.npz"))
67
  X = np.random.default_rng(0).random((2000, 20)) * 5; X[::40, 5] = np.nan
68
  assert (h.predict_proba(X)[:, 1] == T.p_correct(X)).all()
69
 
70
 
71
  def test_sidecar_roundtrip(tmp_path):
72
  pa = pytest.importorskip("pyarrow"); import pyarrow.parquet as pq
73
- from ourlid.sidecar import sidecar
74
  ids = sorted(EX)
75
  pq.write_table(pa.table({"document_id": ids, "source_name": ["s"] * len(ids), "text": [EX[i] for i in ids]}), tmp_path / "d.parquet")
76
  rep = sidecar(str(tmp_path / "d.parquet"), str(tmp_path / "o.parquet"), workers=1, log=lambda *a: None)
77
  t = pq.read_table(tmp_path / "o.parquet")
78
  assert t.column("ourlid_label_final").to_pylist() == ids
79
- assert t.schema.metadata[b"ourlid_version"] == b"1.0.0" and sum(rep["gate"].values()) == len(ids)
80
  for c in ("gate_v2", "gate_v2_reason", "gate_v2_p_correct", "ourlid_oth_sub", "text_sha256"):
81
  assert c in t.column_names
 
 
 
 
 
 
 
 
 
 
1
+ """WikaLID 1.0.1 package tests: version pin, weight manifest, labels on known texts, gate decisions, determinism, sidecar."""
2
  import json, os, subprocess, sys
3
  import numpy as np
4
  import pytest
5
+ import wikalid
6
+ from wikalid.gate import Gate, TreeHead
7
 
8
  EX = {"tgl": "Nag-submit na ako ng report pero hindi pa daw na-approve ng boss namin kahapon.",
9
  "hil": "Indi ko mahibal-an kon ngaa wala sila nagkadto sa balay ni Lola sang Domingo.",
 
14
 
15
  @pytest.fixture(scope="module")
16
  def P():
17
+ return wikalid.load()
18
 
19
 
20
  def test_version_and_labels():
21
+ assert wikalid.__version__ == "1.0.1" and wikalid.TOOL_ID == "wikalid-1.0.1+gate_v2"
22
+ assert len(wikalid.LABELS) == 17 and "nol" in wikalid.LABELS
23
 
24
 
25
  def test_manifest_matches_bundled_weights():
26
+ r = subprocess.run([sys.executable, "-m", "wikalid", "verify", "--expect-version", "1.0.1"], capture_output=True, text=True)
27
  assert r.returncode == 0, r.stdout + r.stderr
28
+ bad = subprocess.run([sys.executable, "-m", "wikalid", "verify", "--expect-version", "9.9.9"], capture_output=True, text=True)
29
  assert bad.returncode == 1
30
 
31
 
 
41
 
42
 
43
  def test_deterministic(P):
44
+ a, b = P.analyze(EX["tgl"]), wikalid.load().analyze(EX["tgl"])
45
  assert a == b
46
 
47
 
48
  def test_classify_gate_fields():
49
+ r = wikalid.classify(EX["tgl"])
50
  assert r["gate"] == "accept" and r["gate_reason"] == "accepted" and 0 < r["gate_p_correct"] <= 1 and r["label_final"] == "tgl"
51
+ j = wikalid.classify("1. Juan Dela Cruz 2. Pedro Penduko 3. Maria Clara 4. Jose Rizal")
52
  assert j["gate"] in ("junk", "review")
53
 
54
 
55
  def test_aklanon_rule_bucket():
56
+ r = wikalid.classify("Ro Altavas, a opisyal nga Banwa it Altavas, hay sangka banwa sa probinsya it Aklan. Eon ro mayad-ayad nga baeay.")
57
  assert r["label_final"] == "oth" and r["oth_sub"] == "akl" and r["gate"] == "accept_ph_other"
58
 
59
 
 
63
  if not os.path.exists(pkl):
64
  pytest.skip("research gate pickle not on this machine")
65
  import pickle
66
+ h = pickle.load(open(pkl, "rb")); T = TreeHead(os.path.join(wikalid.model_dir(), "gate_head.npz"))
67
  X = np.random.default_rng(0).random((2000, 20)) * 5; X[::40, 5] = np.nan
68
  assert (h.predict_proba(X)[:, 1] == T.p_correct(X)).all()
69
 
70
 
71
  def test_sidecar_roundtrip(tmp_path):
72
  pa = pytest.importorskip("pyarrow"); import pyarrow.parquet as pq
73
+ from wikalid.sidecar import sidecar
74
  ids = sorted(EX)
75
  pq.write_table(pa.table({"document_id": ids, "source_name": ["s"] * len(ids), "text": [EX[i] for i in ids]}), tmp_path / "d.parquet")
76
  rep = sidecar(str(tmp_path / "d.parquet"), str(tmp_path / "o.parquet"), workers=1, log=lambda *a: None)
77
  t = pq.read_table(tmp_path / "o.parquet")
78
  assert t.column("ourlid_label_final").to_pylist() == ids
79
+ assert t.schema.metadata[b"ourlid_version"] == b"1.0.1" and sum(rep["gate"].values()) == len(ids)
80
  for c in ("gate_v2", "gate_v2_reason", "gate_v2_p_correct", "ourlid_oth_sub", "text_sha256"):
81
  assert c in t.column_names
82
+
83
+
84
+ def test_ourlid_alias_still_works():
85
+ import ourlid
86
+ from ourlid.gate import Gate as G2
87
+ from ourlid.r3 import PredictorR3 # noqa: F401
88
+ assert ourlid.__version__ == wikalid.__version__ and G2 is Gate and ourlid.classify(EX["tgl"])["label_final"] == "tgl"
89
+ r = subprocess.run([sys.executable, "-m", "ourlid", "verify", "--expect-version", "1.0.1"], capture_output=True, text=True)
90
+ assert r.returncode == 0, r.stdout + r.stderr
tests/test_rules.py CHANGED
@@ -1,5 +1,5 @@
1
  import sys
2
- from ourlid import rules
3
  AKL_POS = ["Gusto ko ro maeamig nga tubi, owa ako't kwarta.", "Ro uwa gatan-aw sa anang ginhalinan hay indi makaabut sa anang ginapaeangpan.", "Hin-uno ka gapanaw, ro baeay mo hay siin?", "Mayad-ayad nga agahon, ro ngaean ko hay Maria.", "Ro ngaean ko si Tom.", "Ro banwa hay ginabug-usan it mga barangay ag ro eugta hay mayad.", "Ro Altavas hay sangka banwa sa probinsya it Aklan.", "Guinakalipay ko ro pakipagkita kimo, imaw ro ginsiling ni sanda eon.", "Saeamat sa imo, eon ako maeakat ag imaw ro nagpauna."]
4
  SGD_POS = ["An Alegria usa ka lungsod nan Surigao del Norte sanan may gilapdon ini na 65 ka kilometro kwadrado.", "Haman ka paingon sija sanan oman an ginhimo nan lungsod?", "Gitudlo-an ini nan 86 na area code, sanan pinaka-ulahi na sensus."]
5
  NEG = ["Ang bata ay naglalaro sa labas kasama ang kanyang mga kaibigan.", "Gusto ko nga mokaon ug saging kay gutom na ko.", "Ang akon mga kaibigan nagapanguli na sa balay kag nagakaon.",
 
1
  import sys
2
+ from wikalid import rules
3
  AKL_POS = ["Gusto ko ro maeamig nga tubi, owa ako't kwarta.", "Ro uwa gatan-aw sa anang ginhalinan hay indi makaabut sa anang ginapaeangpan.", "Hin-uno ka gapanaw, ro baeay mo hay siin?", "Mayad-ayad nga agahon, ro ngaean ko hay Maria.", "Ro ngaean ko si Tom.", "Ro banwa hay ginabug-usan it mga barangay ag ro eugta hay mayad.", "Ro Altavas hay sangka banwa sa probinsya it Aklan.", "Guinakalipay ko ro pakipagkita kimo, imaw ro ginsiling ni sanda eon.", "Saeamat sa imo, eon ako maeakat ag imaw ro nagpauna."]
4
  SGD_POS = ["An Alegria usa ka lungsod nan Surigao del Norte sanan may gilapdon ini na 65 ka kilometro kwadrado.", "Haman ka paingon sija sanan oman an ginhimo nan lungsod?", "Gitudlo-an ini nan 86 na area code, sanan pinaka-ulahi na sensus."]
5
  NEG = ["Ang bata ay naglalaro sa labas kasama ang kanyang mga kaibigan.", "Gusto ko nga mokaon ug saging kay gutom na ko.", "Ang akon mga kaibigan nagapanguli na sa balay kag nagakaon.",