WikaLID 1.0.1: rename from OurLID, public model card, ourlid alias kept
Browse files- .gitattributes +1 -0
- CHANGELOG.md +6 -0
- MODEL_LICENSE.md +2 -2
- README.md +286 -110
- dist/{ourlid-1.0.0-py3-none-any.whl → wikalid-1.0.1-py3-none-any.whl} +2 -2
- eval/RESULTS_v1.0.1.json +2597 -0
- eval/RESULTS_v1.0.1.md +117 -0
- eval/evaluate.py +10 -10
- eval/parity_himay.py +2 -2
- pyproject.toml +6 -5
- src/ourlid/__init__.py +9 -48
- src/ourlid/__main__.py +1 -1
- src/wikalid/__init__.py +50 -0
- src/wikalid/__main__.py +2 -0
- src/{ourlid → wikalid}/cbk_vs_spa.json +0 -0
- src/{ourlid → wikalid}/cli.py +9 -9
- src/{ourlid → wikalid}/doc.py +0 -0
- src/{ourlid → wikalid}/features.py +0 -0
- src/{ourlid → wikalid}/gate.py +1 -1
- src/{ourlid → wikalid}/head.py +0 -0
- src/{ourlid → wikalid}/infer.py +0 -0
- src/{ourlid → wikalid}/lexicon.py +0 -0
- src/{ourlid → wikalid}/model/MANIFEST.json +3 -2
- src/{ourlid → wikalid}/model/conformal.json +0 -0
- src/{ourlid → wikalid}/model/ensemble.txt +0 -0
- src/{ourlid → wikalid}/model/gate_head.npz +0 -0
- src/{ourlid → wikalid}/model/gate_thresholds.json +0 -0
- src/{ourlid → wikalid}/model/head.npz +0 -0
- src/{ourlid → wikalid}/model/head_report.json +0 -0
- src/{ourlid → wikalid}/model/lexicon.json +0 -0
- src/{ourlid → wikalid}/model/model_int8.npz +0 -0
- src/{ourlid → wikalid}/model/model_int8_b.npz +0 -0
- src/{ourlid → wikalid}/model/r3opts.json +0 -0
- src/{ourlid → wikalid}/model/train_cfg.json +0 -0
- src/{ourlid → wikalid}/predictor.py +0 -0
- src/{ourlid → wikalid}/r3.py +0 -0
- src/{ourlid → wikalid}/rules.py +0 -0
- src/{ourlid → wikalid}/sidecar.py +3 -3
- src/{ourlid → wikalid}/synth.py +0 -0
- src/{ourlid → wikalid}/textutil.py +0 -0
- tests/test_package.py +24 -15
- tests/test_rules.py +1 -1
.gitattributes
CHANGED
|
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
dist/ourlid-1.0.0-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
dist/ourlid-1.0.0-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
dist/wikalid-1.0.1-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
|
CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
| 1 |
# Changelog
|
| 2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
## 1.0.0 — 2026-11-15
|
| 4 |
First versioned package. Same weights and decisions as the research model `c7` + gate v2 that labelled Himay
|
| 5 |
v0.2.9.5 → v0.3.0.4 (labels/gates bit-identical on a 3,000-doc Himay sample and on every eval item).
|
|
|
|
| 1 |
# Changelog
|
| 2 |
|
| 3 |
+
## 1.0.1 — 2026-11-15
|
| 4 |
+
Renamed OurLID → **WikaLID** (*wika* = "language" in Tagalog). Package and CLI are now `wikalid`; `import ourlid`,
|
| 5 |
+
`python -m ourlid` and the `ourlid` command remain as aliases. Weights, outputs and sidecar column names (`ourlid_*`)
|
| 6 |
+
are unchanged: all 17 sidecar columns bit-identical to 1.0.0 on a 3,000-doc Himay sample, and every evaluation number
|
| 7 |
+
identical (`eval/RESULTS_v1.0.1.md`). Public model card rewritten.
|
| 8 |
+
|
| 9 |
## 1.0.0 — 2026-11-15
|
| 10 |
First versioned package. Same weights and decisions as the research model `c7` + gate v2 that labelled Himay
|
| 11 |
v0.2.9.5 → v0.3.0.4 (labels/gates bit-identical on a 3,000-doc Himay sample and on every eval item).
|
MODEL_LICENSE.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
-
#
|
| 2 |
|
| 3 |
* **Code** (`src/`, `eval/*.py`, `tests/`): Apache-2.0 (`LICENSE`).
|
| 4 |
-
* **Model weights** (`src/
|
| 5 |
do not redistribute, mirror or make public. The public licence is an open decision for the owner.
|
| 6 |
Intended licence if published: **CC BY-SA 4.0** (share-alike, because Wikipedia CC BY-SA text is the largest training
|
| 7 |
source), with attribution to Wikipedia contributors, Tatoeba contributors, FineWeb-2 (ODC-By) and BasahaCorpus
|
|
|
|
| 1 |
+
# WikaLID 1.0.1 (formerly OurLID) — licence status
|
| 2 |
|
| 3 |
* **Code** (`src/`, `eval/*.py`, `tests/`): Apache-2.0 (`LICENSE`).
|
| 4 |
+
* **Model weights** (`src/wikalid/model/`): **private preview — no licence granted.** Internal use by sapinsapin only;
|
| 5 |
do not redistribute, mirror or make public. The public licence is an open decision for the owner.
|
| 6 |
Intended licence if published: **CC BY-SA 4.0** (share-alike, because Wikipedia CC BY-SA text is the largest training
|
| 7 |
source), with attribution to Wikipedia contributors, Tatoeba contributors, FineWeb-2 (ODC-By) and BasahaCorpus
|
README.md
CHANGED
|
@@ -1,138 +1,314 @@
|
|
| 1 |
---
|
| 2 |
license: other
|
| 3 |
-
license_name:
|
| 4 |
-
license_link: https://huggingface.co/sapinsapin/
|
| 5 |
-
language:
|
| 6 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
pipeline_tag: text-classification
|
| 8 |
-
tags:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
---
|
| 10 |
|
| 11 |
-
#
|
| 12 |
|
| 13 |
-
**
|
| 14 |
-
on a public licence (see `MODEL_LICENSE.md`).
|
| 15 |
|
| 16 |
-
|
| 17 |
-
Waray, Ilocano, Bikol, Kapampangan, Pangasinan, Chavacano, plus experimental Kinaray-a and Tausug — and separates them
|
| 18 |
-
from English, Spanish, Indonesian, Malay, "another language" and "not language at all" (lists, rosters, score tables).
|
| 19 |
-
It also says **how mixed** the text is (English share, `cs_level` none/light/heavy) and **how sure it is** (calibrated
|
| 20 |
-
confidence, a conformal label set, and a gate decision: accept / review / junk).
|
| 21 |
|
| 22 |
-
|
| 23 |
|
| 24 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
```bash
|
| 27 |
-
pip install
|
| 28 |
-
|
| 29 |
-
ourlid sidecar documents.parquet labels.parquet --workers 3 # columns document_id, source_name, text
|
| 30 |
-
ourlid verify --expect-version 1.0.0 # weight sha256 + version pin
|
| 31 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
```python
|
| 33 |
-
import
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
```
|
| 38 |
|
| 39 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|---|---|
|
| 41 |
-
| `label` |
|
| 42 |
-
| `
|
| 43 |
-
| `
|
| 44 |
-
| `
|
| 45 |
-
| `
|
| 46 |
-
| `
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
|
| 48 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
`oth` = any other language, including the rule bucket for Aklanon/Surigaonon. `nol` = not a language.
|
| 53 |
|
| 54 |
-
|
|
|
|
| 55 |
|
| 56 |
-
|
| 57 |
-
label every word; posteriors are averaged, smoothed by a sticky HMM over words, with extra "ALLCAPS→Title" and
|
| 58 |
-
"accent-stripped" views.
|
| 59 |
-
2. A small softmax head turns the word-level mixture plus closed-class lexicon evidence into a document label.
|
| 60 |
-
3. Split-conformal calibration gives a label set; a 150-tree gradient-boosted gate head (fit on dev halves only, exported
|
| 61 |
-
to plain arrays, no pickle) plus named rules decides accept / review / junk.
|
| 62 |
-
4. Hand-written, unit-tested marker rules move Aklanon/Surigaonon-looking text into `oth` (they never create a supported label).
|
| 63 |
|
| 64 |
-
##
|
| 65 |
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 70 |
|
| 71 |
-
|
|
|
|
|
|
|
| 72 |
|---|---|---|---|---|---|
|
| 73 |
-
|
|
| 74 |
-
|
|
| 75 |
-
|
|
| 76 |
-
| PLD read speech (1,653) |
|
| 77 |
-
| Wikipedia held-out (2,563) |
|
| 78 |
-
| PH-LID-Hard v0 (6,379) |
|
| 79 |
-
| BasahaCorpus
|
| 80 |
-
| Tatoeba test (968) |
|
| 81 |
-
| **
|
| 82 |
-
|
| 83 |
-
Paired bootstrap vs GlotLID (sealed): OurLID is better on 7 of 8 sets with 95% intervals above zero (crawl +7.5 pts,
|
| 84 |
-
PLD +8.3, PH-LID-Hard +7.2, Basaha +15.6, gold spans +4.0, gold docs +2.7, Wikipedia +1.8) and **worse on Tatoeba
|
| 85 |
-
(−2.7 pts, CI −4.6 to −0.9)** — very short, clean sentences, where GlotLID's breadth wins.
|
| 86 |
-
|
| 87 |
-
Human-labelled external sets (all rows): FLORES-200 via SIB-200, 9 languages, n=9,036: 0.989 (GlotLID 0.986, OpenLID
|
| 88 |
-
0.985). ALLCAPS/Title/lower variants: 0.999 (GlotLID 0.975). Central Bikol story sentences: Bikol recall 0.961 (GlotLID
|
| 89 |
-
0.894). 2–5-word fragments: 0.832 (GlotLID 0.771); 2 words alone 0.704.
|
| 90 |
-
|
| 91 |
-
**Gate v2.** On sealed halves, rows the gate accepts are 98.5–100% correct while covering 36–78% of these deliberately
|
| 92 |
-
hard sets. On real news (Himay parent, 240,386 docs) it accepts 88.6% and routes 11.4% to `review`.
|
| 93 |
-
|
| 94 |
-
**Speed.** ~2–7k docs/min per core depending on length (two taggers); 3 workers labelled a 3,000-article sample in ~10 s
|
| 95 |
-
end to end on a shared 8-core box.
|
| 96 |
-
|
| 97 |
-
## Known limits (read before trusting a label)
|
| 98 |
-
|
| 99 |
-
* **Surigaonon vs Cebuano, Aklanon vs Hiligaynon:** no class exists (no open training text). The marker rule catches only
|
| 100 |
-
34.5% of a templated Aklanon/Surigaonon set (CI 0.26–0.44); everything else is called `ceb`/`hil`. Recall on free text is unknown.
|
| 101 |
-
* **Hiligaynon vs Kinaray-a vs Cebuano:** improved (Basaha +15.6 pts vs GlotLID) but still the main confusion family;
|
| 102 |
-
`krj` is experimental.
|
| 103 |
-
* **`krj` / `pag` labels on news:** on Himay, sampled `krj` (3/3) and `pag` (2/2) rows were lists, Aklanon or Hiligaynon.
|
| 104 |
-
Himay forces them to `review`.
|
| 105 |
-
* **Tagalog-dominant Taglish labelled `eng`:** English-heavy showbiz/tabloid Taglish with ALLCAPS leads can come out
|
| 106 |
-
`eng`; the conformal set is then usually empty and the gate sends it to `review` (when OurLID abstains on `eng`, the
|
| 107 |
-
top label is right only ~22% of the time). Use `gate != "review"` as the clean default.
|
| 108 |
-
* **Short strings:** 2 words 0.70, 3 words 0.82; a third of the errors are Indonesian↔Malay.
|
| 109 |
-
* **Tatoeba-style short clean sentences:** GlotLID is better (see above).
|
| 110 |
-
* **Gold quality:** the news gold sets have a single, non-native annotator; native-speaker labels are pending.
|
| 111 |
-
* **Calibration:** cross-domain conformal coverage at alpha 0.10 measured 0.82–0.92 against a 0.90 target.
|
| 112 |
-
|
| 113 |
-
## Training data (none of it ships in this repo)
|
| 114 |
-
|
| 115 |
-
Wikipedia dumps (CC BY-SA 4.0), Tatoeba (CC BY 2.0 FR), FineWeb-2 web text (ODC-By 1.0; page copyright stays with
|
| 116 |
-
publishers; used for training only, never redistributed), BasahaCorpus stories (CC BY 4.0 per its authors, Imperial &
|
| 117 |
-
Kochmar 2023 / Let's Read Asia). **No scraped news and no Himay text is in the training data.** GlotLID/OpenLID/lid176
|
| 118 |
-
were used only as label-noise detectors; no third-party weights are inside OurLID. Details: `docs/DATA_CARD.md`.
|
| 119 |
-
|
| 120 |
-
What ships: code (Apache-2.0), weights, and the three public evaluation sets derived from Wikipedia/Tatoeba
|
| 121 |
-
(`eval/data`, CC BY-SA 4.0 / CC BY 2.0 FR). Private evaluation sets (news gold, crawl realism, PLD, Basaha, trust sets)
|
| 122 |
-
and the native-speaker labelling sheets are **not** included.
|
| 123 |
-
|
| 124 |
-
## Reproduce
|
| 125 |
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 131 |
```
|
| 132 |
-
Private sets are read from `$OURLID_PRIVATE_EVAL` and skipped when absent; the public three always run.
|
| 133 |
|
| 134 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 135 |
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
bit-identical to the research pipeline that labelled Himay v0.3.0.4 (`eval/PARITY_HIMAY_v1.0.0.json`).
|
|
|
|
| 1 |
---
|
| 2 |
license: other
|
| 3 |
+
license_name: wikalid-preview
|
| 4 |
+
license_link: https://huggingface.co/sapinsapin/wikalid/blob/main/MODEL_LICENSE.md
|
| 5 |
+
language:
|
| 6 |
+
- tl
|
| 7 |
+
- ceb
|
| 8 |
+
- hil
|
| 9 |
+
- war
|
| 10 |
+
- ilo
|
| 11 |
+
- bcl
|
| 12 |
+
- pam
|
| 13 |
+
- pag
|
| 14 |
+
- cbk
|
| 15 |
+
- krj
|
| 16 |
+
- tsg
|
| 17 |
+
- en
|
| 18 |
+
- es
|
| 19 |
+
- id
|
| 20 |
+
- ms
|
| 21 |
+
- multilingual
|
| 22 |
+
library_name: wikalid
|
| 23 |
pipeline_tag: text-classification
|
| 24 |
+
tags:
|
| 25 |
+
- language-identification
|
| 26 |
+
- lid
|
| 27 |
+
- philippines
|
| 28 |
+
- filipino
|
| 29 |
+
- tagalog
|
| 30 |
+
- taglish
|
| 31 |
+
- code-switching
|
| 32 |
+
- cebuano
|
| 33 |
+
- hiligaynon
|
| 34 |
+
- low-resource
|
| 35 |
+
- cpu
|
| 36 |
---
|
| 37 |
|
| 38 |
+
# WikaLID
|
| 39 |
|
| 40 |
+
**Language identification for Philippine languages, with code-switching measurement and calibrated abstention.**
|
|
|
|
| 41 |
|
| 42 |
+
*Wika* is Tagalog for "language". WikaLID was released as OurLID 1.0.0; 1.0.1 has the same weights under the new name.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 43 |
|
| 44 |
+
## Overview
|
| 45 |
|
| 46 |
+
General-purpose language identifiers such as GlotLID, OpenLID and fastText `lid.176` cover hundreds of languages, but
|
| 47 |
+
they struggle with Philippine web and news text. They mix up the Visayan languages, fail on ALLCAPS headlines and
|
| 48 |
+
tabloid style, can't separate Chavacano from Spanish, and treat Tagalog–English code-switching (Taglish) as noise.
|
| 49 |
+
WikaLID is a small, CPU-only model built for that setting:
|
| 50 |
+
|
| 51 |
+
* **17 labels**: 11 Philippine languages, 4 contact languages (English, Spanish, Indonesian, Malay), "other language"
|
| 52 |
+
and "not a language".
|
| 53 |
+
* **Token-level tagging.** Every word gets a language, so the output includes the language mixture and the English
|
| 54 |
+
share. Taglish is reported as a measurement, not a separate class.
|
| 55 |
+
* **Calibrated uncertainty.** It returns a temperature-scaled confidence and a conformal label set, plus a gate that
|
| 56 |
+
sorts each text into *accept*, *review* or *junk*.
|
| 57 |
+
* **Small and deterministic.** Two 26 MB int8 taggers, numpy-only inference, no GPU, no PyTorch, no fastText.
|
| 58 |
+
|
| 59 |
+
It was built to label a Filipino text corpus for language-model pretraining. The gate exists so that uncertain rows
|
| 60 |
+
can be held back instead of being silently mislabelled.
|
| 61 |
+
|
| 62 |
+
## Supported labels
|
| 63 |
+
|
| 64 |
+
| code | language | notes |
|
| 65 |
+
|---|---|---|
|
| 66 |
+
| `tgl` | Tagalog / Filipino | Includes Taglish. Code-switching is reported in `cs_level` / `cs_share` |
|
| 67 |
+
| `ceb` | Cebuano | |
|
| 68 |
+
| `hil` | Hiligaynon | Most frequent confusions are with `ceb` and `krj` |
|
| 69 |
+
| `war` | Waray | |
|
| 70 |
+
| `ilo` | Ilocano | |
|
| 71 |
+
| `bcl` | Bikol | Central Bikol; Rinconada is also labelled `bcl` |
|
| 72 |
+
| `pam` | Kapampangan | |
|
| 73 |
+
| `pag` | Pangasinan | Strong on formal text; rare in news, so treat news predictions with care |
|
| 74 |
+
| `cbk` | Chavacano | Separated from Spanish |
|
| 75 |
+
| `krj` | Kinaray-a | **Experimental** |
|
| 76 |
+
| `tsg` | Tausug | **Experimental**: no human-labelled free-text test set |
|
| 77 |
+
| `eng` | English | |
|
| 78 |
+
| `spa` | Spanish | |
|
| 79 |
+
| `ind` | Indonesian | Often confused with `msa` on very short strings |
|
| 80 |
+
| `msa` | Malay | |
|
| 81 |
+
| `oth` | any other language | Includes Aklanon / Surigaonon detected by marker rules (`oth_sub` = `akl` / `sgd`) |
|
| 82 |
+
| `nol` | not a language | Numbered lists, name rosters, score tables (≈ ISO `zxx`) |
|
| 83 |
+
|
| 84 |
+
## Install
|
| 85 |
+
|
| 86 |
+
WikaLID is distributed as a wheel with the weights bundled (about 52 MB). It needs Python ≥ 3.10 and numpy.
|
| 87 |
|
| 88 |
```bash
|
| 89 |
+
pip install wikalid-1.0.1-py3-none-any.whl # add pyarrow for parquet batch jobs
|
| 90 |
+
wikalid verify --expect-version 1.0.1 # checks the version and the sha256 of every weight file
|
|
|
|
|
|
|
| 91 |
```
|
| 92 |
+
|
| 93 |
+
## Quickstart
|
| 94 |
+
|
| 95 |
+
### Python
|
| 96 |
+
|
| 97 |
```python
|
| 98 |
+
import wikalid
|
| 99 |
+
|
| 100 |
+
lid = wikalid.load() # bundled model; deterministic
|
| 101 |
+
lid.analyze("Nag-submit na ako ng report pero hindi pa daw na-approve.")
|
| 102 |
+
# {'label': 'tgl', 'conf': 0.9968, 'label_set': ['tgl'], 'abstain': False,
|
| 103 |
+
# 'mixture': {'tgl': 0.9998}, 'cs_level': 'none', 'cs_share': 2.3e-06, ...}
|
| 104 |
+
|
| 105 |
+
lid.analyze("Grabe yung traffic kanina sa EDSA, super late na ako sa meeting with the client.")
|
| 106 |
+
# {'label': 'tgl', 'conf': 0.9344, 'label_set': ['tgl'], 'mixture': {'tgl': 0.7477, 'eng': 0.252},
|
| 107 |
+
# 'cs_level': 'heavy', 'cs_share': 0.252, ...}
|
| 108 |
+
|
| 109 |
+
wikalid.classify("Indi ko mahibal-an kon ngaa wala sila nagkadto.") # analysis + gate decision
|
| 110 |
+
# {..., 'label': 'hil', 'conf': 0.9993, 'label_final': 'hil', 'oth_sub': None,
|
| 111 |
+
# 'gate': 'accept_ph_other', 'gate_reason': 'accepted', 'gate_p_correct': 0.997}
|
| 112 |
```
|
| 113 |
|
| 114 |
+
### Command line
|
| 115 |
+
|
| 116 |
+
```bash
|
| 117 |
+
$ wikalid predict "Wala ko kahibalo nganong wala sila miadto."
|
| 118 |
+
{"text": "Wala ko kahibalo nganong wala sila miadto.", "label": "ceb", "conf": 0.9895, "abstain": false,
|
| 119 |
+
"label_set": ["ceb"], "mixture": {"ceb": 1.0}, "cs_level": "none", ...}
|
| 120 |
+
|
| 121 |
+
$ echo "Ano ang pangalan mo?" | wikalid predict --gate
|
| 122 |
+
{"text": "Ano ang pangalan mo?", "label": "tgl", "conf": 0.9967, ..., "gate": "review", "gate_reason": "meta_low", "gate_p_correct": 0.504}
|
| 123 |
+
|
| 124 |
+
$ wikalid sidecar docs.parquet labels.parquet --workers 4 # input columns: document_id, source_name, text
|
| 125 |
+
```
|
| 126 |
+
|
| 127 |
+
The outputs above are abridged; values are rounded from real runs of 1.0.1.
|
| 128 |
+
|
| 129 |
+
`wikalid sidecar` writes one row per document with the model columns (`ourlid_label`, `ourlid_conf`,
|
| 130 |
+
`ourlid_label_set`, `ourlid_mixture`, `ourlid_cs_share`, …) and the gate columns (`ourlid_label_final`, `ourlid_oth_sub`,
|
| 131 |
+
`gate_v2`, `gate_v2_reason`, `gate_v2_p_correct`). The `ourlid_` prefix is kept for compatibility with existing
|
| 132 |
+
consumers. `wikalid score` and `wikalid gate` run the two steps separately. `wikalid info` prints the version, labels and
|
| 133 |
+
weight hashes.
|
| 134 |
+
|
| 135 |
+
### Output fields
|
| 136 |
+
|
| 137 |
+
| field | meaning |
|
| 138 |
|---|---|
|
| 139 |
+
| `label` | Majority language of the text (standard LID convention) |
|
| 140 |
+
| `conf` | Temperature-scaled probability of `label` |
|
| 141 |
+
| `label_set`, `abstain` | Split-conformal prediction set at α = 0.10; `abstain` is true when the set is not exactly one label |
|
| 142 |
+
| `mixture` | Share of word tokens per language (languages ≥ 1%) |
|
| 143 |
+
| `cs_level`, `cs_share` | English share for non-English texts (`none` < 0.03 ≤ `light` < 0.25 ≤ `heavy`) |
|
| 144 |
+
| `label_final`, `oth_sub` | `label`, or `oth` + `akl`/`sgd` when the Aklanon/Surigaonon marker rule fires |
|
| 145 |
+
| `gate`, `gate_reason`, `gate_p_correct` | Gate decision (below) |
|
| 146 |
+
|
| 147 |
+
## The gate: accept, review, junk
|
| 148 |
+
|
| 149 |
+
A label is only useful if you know when to trust it. The gate combines named rules with a small gradient-boosted
|
| 150 |
+
"is this label correct?" model. That model was fit only on development data, so its scores are not tuned to the test
|
| 151 |
+
sets reported below.
|
| 152 |
|
| 153 |
+
| gate | meaning | example (real outputs) |
|
| 154 |
+
|---|---|---|
|
| 155 |
+
| `accept` | confident Tagalog | "Nag-submit na ako ng report…" → `tgl`, p_correct 0.995 |
|
| 156 |
+
| `accept_ph_other` | confident other Philippine language, or a marker-rule hit | "Indi ko mahibal-an…" → `hil`; an Aklanon sentence → `oth`/`akl`, reason `unsupported_ph_akl` |
|
| 157 |
+
| `accept_eng` | confident English | |
|
| 158 |
+
| `review` | the model abstained, the text is very short, a sibling language is close, or confidence is below that label's threshold | heavy Taglish → `tgl` but reason `low_conf_tgl`; "Ano ang pangalan mo?" → reason `meta_low` |
|
| 159 |
+
| `junk` | not language | a numbered name roster → `nol`, reason `junk_nol` |
|
| 160 |
|
| 161 |
+
The gate is deliberately strict. The English threshold is very high, so many clean English sentences land in `review`.
|
| 162 |
+
That suits a corpus where English is opt-in, but you may want your own thresholds for other uses.
|
|
|
|
| 163 |
|
| 164 |
+
Rows the gate accepts were 97.2–100% correct on every sealed evaluation set, and at least 98.5% correct on every set except BasahaCorpus (table below). On about 240k Filipino
|
| 165 |
+
news articles, it accepted 88.6% and sent 11.4% to review.
|
| 166 |
|
| 167 |
+
## Evaluation
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 168 |
|
| 169 |
+
### Method
|
| 170 |
|
| 171 |
+
* Each evaluation set is split 50/50 by a hash of a group key, so related items stay on the same side. Every modelling
|
| 172 |
+
and threshold decision was made on the *dev* half. The numbers below are from the *sealed* half, which was never
|
| 173 |
+
used for any decision.
|
| 174 |
+
* All baselines were run on exactly the same items with their published models.
|
| 175 |
+
* "Corrected gold": some gold labels in the Wikipedia-derived sets are known to be wrong (English sentences inside
|
| 176 |
+
regional-language wikis, Spanish inside Chavacano). Where at least two of GlotLID, OpenLID and lid.176 agree on
|
| 177 |
+
English or Spanish with probability ≥ 0.6, the gold is corrected. This rule relies on the baselines, so it can only
|
| 178 |
+
favour them.
|
| 179 |
+
* Three sets are public and ship with the source repository: Wikipedia held-out, PH-LID-Hard v0 and the Tatoeba test
|
| 180 |
+
split. The others are private, because they contain copyrighted news text or come from restricted sources.
|
| 181 |
|
| 182 |
+
### Results (sealed halves; accuracy on corrected gold)
|
| 183 |
+
|
| 184 |
+
| set (n) | description | WikaLID | GlotLID v3 | OpenLID-v3 | fastText lid.176 |
|
| 185 |
|---|---|---|---|---|---|
|
| 186 |
+
| Crawl realism (335) | Real Filipino news spans: Taglish, headlines, lists, bylines | **0.949** | 0.875 | 0.785 | 0.734 |
|
| 187 |
+
| News gold spans (327) | Hand-labelled news spans | **0.939** | 0.899 | 0.881 | 0.783 |
|
| 188 |
+
| News gold docs (148) | Hand-labelled whole articles | **0.986** | 0.959 | 0.919 | 0.818 |
|
| 189 |
+
| PLD read speech (1,653) | Prompts from the Philippine Languages Database, 10 languages | **0.917** | 0.834 | 0.417 | 0.341 |
|
| 190 |
+
| Wikipedia held-out (2,563) | Public | **0.925** | 0.907 | 0.710 | 0.664 |
|
| 191 |
+
| PH-LID-Hard v0 (6,379) | Public; synthetic noise: caps, SMS spelling, typos, Taglish, lists | **0.901** | 0.829 | 0.641 | 0.599 |
|
| 192 |
+
| BasahaCorpus (794) | Children's stories (Kinaray-a, Hiligaynon, …) | **0.739** | 0.583 | 0.026 | 0.031 |
|
| 193 |
+
| Tatoeba test (968) | Public; short, clean sentences | 0.928 | **0.955** | 0.758 | 0.805 |
|
| 194 |
+
| **Mean** | | **0.911** | 0.855 | 0.642 | 0.597 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 195 |
|
| 196 |
+
Paired bootstrap against GlotLID (2,000 resamples, 95% CI) on the sealed halves:
|
| 197 |
+
|
| 198 |
+
* WikaLID is better on seven sets: crawl +7.5 pts [+4.5, +11.0], gold spans +4.0 [+0.9, +7.3], gold docs +2.7
|
| 199 |
+
[+0.7, +5.4], PLD +8.3 [+6.7, +9.9], Wikipedia +1.8 [+1.0, +2.7], PH-LID-Hard +7.2 [+6.3, +8.0], Basaha +15.6
|
| 200 |
+
[+12.3, +18.9].
|
| 201 |
+
* **It is worse on Tatoeba: −2.7 pts [−4.6, −0.9].** On short, clean, textbook-style sentences, GlotLID's breadth wins.
|
| 202 |
+
|
| 203 |
+
### Human-labelled external sets (all items)
|
| 204 |
+
|
| 205 |
+
| set | WikaLID | GlotLID v3 | OpenLID-v3 | lid.176 |
|
| 206 |
+
|---|---|---|---|---|
|
| 207 |
+
| FLORES-200 via SIB-200, 9 languages (9,036) | **0.989** | 0.986 | 0.985 | 0.774 |
|
| 208 |
+
| Same sentences in UPPER / Title / lower case (5,388) | **0.999** | 0.975 | 0.575 | 0.522 |
|
| 209 |
+
| Central Bikol story sentences: Bikol recall (900) | **0.961** | 0.894 | 0.000 | 0.003 |
|
| 210 |
+
| 2–5 word fragments of FLORES sentences (4,320) | **0.832** | 0.771 | 0.773 | 0.590 |
|
| 211 |
+
| … of which 2 words (1,080) | **0.704** | 0.624 | 0.621 | 0.514 |
|
| 212 |
+
| Short Tatoeba sentences (543) | 0.856 | **0.866** | 0.632 | 0.685 |
|
| 213 |
+
| Aklanon / Surigaonon sentences → `oth` (110) | **0.345** | 0.300 | 0.082 | 0.055 |
|
| 214 |
+
|
| 215 |
+
### Gate (sealed halves)
|
| 216 |
+
|
| 217 |
+
| set | share accepted | accuracy of accepted rows |
|
| 218 |
+
|---|---|---|
|
| 219 |
+
| Crawl realism | 0.510 | 1.000 |
|
| 220 |
+
| News gold spans | 0.514 | 1.000 |
|
| 221 |
+
| News gold docs | 0.784 | 0.991 |
|
| 222 |
+
| PLD read speech | 0.658 | 0.985 |
|
| 223 |
+
| Wikipedia held-out | 0.566 | 0.989 |
|
| 224 |
+
| PH-LID-Hard v0 | 0.510 | 0.994 |
|
| 225 |
+
| BasahaCorpus | 0.358 | 0.972 |
|
| 226 |
+
| Tatoeba test | 0.395 | 0.995 |
|
| 227 |
+
|
| 228 |
+
These sets are deliberately hard and short, so acceptance rates are much lower than on ordinary news.
|
| 229 |
+
|
| 230 |
+
### Speed
|
| 231 |
+
|
| 232 |
+
Measured on a shared 8-core CPU, one BLAS thread per worker: about 7,000 documents per minute per core for 100-word
|
| 233 |
+
documents, and about 2,200 per minute per core for full news articles (~400 tokens). Throughput scales roughly
|
| 234 |
+
linearly with workers.
|
| 235 |
+
|
| 236 |
+
## Limitations and bias
|
| 237 |
+
|
| 238 |
+
* **Languages without a class.** Aklanon, Surigaonon and many smaller Philippine languages have no open training text.
|
| 239 |
+
Hand-written marker rules move only about a third of Aklanon/Surigaonon sentences to `oth`. The rest are labelled
|
| 240 |
+
`hil` or `ceb`. Do not use WikaLID to count speakers of unsupported languages.
|
| 241 |
+
* **The Visayan family.** Hiligaynon, Kinaray-a and Cebuano remain the main source of errors. `krj` and `tsg` are
|
| 242 |
+
experimental.
|
| 243 |
+
* **English-heavy Taglish.** Tabloid and showbiz text with ALLCAPS leads and English quotes can be labelled `eng`.
|
| 244 |
+
Usually the label set comes back empty and the gate sends the text to `review`. When WikaLID abstains on `eng`, the top
|
| 245 |
+
label is right only about 22% of the time on our gold data.
|
| 246 |
+
* **Short strings.** Two-word inputs are 70% accurate, and a third of the remaining errors are Indonesian vs Malay.
|
| 247 |
+
* **Rare labels on news.** In a manual sample of news predictions, every `krj` (3/3) and `pag` (2/2) prediction was
|
| 248 |
+
wrong: lists, Aklanon or Hiligaynon text. Route these to review.
|
| 249 |
+
* **Calibration across domains.** The conformal sets target 90% coverage at α = 0.10, but that target holds only near
|
| 250 |
+
the calibration data. On the sealed halves, the gold label was inside the set for 82–91% of items on the news, speech,
|
| 251 |
+
Wikipedia, PH-LID-Hard and Tatoeba sets, and for 56% on BasahaCorpus. Between 8% and 29% of sets come back empty, which
|
| 252 |
+
means "no label is safe"; treat an empty set as a signal to review, not as a prediction.
|
| 253 |
+
* **Evaluation caveats.**
|
| 254 |
+
* The news gold sets were labelled by a single, non-native annotator. A native-speaker re-annotation is planned.
|
| 255 |
+
* PH-LID-Hard is synthetic and shares design ideas with the training-data augmentation.
|
| 256 |
+
* BasahaCorpus accuracy counts Minasbate as `oth`, which flatters every system.
|
| 257 |
+
* Real-news results should carry the most weight.
|
| 258 |
+
* **Register and region.** The training text is mostly encyclopedic, web and story text in standard orthographies.
|
| 259 |
+
Older spellings, regional varieties and social-media slang are less well covered.
|
| 260 |
+
|
| 261 |
+
## Training data
|
| 262 |
+
|
| 263 |
+
No training text is distributed with the model. The taggers were trained on sentences from:
|
| 264 |
+
|
| 265 |
+
* Wikipedia (Tagalog, Cebuano, Waray, Ilocano, Bikol, Kapampangan, Pangasinan, Chavacano, English, Spanish, Indonesian,
|
| 266 |
+
Malay and about 30 other languages): CC BY-SA 4.0.
|
| 267 |
+
* Tatoeba: CC BY 2.0 FR.
|
| 268 |
+
* FineWeb-2 web text for the lower-resource languages, filtered for lexical purity: ODC-By 1.0. Page copyright remains
|
| 269 |
+
with the publishers, and this text is used for training only.
|
| 270 |
+
* BasahaCorpus children's stories (Imperial & Kochmar, 2023) for Kinaray-a and Hiligaynon: CC BY 4.0 as stated by its
|
| 271 |
+
authors.
|
| 272 |
+
|
| 273 |
+
Training sentences are augmented with synthetic code-switching, casing, SMS-style spelling and accent variants. No
|
| 274 |
+
scraped news is in the training data. Existing LID tools (GlotLID, OpenLID, lid.176) were used only to detect
|
| 275 |
+
label noise. No third-party model weights are included.
|
| 276 |
+
|
| 277 |
+
## Versioning
|
| 278 |
+
|
| 279 |
+
* Releases follow semantic versioning. A version always pins exact weights. `wikalid verify --expect-version X`
|
| 280 |
+
checks the version and the sha256 of every bundled file against `MANIFEST.json`.
|
| 281 |
+
* **1.0.1** (2026-11-15): renamed from OurLID to WikaLID. Weights and outputs are identical to 1.0.0. `import ourlid`
|
| 282 |
+
and the `ourlid` command remain as aliases.
|
| 283 |
+
* **1.0.0** (2026-11-15): first versioned release, as OurLID.
|
| 284 |
+
* A retrained model will be released as a new minor version with a fresh evaluation. Weights are never swapped silently.
|
| 285 |
+
|
| 286 |
+
## License
|
| 287 |
+
|
| 288 |
+
* **Code:** Apache-2.0.
|
| 289 |
+
* **Model weights:** preview release, all rights reserved; redistribution is not permitted. A public licence is under
|
| 290 |
+
consideration. Because Wikipedia is the largest training source, CC BY-SA 4.0 is the likely choice. See
|
| 291 |
+
`MODEL_LICENSE.md`.
|
| 292 |
+
* **Public evaluation sets:** CC BY-SA 4.0 (Wikipedia-derived) and CC BY 2.0 FR (Tatoeba).
|
| 293 |
+
|
| 294 |
+
## Citation
|
| 295 |
+
|
| 296 |
+
```bibtex
|
| 297 |
+
@software{wikalid_2026,
|
| 298 |
+
title = {WikaLID: Language Identification for Philippine Languages},
|
| 299 |
+
author = {{sapinsapin}},
|
| 300 |
+
year = {2026},
|
| 301 |
+
version = {1.0.1},
|
| 302 |
+
url = {https://huggingface.co/sapinsapin/wikalid}
|
| 303 |
+
}
|
| 304 |
```
|
|
|
|
| 305 |
|
| 306 |
+
## Acknowledgements and contact
|
| 307 |
+
|
| 308 |
+
WikaLID builds on public resources from Wikipedia and Tatoeba contributors, FineWeb-2, BasahaCorpus (Imperial &
|
| 309 |
+
Kochmar, 2023; stories from Let's Read Asia / The Asia Foundation), FLORES-200 / SIB-200 (used for evaluation only) and
|
| 310 |
+
the Philippine Languages Database (used for evaluation only). Baselines: GlotLID (Kargaran et al., 2023), OpenLID
|
| 311 |
+
(Burchell et al., 2023) and fastText lid.176 (Joulin et al., 2016).
|
| 312 |
|
| 313 |
+
Questions, error reports and offers to help with native-speaker review are welcome in the Community tab of this
|
| 314 |
+
repository.
|
|
|
dist/{ourlid-1.0.0-py3-none-any.whl → wikalid-1.0.1-py3-none-any.whl}
RENAMED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6815be3abfcad2f538096489d32e8e68c0ee1e46c5aa533787644fe6be5c249d
|
| 3 |
+
size 52469878
|
eval/RESULTS_v1.0.1.json
ADDED
|
@@ -0,0 +1,2597 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"version": "wikalid-1.0.1",
|
| 3 |
+
"speed_items_per_s": {
|
| 4 |
+
"crawl": 260.0556615657041,
|
| 5 |
+
"goldspan": 279.75437264136514,
|
| 6 |
+
"golddoc": 82.7144606130041,
|
| 7 |
+
"pld": 361.29866678479993,
|
| 8 |
+
"wiki": 412.01515781078336,
|
| 9 |
+
"hard": 417.6514915312866,
|
| 10 |
+
"basaha": 340.77461396884735,
|
| 11 |
+
"tato": 416.0644453947067,
|
| 12 |
+
"t_sib": 433.517085761704,
|
| 13 |
+
"t_frag": 413.52074818541155,
|
| 14 |
+
"t_caps": 394.02282398376394,
|
| 15 |
+
"t_bikol": 390.347293016329,
|
| 16 |
+
"t_aklsgd": 90.24064579563718,
|
| 17 |
+
"t_tshort": 312.07255571899447
|
| 18 |
+
},
|
| 19 |
+
"core": {
|
| 20 |
+
"sealed": {
|
| 21 |
+
"crawl": {
|
| 22 |
+
"n": 335,
|
| 23 |
+
"public": false,
|
| 24 |
+
"wikalid-1.0.1": {
|
| 25 |
+
"acc_raw": 0.9492537313432836,
|
| 26 |
+
"acc_corrected": 0.9492537313432836,
|
| 27 |
+
"macro_f1_corrected": 0.9452212148681051
|
| 28 |
+
},
|
| 29 |
+
"glotlid": {
|
| 30 |
+
"acc_raw": 0.8746268656716418,
|
| 31 |
+
"acc_corrected": 0.8746268656716418,
|
| 32 |
+
"macro_f1_corrected": 0.9258273703041144
|
| 33 |
+
},
|
| 34 |
+
"openlid3": {
|
| 35 |
+
"acc_raw": 0.7850746268656716,
|
| 36 |
+
"acc_corrected": 0.7850746268656716,
|
| 37 |
+
"macro_f1_corrected": 0.6166170580237387
|
| 38 |
+
},
|
| 39 |
+
"lid176": {
|
| 40 |
+
"acc_raw": 0.7343283582089553,
|
| 41 |
+
"acc_corrected": 0.7343283582089553,
|
| 42 |
+
"macro_f1_corrected": 0.5681250135015445
|
| 43 |
+
}
|
| 44 |
+
},
|
| 45 |
+
"goldspan": {
|
| 46 |
+
"n": 327,
|
| 47 |
+
"public": false,
|
| 48 |
+
"wikalid-1.0.1": {
|
| 49 |
+
"acc_raw": 0.9388379204892966,
|
| 50 |
+
"acc_corrected": 0.9388379204892966,
|
| 51 |
+
"macro_f1_corrected": 0.9339768728004022
|
| 52 |
+
},
|
| 53 |
+
"glotlid": {
|
| 54 |
+
"acc_raw": 0.8990825688073395,
|
| 55 |
+
"acc_corrected": 0.8990825688073395,
|
| 56 |
+
"macro_f1_corrected": 0.9213529123029144
|
| 57 |
+
},
|
| 58 |
+
"openlid3": {
|
| 59 |
+
"acc_raw": 0.8807339449541285,
|
| 60 |
+
"acc_corrected": 0.8807339449541285,
|
| 61 |
+
"macro_f1_corrected": 0.8725169369237166
|
| 62 |
+
},
|
| 63 |
+
"lid176": {
|
| 64 |
+
"acc_raw": 0.7828746177370031,
|
| 65 |
+
"acc_corrected": 0.7828746177370031,
|
| 66 |
+
"macro_f1_corrected": 0.7374188657622348
|
| 67 |
+
}
|
| 68 |
+
},
|
| 69 |
+
"golddoc": {
|
| 70 |
+
"n": 148,
|
| 71 |
+
"public": false,
|
| 72 |
+
"wikalid-1.0.1": {
|
| 73 |
+
"acc_raw": 0.9864864864864865,
|
| 74 |
+
"acc_corrected": 0.9864864864864865,
|
| 75 |
+
"macro_f1_corrected": 0.99581589958159
|
| 76 |
+
},
|
| 77 |
+
"glotlid": {
|
| 78 |
+
"acc_raw": 0.9594594594594594,
|
| 79 |
+
"acc_corrected": 0.9594594594594594,
|
| 80 |
+
"macro_f1_corrected": 0.99581589958159
|
| 81 |
+
},
|
| 82 |
+
"openlid3": {
|
| 83 |
+
"acc_raw": 0.918918918918919,
|
| 84 |
+
"acc_corrected": 0.918918918918919,
|
| 85 |
+
"macro_f1_corrected": 0.9916666666666667
|
| 86 |
+
},
|
| 87 |
+
"lid176": {
|
| 88 |
+
"acc_raw": 0.8175675675675675,
|
| 89 |
+
"acc_corrected": 0.8175675675675675,
|
| 90 |
+
"macro_f1_corrected": 0.9008264462809917
|
| 91 |
+
}
|
| 92 |
+
},
|
| 93 |
+
"pld": {
|
| 94 |
+
"n": 1653,
|
| 95 |
+
"public": false,
|
| 96 |
+
"wikalid-1.0.1": {
|
| 97 |
+
"acc_raw": 0.9159104658197217,
|
| 98 |
+
"acc_corrected": 0.9171203871748337,
|
| 99 |
+
"macro_f1_corrected": 0.9256639461728243
|
| 100 |
+
},
|
| 101 |
+
"glotlid": {
|
| 102 |
+
"acc_raw": 0.8300060496067756,
|
| 103 |
+
"acc_corrected": 0.8342407743496673,
|
| 104 |
+
"macro_f1_corrected": 0.8384979959772872
|
| 105 |
+
},
|
| 106 |
+
"openlid3": {
|
| 107 |
+
"acc_raw": 0.41681790683605563,
|
| 108 |
+
"acc_corrected": 0.41742286751361163,
|
| 109 |
+
"macro_f1_corrected": 0.39562927937811565
|
| 110 |
+
},
|
| 111 |
+
"lid176": {
|
| 112 |
+
"acc_raw": 0.3369630973986691,
|
| 113 |
+
"acc_corrected": 0.3411978221415608,
|
| 114 |
+
"macro_f1_corrected": 0.28759118174467524
|
| 115 |
+
}
|
| 116 |
+
},
|
| 117 |
+
"wiki": {
|
| 118 |
+
"n": 2563,
|
| 119 |
+
"public": true,
|
| 120 |
+
"wikalid-1.0.1": {
|
| 121 |
+
"acc_raw": 0.8985563792430745,
|
| 122 |
+
"acc_corrected": 0.9250877877487319,
|
| 123 |
+
"macro_f1_corrected": 0.9284581306444153
|
| 124 |
+
},
|
| 125 |
+
"glotlid": {
|
| 126 |
+
"acc_raw": 0.8685134607881388,
|
| 127 |
+
"acc_corrected": 0.906749902458057,
|
| 128 |
+
"macro_f1_corrected": 0.9068774909539535
|
| 129 |
+
},
|
| 130 |
+
"openlid3": {
|
| 131 |
+
"acc_raw": 0.6761607491221225,
|
| 132 |
+
"acc_corrected": 0.7104955130706204,
|
| 133 |
+
"macro_f1_corrected": 0.6253482661073548
|
| 134 |
+
},
|
| 135 |
+
"lid176": {
|
| 136 |
+
"acc_raw": 0.6195864221615295,
|
| 137 |
+
"acc_corrected": 0.6640655481857198,
|
| 138 |
+
"macro_f1_corrected": 0.5537504126384613
|
| 139 |
+
}
|
| 140 |
+
},
|
| 141 |
+
"hard": {
|
| 142 |
+
"n": 6379,
|
| 143 |
+
"public": true,
|
| 144 |
+
"wikalid-1.0.1": {
|
| 145 |
+
"acc_raw": 0.8789778962219784,
|
| 146 |
+
"acc_corrected": 0.9007681454773475,
|
| 147 |
+
"macro_f1_corrected": 0.8800734956692698
|
| 148 |
+
},
|
| 149 |
+
"glotlid": {
|
| 150 |
+
"acc_raw": 0.7990280608245807,
|
| 151 |
+
"acc_corrected": 0.8291268223859539,
|
| 152 |
+
"macro_f1_corrected": 0.7603844713921172
|
| 153 |
+
},
|
| 154 |
+
"openlid3": {
|
| 155 |
+
"acc_raw": 0.619376077755134,
|
| 156 |
+
"acc_corrected": 0.6411663270105032,
|
| 157 |
+
"macro_f1_corrected": 0.5499190269122138
|
| 158 |
+
},
|
| 159 |
+
"lid176": {
|
| 160 |
+
"acc_raw": 0.5674870669383916,
|
| 161 |
+
"acc_corrected": 0.5989967079479542,
|
| 162 |
+
"macro_f1_corrected": 0.4522667230292048
|
| 163 |
+
}
|
| 164 |
+
},
|
| 165 |
+
"basaha": {
|
| 166 |
+
"n": 794,
|
| 167 |
+
"public": false,
|
| 168 |
+
"wikalid-1.0.1": {
|
| 169 |
+
"acc_raw": 0.739294710327456,
|
| 170 |
+
"acc_corrected": 0.739294710327456,
|
| 171 |
+
"macro_f1_corrected": 0.7949411406077508
|
| 172 |
+
},
|
| 173 |
+
"glotlid": {
|
| 174 |
+
"acc_raw": 0.5831234256926953,
|
| 175 |
+
"acc_corrected": 0.5831234256926953,
|
| 176 |
+
"macro_f1_corrected": 0.6413626117787863
|
| 177 |
+
},
|
| 178 |
+
"openlid3": {
|
| 179 |
+
"acc_raw": 0.02644836272040302,
|
| 180 |
+
"acc_corrected": 0.02644836272040302,
|
| 181 |
+
"macro_f1_corrected": 0.01195219123505976
|
| 182 |
+
},
|
| 183 |
+
"lid176": {
|
| 184 |
+
"acc_raw": 0.031486146095717885,
|
| 185 |
+
"acc_corrected": 0.031486146095717885,
|
| 186 |
+
"macro_f1_corrected": 0.027777777777777776
|
| 187 |
+
}
|
| 188 |
+
},
|
| 189 |
+
"tato": {
|
| 190 |
+
"n": 968,
|
| 191 |
+
"public": true,
|
| 192 |
+
"wikalid-1.0.1": {
|
| 193 |
+
"acc_raw": 0.9307851239669421,
|
| 194 |
+
"acc_corrected": 0.9276859504132231,
|
| 195 |
+
"macro_f1_corrected": 0.9397934525618807
|
| 196 |
+
},
|
| 197 |
+
"glotlid": {
|
| 198 |
+
"acc_raw": 0.9493801652892562,
|
| 199 |
+
"acc_corrected": 0.9545454545454546,
|
| 200 |
+
"macro_f1_corrected": 0.9674877115849037
|
| 201 |
+
},
|
| 202 |
+
"openlid3": {
|
| 203 |
+
"acc_raw": 0.7541322314049587,
|
| 204 |
+
"acc_corrected": 0.7582644628099173,
|
| 205 |
+
"macro_f1_corrected": 0.6814969661391874
|
| 206 |
+
},
|
| 207 |
+
"lid176": {
|
| 208 |
+
"acc_raw": 0.8006198347107438,
|
| 209 |
+
"acc_corrected": 0.8047520661157025,
|
| 210 |
+
"macro_f1_corrected": 0.8037869284335353
|
| 211 |
+
}
|
| 212 |
+
}
|
| 213 |
+
},
|
| 214 |
+
"dev": {
|
| 215 |
+
"crawl": {
|
| 216 |
+
"n": 326,
|
| 217 |
+
"public": false,
|
| 218 |
+
"wikalid-1.0.1": {
|
| 219 |
+
"acc_raw": 0.9631901840490797,
|
| 220 |
+
"acc_corrected": 0.9601226993865031,
|
| 221 |
+
"macro_f1_corrected": 0.9625247361477572
|
| 222 |
+
},
|
| 223 |
+
"glotlid": {
|
| 224 |
+
"acc_raw": 0.8926380368098159,
|
| 225 |
+
"acc_corrected": 0.8895705521472392,
|
| 226 |
+
"macro_f1_corrected": 0.9019766397124888
|
| 227 |
+
},
|
| 228 |
+
"openlid3": {
|
| 229 |
+
"acc_raw": 0.7699386503067485,
|
| 230 |
+
"acc_corrected": 0.7730061349693251,
|
| 231 |
+
"macro_f1_corrected": 0.5809186296991175
|
| 232 |
+
},
|
| 233 |
+
"lid176": {
|
| 234 |
+
"acc_raw": 0.7331288343558282,
|
| 235 |
+
"acc_corrected": 0.7361963190184049,
|
| 236 |
+
"macro_f1_corrected": 0.5275206062736826
|
| 237 |
+
}
|
| 238 |
+
},
|
| 239 |
+
"goldspan": {
|
| 240 |
+
"n": 266,
|
| 241 |
+
"public": false,
|
| 242 |
+
"wikalid-1.0.1": {
|
| 243 |
+
"acc_raw": 0.9624060150375939,
|
| 244 |
+
"acc_corrected": 0.9624060150375939,
|
| 245 |
+
"macro_f1_corrected": 0.9714543812104788
|
| 246 |
+
},
|
| 247 |
+
"glotlid": {
|
| 248 |
+
"acc_raw": 0.8984962406015038,
|
| 249 |
+
"acc_corrected": 0.8984962406015038,
|
| 250 |
+
"macro_f1_corrected": 0.9387562793957143
|
| 251 |
+
},
|
| 252 |
+
"openlid3": {
|
| 253 |
+
"acc_raw": 0.8458646616541353,
|
| 254 |
+
"acc_corrected": 0.8458646616541353,
|
| 255 |
+
"macro_f1_corrected": 0.6227779940314
|
| 256 |
+
},
|
| 257 |
+
"lid176": {
|
| 258 |
+
"acc_raw": 0.7706766917293233,
|
| 259 |
+
"acc_corrected": 0.7706766917293233,
|
| 260 |
+
"macro_f1_corrected": 0.5042274392781381
|
| 261 |
+
}
|
| 262 |
+
},
|
| 263 |
+
"golddoc": {
|
| 264 |
+
"n": 146,
|
| 265 |
+
"public": false,
|
| 266 |
+
"wikalid-1.0.1": {
|
| 267 |
+
"acc_raw": 1.0,
|
| 268 |
+
"acc_corrected": 1.0,
|
| 269 |
+
"macro_f1_corrected": 1.0
|
| 270 |
+
},
|
| 271 |
+
"glotlid": {
|
| 272 |
+
"acc_raw": 0.9383561643835616,
|
| 273 |
+
"acc_corrected": 0.9383561643835616,
|
| 274 |
+
"macro_f1_corrected": 0.9773755656108597
|
| 275 |
+
},
|
| 276 |
+
"openlid3": {
|
| 277 |
+
"acc_raw": 0.8698630136986302,
|
| 278 |
+
"acc_corrected": 0.8698630136986302,
|
| 279 |
+
"macro_f1_corrected": 0.9819819819819819
|
| 280 |
+
},
|
| 281 |
+
"lid176": {
|
| 282 |
+
"acc_raw": 0.773972602739726,
|
| 283 |
+
"acc_corrected": 0.773972602739726,
|
| 284 |
+
"macro_f1_corrected": 0.8715596330275229
|
| 285 |
+
}
|
| 286 |
+
},
|
| 287 |
+
"pld": {
|
| 288 |
+
"n": 1569,
|
| 289 |
+
"public": false,
|
| 290 |
+
"wikalid-1.0.1": {
|
| 291 |
+
"acc_raw": 0.9056724028043339,
|
| 292 |
+
"acc_corrected": 0.9056724028043339,
|
| 293 |
+
"macro_f1_corrected": 0.9169691092184136
|
| 294 |
+
},
|
| 295 |
+
"glotlid": {
|
| 296 |
+
"acc_raw": 0.8527724665391969,
|
| 297 |
+
"acc_corrected": 0.8553218610579987,
|
| 298 |
+
"macro_f1_corrected": 0.8679972593410478
|
| 299 |
+
},
|
| 300 |
+
"openlid3": {
|
| 301 |
+
"acc_raw": 0.4569789674952199,
|
| 302 |
+
"acc_corrected": 0.45825366475462076,
|
| 303 |
+
"macro_f1_corrected": 0.4136458247569358
|
| 304 |
+
},
|
| 305 |
+
"lid176": {
|
| 306 |
+
"acc_raw": 0.3753983428935628,
|
| 307 |
+
"acc_corrected": 0.37794773741236454,
|
| 308 |
+
"macro_f1_corrected": 0.29795429135587
|
| 309 |
+
}
|
| 310 |
+
},
|
| 311 |
+
"wiki": {
|
| 312 |
+
"n": 2474,
|
| 313 |
+
"public": true,
|
| 314 |
+
"wikalid-1.0.1": {
|
| 315 |
+
"acc_raw": 0.898949070331447,
|
| 316 |
+
"acc_corrected": 0.9248181083265966,
|
| 317 |
+
"macro_f1_corrected": 0.9275778391492562
|
| 318 |
+
},
|
| 319 |
+
"glotlid": {
|
| 320 |
+
"acc_raw": 0.872271624898949,
|
| 321 |
+
"acc_corrected": 0.9078415521422797,
|
| 322 |
+
"macro_f1_corrected": 0.9072448684695931
|
| 323 |
+
},
|
| 324 |
+
"openlid3": {
|
| 325 |
+
"acc_raw": 0.6750202101859337,
|
| 326 |
+
"acc_corrected": 0.7012934518997574,
|
| 327 |
+
"macro_f1_corrected": 0.613474975198989
|
| 328 |
+
},
|
| 329 |
+
"lid176": {
|
| 330 |
+
"acc_raw": 0.6261115602263541,
|
| 331 |
+
"acc_corrected": 0.6649151172190784,
|
| 332 |
+
"macro_f1_corrected": 0.559276551222542
|
| 333 |
+
}
|
| 334 |
+
},
|
| 335 |
+
"hard": {
|
| 336 |
+
"n": 6141,
|
| 337 |
+
"public": true,
|
| 338 |
+
"wikalid-1.0.1": {
|
| 339 |
+
"acc_raw": 0.882266731802638,
|
| 340 |
+
"acc_corrected": 0.9057156814851002,
|
| 341 |
+
"macro_f1_corrected": 0.8873738321187085
|
| 342 |
+
},
|
| 343 |
+
"glotlid": {
|
| 344 |
+
"acc_raw": 0.7951473701351571,
|
| 345 |
+
"acc_corrected": 0.82641263637844,
|
| 346 |
+
"macro_f1_corrected": 0.7611605654646197
|
| 347 |
+
},
|
| 348 |
+
"openlid3": {
|
| 349 |
+
"acc_raw": 0.6143950496661782,
|
| 350 |
+
"acc_corrected": 0.6363784399934864,
|
| 351 |
+
"macro_f1_corrected": 0.55206952431819
|
| 352 |
+
},
|
| 353 |
+
"lid176": {
|
| 354 |
+
"acc_raw": 0.5684741898713565,
|
| 355 |
+
"acc_corrected": 0.6021820550398957,
|
| 356 |
+
"macro_f1_corrected": 0.45472278889467443
|
| 357 |
+
}
|
| 358 |
+
},
|
| 359 |
+
"basaha": {
|
| 360 |
+
"n": 806,
|
| 361 |
+
"public": false,
|
| 362 |
+
"wikalid-1.0.1": {
|
| 363 |
+
"acc_raw": 0.7531017369727047,
|
| 364 |
+
"acc_corrected": 0.7531017369727047,
|
| 365 |
+
"macro_f1_corrected": 0.7817121529694779
|
| 366 |
+
},
|
| 367 |
+
"glotlid": {
|
| 368 |
+
"acc_raw": 0.5496277915632755,
|
| 369 |
+
"acc_corrected": 0.5496277915632755,
|
| 370 |
+
"macro_f1_corrected": 0.5981772873473876
|
| 371 |
+
},
|
| 372 |
+
"openlid3": {
|
| 373 |
+
"acc_raw": 0.016129032258064516,
|
| 374 |
+
"acc_corrected": 0.016129032258064516,
|
| 375 |
+
"macro_f1_corrected": 0.015765765765765764
|
| 376 |
+
},
|
| 377 |
+
"lid176": {
|
| 378 |
+
"acc_raw": 0.031017369727047148,
|
| 379 |
+
"acc_corrected": 0.031017369727047148,
|
| 380 |
+
"macro_f1_corrected": 0.03524229074889868
|
| 381 |
+
}
|
| 382 |
+
},
|
| 383 |
+
"tato": {
|
| 384 |
+
"n": 879,
|
| 385 |
+
"public": true,
|
| 386 |
+
"wikalid-1.0.1": {
|
| 387 |
+
"acc_raw": 0.9351535836177475,
|
| 388 |
+
"acc_corrected": 0.9271899886234357,
|
| 389 |
+
"macro_f1_corrected": 0.9424981517429267
|
| 390 |
+
},
|
| 391 |
+
"glotlid": {
|
| 392 |
+
"acc_raw": 0.9408418657565415,
|
| 393 |
+
"acc_corrected": 0.9431171786120591,
|
| 394 |
+
"macro_f1_corrected": 0.9537518269150536
|
| 395 |
+
},
|
| 396 |
+
"openlid3": {
|
| 397 |
+
"acc_raw": 0.7667804323094426,
|
| 398 |
+
"acc_corrected": 0.7736063708759955,
|
| 399 |
+
"macro_f1_corrected": 0.6932009448006917
|
| 400 |
+
},
|
| 401 |
+
"lid176": {
|
| 402 |
+
"acc_raw": 0.7986348122866894,
|
| 403 |
+
"acc_corrected": 0.8065984072810012,
|
| 404 |
+
"macro_f1_corrected": 0.79135387842101
|
| 405 |
+
}
|
| 406 |
+
}
|
| 407 |
+
},
|
| 408 |
+
"all": {
|
| 409 |
+
"crawl": {
|
| 410 |
+
"n": 661,
|
| 411 |
+
"public": false,
|
| 412 |
+
"wikalid-1.0.1": {
|
| 413 |
+
"acc_raw": 0.9561270801815431,
|
| 414 |
+
"acc_corrected": 0.9546142208774584,
|
| 415 |
+
"macro_f1_corrected": 0.9237178861239013
|
| 416 |
+
},
|
| 417 |
+
"glotlid": {
|
| 418 |
+
"acc_raw": 0.8835098335854765,
|
| 419 |
+
"acc_corrected": 0.8819969742813918,
|
| 420 |
+
"macro_f1_corrected": 0.8054684637926802
|
| 421 |
+
},
|
| 422 |
+
"openlid3": {
|
| 423 |
+
"acc_raw": 0.7776096822995462,
|
| 424 |
+
"acc_corrected": 0.7791225416036308,
|
| 425 |
+
"macro_f1_corrected": 0.5167738175035881
|
| 426 |
+
},
|
| 427 |
+
"lid176": {
|
| 428 |
+
"acc_raw": 0.7337367624810892,
|
| 429 |
+
"acc_corrected": 0.735249621785174,
|
| 430 |
+
"macro_f1_corrected": 0.41546818702156924
|
| 431 |
+
}
|
| 432 |
+
},
|
| 433 |
+
"goldspan": {
|
| 434 |
+
"n": 593,
|
| 435 |
+
"public": false,
|
| 436 |
+
"wikalid-1.0.1": {
|
| 437 |
+
"acc_raw": 0.9494097807757167,
|
| 438 |
+
"acc_corrected": 0.9494097807757167,
|
| 439 |
+
"macro_f1_corrected": 0.8577709147562972
|
| 440 |
+
},
|
| 441 |
+
"glotlid": {
|
| 442 |
+
"acc_raw": 0.8988195615514334,
|
| 443 |
+
"acc_corrected": 0.8988195615514334,
|
| 444 |
+
"macro_f1_corrected": 0.6611717645352578
|
| 445 |
+
},
|
| 446 |
+
"openlid3": {
|
| 447 |
+
"acc_raw": 0.8650927487352446,
|
| 448 |
+
"acc_corrected": 0.8650927487352446,
|
| 449 |
+
"macro_f1_corrected": 0.5199837315153472
|
| 450 |
+
},
|
| 451 |
+
"lid176": {
|
| 452 |
+
"acc_raw": 0.7774030354131535,
|
| 453 |
+
"acc_corrected": 0.7774030354131535,
|
| 454 |
+
"macro_f1_corrected": 0.3729074002101596
|
| 455 |
+
}
|
| 456 |
+
},
|
| 457 |
+
"golddoc": {
|
| 458 |
+
"n": 294,
|
| 459 |
+
"public": false,
|
| 460 |
+
"wikalid-1.0.1": {
|
| 461 |
+
"acc_raw": 0.9931972789115646,
|
| 462 |
+
"acc_corrected": 0.9931972789115646,
|
| 463 |
+
"macro_f1_corrected": 0.9810489661935692
|
| 464 |
+
},
|
| 465 |
+
"glotlid": {
|
| 466 |
+
"acc_raw": 0.9489795918367347,
|
| 467 |
+
"acc_corrected": 0.9489795918367347,
|
| 468 |
+
"macro_f1_corrected": 0.9262168849655593
|
| 469 |
+
},
|
| 470 |
+
"openlid3": {
|
| 471 |
+
"acc_raw": 0.8945578231292517,
|
| 472 |
+
"acc_corrected": 0.8945578231292517,
|
| 473 |
+
"macro_f1_corrected": 0.5512265512265512
|
| 474 |
+
},
|
| 475 |
+
"lid176": {
|
| 476 |
+
"acc_raw": 0.7959183673469388,
|
| 477 |
+
"acc_corrected": 0.7959183673469388,
|
| 478 |
+
"macro_f1_corrected": 0.5659224441833137
|
| 479 |
+
}
|
| 480 |
+
},
|
| 481 |
+
"pld": {
|
| 482 |
+
"n": 3222,
|
| 483 |
+
"public": false,
|
| 484 |
+
"wikalid-1.0.1": {
|
| 485 |
+
"acc_raw": 0.9109248913718188,
|
| 486 |
+
"acc_corrected": 0.9115456238361266,
|
| 487 |
+
"macro_f1_corrected": 0.9226356531452579
|
| 488 |
+
},
|
| 489 |
+
"glotlid": {
|
| 490 |
+
"acc_raw": 0.8410924891371818,
|
| 491 |
+
"acc_corrected": 0.8445065176908753,
|
| 492 |
+
"macro_f1_corrected": 0.8531363677352409
|
| 493 |
+
},
|
| 494 |
+
"openlid3": {
|
| 495 |
+
"acc_raw": 0.43637492240844195,
|
| 496 |
+
"acc_corrected": 0.4373060211049038,
|
| 497 |
+
"macro_f1_corrected": 0.40474253590796233
|
| 498 |
+
},
|
| 499 |
+
"lid176": {
|
| 500 |
+
"acc_raw": 0.35567970204841715,
|
| 501 |
+
"acc_corrected": 0.3590937306021105,
|
| 502 |
+
"macro_f1_corrected": 0.2924533326385652
|
| 503 |
+
}
|
| 504 |
+
},
|
| 505 |
+
"wiki": {
|
| 506 |
+
"n": 5037,
|
| 507 |
+
"public": true,
|
| 508 |
+
"wikalid-1.0.1": {
|
| 509 |
+
"acc_raw": 0.8987492555092317,
|
| 510 |
+
"acc_corrected": 0.9249553305539011,
|
| 511 |
+
"macro_f1_corrected": 0.9281129552088062
|
| 512 |
+
},
|
| 513 |
+
"glotlid": {
|
| 514 |
+
"acc_raw": 0.8703593408775064,
|
| 515 |
+
"acc_corrected": 0.9072860829859043,
|
| 516 |
+
"macro_f1_corrected": 0.9072521485914344
|
| 517 |
+
},
|
| 518 |
+
"openlid3": {
|
| 519 |
+
"acc_raw": 0.6756005558864403,
|
| 520 |
+
"acc_corrected": 0.7059757792336708,
|
| 521 |
+
"macro_f1_corrected": 0.6196306323515995
|
| 522 |
+
},
|
| 523 |
+
"lid176": {
|
| 524 |
+
"acc_raw": 0.6227913440540004,
|
| 525 |
+
"acc_corrected": 0.6644828270796109,
|
| 526 |
+
"macro_f1_corrected": 0.5564368479383145
|
| 527 |
+
}
|
| 528 |
+
},
|
| 529 |
+
"hard": {
|
| 530 |
+
"n": 12520,
|
| 531 |
+
"public": true,
|
| 532 |
+
"wikalid-1.0.1": {
|
| 533 |
+
"acc_raw": 0.880591054313099,
|
| 534 |
+
"acc_corrected": 0.9031948881789137,
|
| 535 |
+
"macro_f1_corrected": 0.8837680817481542
|
| 536 |
+
},
|
| 537 |
+
"glotlid": {
|
| 538 |
+
"acc_raw": 0.7971246006389776,
|
| 539 |
+
"acc_corrected": 0.8277955271565495,
|
| 540 |
+
"macro_f1_corrected": 0.7608953515211622
|
| 541 |
+
},
|
| 542 |
+
"openlid3": {
|
| 543 |
+
"acc_raw": 0.6169329073482428,
|
| 544 |
+
"acc_corrected": 0.638817891373802,
|
| 545 |
+
"macro_f1_corrected": 0.5510186484085835
|
| 546 |
+
},
|
| 547 |
+
"lid176": {
|
| 548 |
+
"acc_raw": 0.5679712460063898,
|
| 549 |
+
"acc_corrected": 0.6005591054313099,
|
| 550 |
+
"macro_f1_corrected": 0.45366292115150086
|
| 551 |
+
}
|
| 552 |
+
},
|
| 553 |
+
"basaha": {
|
| 554 |
+
"n": 1600,
|
| 555 |
+
"public": false,
|
| 556 |
+
"wikalid-1.0.1": {
|
| 557 |
+
"acc_raw": 0.74625,
|
| 558 |
+
"acc_corrected": 0.74625,
|
| 559 |
+
"macro_f1_corrected": 0.7884680885328015
|
| 560 |
+
},
|
| 561 |
+
"glotlid": {
|
| 562 |
+
"acc_raw": 0.56625,
|
| 563 |
+
"acc_corrected": 0.56625,
|
| 564 |
+
"macro_f1_corrected": 0.6200922101491063
|
| 565 |
+
},
|
| 566 |
+
"openlid3": {
|
| 567 |
+
"acc_raw": 0.02125,
|
| 568 |
+
"acc_corrected": 0.02125,
|
| 569 |
+
"macro_f1_corrected": 0.013742071881606765
|
| 570 |
+
},
|
| 571 |
+
"lid176": {
|
| 572 |
+
"acc_raw": 0.03125,
|
| 573 |
+
"acc_corrected": 0.03125,
|
| 574 |
+
"macro_f1_corrected": 0.031315240083507306
|
| 575 |
+
}
|
| 576 |
+
},
|
| 577 |
+
"tato": {
|
| 578 |
+
"n": 1847,
|
| 579 |
+
"public": true,
|
| 580 |
+
"wikalid-1.0.1": {
|
| 581 |
+
"acc_raw": 0.9328641039523552,
|
| 582 |
+
"acc_corrected": 0.9274499187872225,
|
| 583 |
+
"macro_f1_corrected": 0.9407261199876595
|
| 584 |
+
},
|
| 585 |
+
"glotlid": {
|
| 586 |
+
"acc_raw": 0.9453167298321603,
|
| 587 |
+
"acc_corrected": 0.9491066594477531,
|
| 588 |
+
"macro_f1_corrected": 0.9610631108461403
|
| 589 |
+
},
|
| 590 |
+
"openlid3": {
|
| 591 |
+
"acc_raw": 0.7601515971846237,
|
| 592 |
+
"acc_corrected": 0.7655657823497564,
|
| 593 |
+
"macro_f1_corrected": 0.6865861879529651
|
| 594 |
+
},
|
| 595 |
+
"lid176": {
|
| 596 |
+
"acc_raw": 0.7996751488900921,
|
| 597 |
+
"acc_corrected": 0.805630752571738,
|
| 598 |
+
"macro_f1_corrected": 0.7984173253607114
|
| 599 |
+
}
|
| 600 |
+
}
|
| 601 |
+
}
|
| 602 |
+
},
|
| 603 |
+
"trust": {
|
| 604 |
+
"t_sib": {
|
| 605 |
+
"n": 9036,
|
| 606 |
+
"wikalid-1.0.1": {
|
| 607 |
+
"acc": 0.9892651615759186,
|
| 608 |
+
"ci95": [
|
| 609 |
+
0.9869227230925369,
|
| 610 |
+
0.9911917606833734
|
| 611 |
+
],
|
| 612 |
+
"by": {
|
| 613 |
+
"ceb": {
|
| 614 |
+
"acc": 0.9970119521912351,
|
| 615 |
+
"n": 1004,
|
| 616 |
+
"ci95": [
|
| 617 |
+
0.9912516700049157,
|
| 618 |
+
0.9989833034274174
|
| 619 |
+
]
|
| 620 |
+
},
|
| 621 |
+
"eng": {
|
| 622 |
+
"acc": 1.0,
|
| 623 |
+
"n": 1004,
|
| 624 |
+
"ci95": [
|
| 625 |
+
0.9961882899058743,
|
| 626 |
+
1.0000000000000002
|
| 627 |
+
]
|
| 628 |
+
},
|
| 629 |
+
"ilo": {
|
| 630 |
+
"acc": 1.0,
|
| 631 |
+
"n": 1004,
|
| 632 |
+
"ci95": [
|
| 633 |
+
0.9961882899058743,
|
| 634 |
+
1.0000000000000002
|
| 635 |
+
]
|
| 636 |
+
},
|
| 637 |
+
"ind": {
|
| 638 |
+
"acc": 0.9591633466135459,
|
| 639 |
+
"n": 1004,
|
| 640 |
+
"ci95": [
|
| 641 |
+
0.9450695618604726,
|
| 642 |
+
0.9697567362403402
|
| 643 |
+
]
|
| 644 |
+
},
|
| 645 |
+
"msa": {
|
| 646 |
+
"acc": 0.953187250996016,
|
| 647 |
+
"n": 1004,
|
| 648 |
+
"ci95": [
|
| 649 |
+
0.9383043206814081,
|
| 650 |
+
0.9646153444723224
|
| 651 |
+
]
|
| 652 |
+
},
|
| 653 |
+
"pag": {
|
| 654 |
+
"acc": 0.9970119521912351,
|
| 655 |
+
"n": 1004,
|
| 656 |
+
"ci95": [
|
| 657 |
+
0.9912516700049157,
|
| 658 |
+
0.9989833034274174
|
| 659 |
+
]
|
| 660 |
+
},
|
| 661 |
+
"sib200": {
|
| 662 |
+
"acc": 0.9892651615759186,
|
| 663 |
+
"n": 9036,
|
| 664 |
+
"ci95": [
|
| 665 |
+
0.9869227230925369,
|
| 666 |
+
0.9911917606833734
|
| 667 |
+
]
|
| 668 |
+
},
|
| 669 |
+
"spa": {
|
| 670 |
+
"acc": 0.9970119521912351,
|
| 671 |
+
"n": 1004,
|
| 672 |
+
"ci95": [
|
| 673 |
+
0.9912516700049157,
|
| 674 |
+
0.9989833034274174
|
| 675 |
+
]
|
| 676 |
+
},
|
| 677 |
+
"tgl": {
|
| 678 |
+
"acc": 1.0,
|
| 679 |
+
"n": 1004,
|
| 680 |
+
"ci95": [
|
| 681 |
+
0.9961882899058743,
|
| 682 |
+
1.0000000000000002
|
| 683 |
+
]
|
| 684 |
+
},
|
| 685 |
+
"war": {
|
| 686 |
+
"acc": 1.0,
|
| 687 |
+
"n": 1004,
|
| 688 |
+
"ci95": [
|
| 689 |
+
0.9961882899058743,
|
| 690 |
+
1.0000000000000002
|
| 691 |
+
]
|
| 692 |
+
}
|
| 693 |
+
}
|
| 694 |
+
},
|
| 695 |
+
"glotlid": {
|
| 696 |
+
"acc": 0.9863877822045153,
|
| 697 |
+
"ci95": [
|
| 698 |
+
0.9837834463036013,
|
| 699 |
+
0.9885787242901842
|
| 700 |
+
],
|
| 701 |
+
"by": {
|
| 702 |
+
"ceb": {
|
| 703 |
+
"acc": 0.9910358565737052,
|
| 704 |
+
"n": 1004,
|
| 705 |
+
"ci95": [
|
| 706 |
+
0.9830514267968079,
|
| 707 |
+
0.9952769136884431
|
| 708 |
+
]
|
| 709 |
+
},
|
| 710 |
+
"eng": {
|
| 711 |
+
"acc": 1.0,
|
| 712 |
+
"n": 1004,
|
| 713 |
+
"ci95": [
|
| 714 |
+
0.9961882899058743,
|
| 715 |
+
1.0000000000000002
|
| 716 |
+
]
|
| 717 |
+
},
|
| 718 |
+
"ilo": {
|
| 719 |
+
"acc": 0.999003984063745,
|
| 720 |
+
"n": 1004,
|
| 721 |
+
"ci95": [
|
| 722 |
+
0.9943796894699304,
|
| 723 |
+
0.9998241616114303
|
| 724 |
+
]
|
| 725 |
+
},
|
| 726 |
+
"ind": {
|
| 727 |
+
"acc": 0.9950199203187251,
|
| 728 |
+
"n": 1004,
|
| 729 |
+
"ci95": [
|
| 730 |
+
0.9883950765339303,
|
| 731 |
+
0.9978710192493755
|
| 732 |
+
]
|
| 733 |
+
},
|
| 734 |
+
"msa": {
|
| 735 |
+
"acc": 0.8944223107569721,
|
| 736 |
+
"n": 1004,
|
| 737 |
+
"ci95": [
|
| 738 |
+
0.873887228535692,
|
| 739 |
+
0.9119505459717305
|
| 740 |
+
]
|
| 741 |
+
},
|
| 742 |
+
"pag": {
|
| 743 |
+
"acc": 1.0,
|
| 744 |
+
"n": 1004,
|
| 745 |
+
"ci95": [
|
| 746 |
+
0.9961882899058743,
|
| 747 |
+
1.0000000000000002
|
| 748 |
+
]
|
| 749 |
+
},
|
| 750 |
+
"sib200": {
|
| 751 |
+
"acc": 0.9863877822045153,
|
| 752 |
+
"n": 9036,
|
| 753 |
+
"ci95": [
|
| 754 |
+
0.9837834463036013,
|
| 755 |
+
0.9885787242901842
|
| 756 |
+
]
|
| 757 |
+
},
|
| 758 |
+
"spa": {
|
| 759 |
+
"acc": 0.99800796812749,
|
| 760 |
+
"n": 1004,
|
| 761 |
+
"ci95": [
|
| 762 |
+
0.9927658578932684,
|
| 763 |
+
0.9994535543635784
|
| 764 |
+
]
|
| 765 |
+
},
|
| 766 |
+
"tgl": {
|
| 767 |
+
"acc": 1.0,
|
| 768 |
+
"n": 1004,
|
| 769 |
+
"ci95": [
|
| 770 |
+
0.9961882899058743,
|
| 771 |
+
1.0000000000000002
|
| 772 |
+
]
|
| 773 |
+
},
|
| 774 |
+
"war": {
|
| 775 |
+
"acc": 1.0,
|
| 776 |
+
"n": 1004,
|
| 777 |
+
"ci95": [
|
| 778 |
+
0.9961882899058743,
|
| 779 |
+
1.0000000000000002
|
| 780 |
+
]
|
| 781 |
+
}
|
| 782 |
+
}
|
| 783 |
+
},
|
| 784 |
+
"openlid3": {
|
| 785 |
+
"acc": 0.9846170872067287,
|
| 786 |
+
"ci95": [
|
| 787 |
+
0.9818657500423222,
|
| 788 |
+
0.9869565355163097
|
| 789 |
+
],
|
| 790 |
+
"by": {
|
| 791 |
+
"ceb": {
|
| 792 |
+
"acc": 1.0,
|
| 793 |
+
"n": 1004,
|
| 794 |
+
"ci95": [
|
| 795 |
+
0.9961882899058743,
|
| 796 |
+
1.0000000000000002
|
| 797 |
+
]
|
| 798 |
+
},
|
| 799 |
+
"eng": {
|
| 800 |
+
"acc": 0.99800796812749,
|
| 801 |
+
"n": 1004,
|
| 802 |
+
"ci95": [
|
| 803 |
+
0.9927658578932684,
|
| 804 |
+
0.9994535543635784
|
| 805 |
+
]
|
| 806 |
+
},
|
| 807 |
+
"ilo": {
|
| 808 |
+
"acc": 1.0,
|
| 809 |
+
"n": 1004,
|
| 810 |
+
"ci95": [
|
| 811 |
+
0.9961882899058743,
|
| 812 |
+
1.0000000000000002
|
| 813 |
+
]
|
| 814 |
+
},
|
| 815 |
+
"ind": {
|
| 816 |
+
"acc": 0.9183266932270916,
|
| 817 |
+
"n": 1004,
|
| 818 |
+
"ci95": [
|
| 819 |
+
0.8997488848216905,
|
| 820 |
+
0.9337154214740607
|
| 821 |
+
]
|
| 822 |
+
},
|
| 823 |
+
"msa": {
|
| 824 |
+
"acc": 0.9631474103585658,
|
| 825 |
+
"n": 1004,
|
| 826 |
+
"ci95": [
|
| 827 |
+
0.9496171963573046,
|
| 828 |
+
0.9731468570415629
|
| 829 |
+
]
|
| 830 |
+
},
|
| 831 |
+
"pag": {
|
| 832 |
+
"acc": 0.9960159362549801,
|
| 833 |
+
"n": 1004,
|
| 834 |
+
"ci95": [
|
| 835 |
+
0.9898008918922921,
|
| 836 |
+
0.9984496427155273
|
| 837 |
+
]
|
| 838 |
+
},
|
| 839 |
+
"sib200": {
|
| 840 |
+
"acc": 0.9846170872067287,
|
| 841 |
+
"n": 9036,
|
| 842 |
+
"ci95": [
|
| 843 |
+
0.9818657500423222,
|
| 844 |
+
0.9869565355163097
|
| 845 |
+
]
|
| 846 |
+
},
|
| 847 |
+
"spa": {
|
| 848 |
+
"acc": 0.9860557768924303,
|
| 849 |
+
"n": 1004,
|
| 850 |
+
"ci95": [
|
| 851 |
+
0.9767302763202146,
|
| 852 |
+
0.9916758700424679
|
| 853 |
+
]
|
| 854 |
+
},
|
| 855 |
+
"tgl": {
|
| 856 |
+
"acc": 1.0,
|
| 857 |
+
"n": 1004,
|
| 858 |
+
"ci95": [
|
| 859 |
+
0.9961882899058743,
|
| 860 |
+
1.0000000000000002
|
| 861 |
+
]
|
| 862 |
+
},
|
| 863 |
+
"war": {
|
| 864 |
+
"acc": 1.0,
|
| 865 |
+
"n": 1004,
|
| 866 |
+
"ci95": [
|
| 867 |
+
0.9961882899058743,
|
| 868 |
+
1.0000000000000002
|
| 869 |
+
]
|
| 870 |
+
}
|
| 871 |
+
}
|
| 872 |
+
},
|
| 873 |
+
"lid176": {
|
| 874 |
+
"acc": 0.7737937140327579,
|
| 875 |
+
"ci95": [
|
| 876 |
+
0.7650519445652785,
|
| 877 |
+
0.7823027789953644
|
| 878 |
+
],
|
| 879 |
+
"by": {
|
| 880 |
+
"ceb": {
|
| 881 |
+
"acc": 0.9223107569721115,
|
| 882 |
+
"n": 1004,
|
| 883 |
+
"ci95": [
|
| 884 |
+
0.9040963910139844,
|
| 885 |
+
0.9373056705798216
|
| 886 |
+
]
|
| 887 |
+
},
|
| 888 |
+
"eng": {
|
| 889 |
+
"acc": 1.0,
|
| 890 |
+
"n": 1004,
|
| 891 |
+
"ci95": [
|
| 892 |
+
0.9961882899058743,
|
| 893 |
+
1.0000000000000002
|
| 894 |
+
]
|
| 895 |
+
},
|
| 896 |
+
"ilo": {
|
| 897 |
+
"acc": 0.9850597609561753,
|
| 898 |
+
"n": 1004,
|
| 899 |
+
"ci95": [
|
| 900 |
+
0.975496224606418,
|
| 901 |
+
0.9909254829317506
|
| 902 |
+
]
|
| 903 |
+
},
|
| 904 |
+
"ind": {
|
| 905 |
+
"acc": 0.9691235059760956,
|
| 906 |
+
"n": 1004,
|
| 907 |
+
"ci95": [
|
| 908 |
+
0.9565068693060876,
|
| 909 |
+
0.9781638170398621
|
| 910 |
+
]
|
| 911 |
+
},
|
| 912 |
+
"msa": {
|
| 913 |
+
"acc": 0.449203187250996,
|
| 914 |
+
"n": 1004,
|
| 915 |
+
"ci95": [
|
| 916 |
+
0.4186863948679612,
|
| 917 |
+
0.48010722508184045
|
| 918 |
+
]
|
| 919 |
+
},
|
| 920 |
+
"pag": {
|
| 921 |
+
"acc": 0.0,
|
| 922 |
+
"n": 1004,
|
| 923 |
+
"ci95": [
|
| 924 |
+
0.0,
|
| 925 |
+
0.0038117100941259022
|
| 926 |
+
]
|
| 927 |
+
},
|
| 928 |
+
"sib200": {
|
| 929 |
+
"acc": 0.7737937140327579,
|
| 930 |
+
"n": 9036,
|
| 931 |
+
"ci95": [
|
| 932 |
+
0.7650519445652785,
|
| 933 |
+
0.7823027789953644
|
| 934 |
+
]
|
| 935 |
+
},
|
| 936 |
+
"spa": {
|
| 937 |
+
"acc": 1.0,
|
| 938 |
+
"n": 1004,
|
| 939 |
+
"ci95": [
|
| 940 |
+
0.9961882899058743,
|
| 941 |
+
1.0000000000000002
|
| 942 |
+
]
|
| 943 |
+
},
|
| 944 |
+
"tgl": {
|
| 945 |
+
"acc": 0.99800796812749,
|
| 946 |
+
"n": 1004,
|
| 947 |
+
"ci95": [
|
| 948 |
+
0.9927658578932684,
|
| 949 |
+
0.9994535543635784
|
| 950 |
+
]
|
| 951 |
+
},
|
| 952 |
+
"war": {
|
| 953 |
+
"acc": 0.6404382470119522,
|
| 954 |
+
"n": 1004,
|
| 955 |
+
"ci95": [
|
| 956 |
+
0.6102712632213478,
|
| 957 |
+
0.6695346110350828
|
| 958 |
+
]
|
| 959 |
+
}
|
| 960 |
+
}
|
| 961 |
+
}
|
| 962 |
+
},
|
| 963 |
+
"t_frag": {
|
| 964 |
+
"n": 4320,
|
| 965 |
+
"wikalid-1.0.1": {
|
| 966 |
+
"acc": 0.8319444444444445,
|
| 967 |
+
"ci95": [
|
| 968 |
+
0.8205002431714228,
|
| 969 |
+
0.8427988009008671
|
| 970 |
+
],
|
| 971 |
+
"by": {
|
| 972 |
+
"ceb": {
|
| 973 |
+
"acc": 0.8145833333333333,
|
| 974 |
+
"n": 480,
|
| 975 |
+
"ci95": [
|
| 976 |
+
0.7773661170146048,
|
| 977 |
+
0.8468050993504205
|
| 978 |
+
]
|
| 979 |
+
},
|
| 980 |
+
"eng": {
|
| 981 |
+
"acc": 0.9770833333333333,
|
| 982 |
+
"n": 480,
|
| 983 |
+
"ci95": [
|
| 984 |
+
0.9594342159747407,
|
| 985 |
+
0.9871565691086419
|
| 986 |
+
]
|
| 987 |
+
},
|
| 988 |
+
"ilo": {
|
| 989 |
+
"acc": 0.9416666666666667,
|
| 990 |
+
"n": 480,
|
| 991 |
+
"ci95": [
|
| 992 |
+
0.9169836611947353,
|
| 993 |
+
0.9593361922449029
|
| 994 |
+
]
|
| 995 |
+
},
|
| 996 |
+
"ind": {
|
| 997 |
+
"acc": 0.8208333333333333,
|
| 998 |
+
"n": 480,
|
| 999 |
+
"ci95": [
|
| 1000 |
+
0.7840199414043012,
|
| 1001 |
+
0.8525520276037377
|
| 1002 |
+
]
|
| 1003 |
+
},
|
| 1004 |
+
"len2": {
|
| 1005 |
+
"acc": 0.7037037037037037,
|
| 1006 |
+
"n": 1080,
|
| 1007 |
+
"ci95": [
|
| 1008 |
+
0.6757869878102885,
|
| 1009 |
+
0.7301763928165486
|
| 1010 |
+
]
|
| 1011 |
+
},
|
| 1012 |
+
"len3": {
|
| 1013 |
+
"acc": 0.8148148148148148,
|
| 1014 |
+
"n": 1080,
|
| 1015 |
+
"ci95": [
|
| 1016 |
+
0.7905457940854126,
|
| 1017 |
+
0.8368521577924264
|
| 1018 |
+
]
|
| 1019 |
+
},
|
| 1020 |
+
"len4": {
|
| 1021 |
+
"acc": 0.8777777777777778,
|
| 1022 |
+
"n": 1080,
|
| 1023 |
+
"ci95": [
|
| 1024 |
+
0.8568925867327652,
|
| 1025 |
+
0.8959849555206416
|
| 1026 |
+
]
|
| 1027 |
+
},
|
| 1028 |
+
"len5": {
|
| 1029 |
+
"acc": 0.9314814814814815,
|
| 1030 |
+
"n": 1080,
|
| 1031 |
+
"ci95": [
|
| 1032 |
+
0.9148340030838412,
|
| 1033 |
+
0.9450702486075498
|
| 1034 |
+
]
|
| 1035 |
+
},
|
| 1036 |
+
"msa": {
|
| 1037 |
+
"acc": 0.6395833333333333,
|
| 1038 |
+
"n": 480,
|
| 1039 |
+
"ci95": [
|
| 1040 |
+
0.5956792716294282,
|
| 1041 |
+
0.681270870731212
|
| 1042 |
+
]
|
| 1043 |
+
},
|
| 1044 |
+
"pag": {
|
| 1045 |
+
"acc": 0.8729166666666667,
|
| 1046 |
+
"n": 480,
|
| 1047 |
+
"ci95": [
|
| 1048 |
+
0.8401304181213104,
|
| 1049 |
+
0.8997811562451764
|
| 1050 |
+
]
|
| 1051 |
+
},
|
| 1052 |
+
"spa": {
|
| 1053 |
+
"acc": 0.8708333333333333,
|
| 1054 |
+
"n": 480,
|
| 1055 |
+
"ci95": [
|
| 1056 |
+
0.8378597177438099,
|
| 1057 |
+
0.8979182724083392
|
| 1058 |
+
]
|
| 1059 |
+
},
|
| 1060 |
+
"tgl": {
|
| 1061 |
+
"acc": 0.84375,
|
| 1062 |
+
"n": 480,
|
| 1063 |
+
"ci95": [
|
| 1064 |
+
0.8085522736473225,
|
| 1065 |
+
0.8734891217184335
|
| 1066 |
+
]
|
| 1067 |
+
},
|
| 1068 |
+
"war": {
|
| 1069 |
+
"acc": 0.70625,
|
| 1070 |
+
"n": 480,
|
| 1071 |
+
"ci95": [
|
| 1072 |
+
0.663993732629122,
|
| 1073 |
+
0.7452311045903318
|
| 1074 |
+
]
|
| 1075 |
+
}
|
| 1076 |
+
}
|
| 1077 |
+
},
|
| 1078 |
+
"glotlid": {
|
| 1079 |
+
"acc": 0.7712962962962963,
|
| 1080 |
+
"ci95": [
|
| 1081 |
+
0.7585339732255689,
|
| 1082 |
+
0.7835765425713093
|
| 1083 |
+
],
|
| 1084 |
+
"by": {
|
| 1085 |
+
"ceb": {
|
| 1086 |
+
"acc": 0.7604166666666666,
|
| 1087 |
+
"n": 480,
|
| 1088 |
+
"ci95": [
|
| 1089 |
+
0.7202600260476222,
|
| 1090 |
+
0.7964380007446172
|
| 1091 |
+
]
|
| 1092 |
+
},
|
| 1093 |
+
"eng": {
|
| 1094 |
+
"acc": 0.9083333333333333,
|
| 1095 |
+
"n": 480,
|
| 1096 |
+
"ci95": [
|
| 1097 |
+
0.8791758375690445,
|
| 1098 |
+
0.931006668441187
|
| 1099 |
+
]
|
| 1100 |
+
},
|
| 1101 |
+
"ilo": {
|
| 1102 |
+
"acc": 0.8916666666666667,
|
| 1103 |
+
"n": 480,
|
| 1104 |
+
"ci95": [
|
| 1105 |
+
0.8606887969411983,
|
| 1106 |
+
0.91642503535433
|
| 1107 |
+
]
|
| 1108 |
+
},
|
| 1109 |
+
"ind": {
|
| 1110 |
+
"acc": 0.8666666666666667,
|
| 1111 |
+
"n": 480,
|
| 1112 |
+
"ci95": [
|
| 1113 |
+
0.8333257856869319,
|
| 1114 |
+
0.8941850360365412
|
| 1115 |
+
]
|
| 1116 |
+
},
|
| 1117 |
+
"len2": {
|
| 1118 |
+
"acc": 0.6240740740740741,
|
| 1119 |
+
"n": 1080,
|
| 1120 |
+
"ci95": [
|
| 1121 |
+
0.5947944606747045,
|
| 1122 |
+
0.6524741438889144
|
| 1123 |
+
]
|
| 1124 |
+
},
|
| 1125 |
+
"len3": {
|
| 1126 |
+
"acc": 0.7388888888888889,
|
| 1127 |
+
"n": 1080,
|
| 1128 |
+
"ci95": [
|
| 1129 |
+
0.7118782358681339,
|
| 1130 |
+
0.7642060923215204
|
| 1131 |
+
]
|
| 1132 |
+
},
|
| 1133 |
+
"len4": {
|
| 1134 |
+
"acc": 0.8351851851851851,
|
| 1135 |
+
"n": 1080,
|
| 1136 |
+
"ci95": [
|
| 1137 |
+
0.8118768985265994,
|
| 1138 |
+
0.8561173914139233
|
| 1139 |
+
]
|
| 1140 |
+
},
|
| 1141 |
+
"len5": {
|
| 1142 |
+
"acc": 0.8870370370370371,
|
| 1143 |
+
"n": 1080,
|
| 1144 |
+
"ci95": [
|
| 1145 |
+
0.8667696567362586,
|
| 1146 |
+
0.9045607664547317
|
| 1147 |
+
]
|
| 1148 |
+
},
|
| 1149 |
+
"msa": {
|
| 1150 |
+
"acc": 0.3416666666666667,
|
| 1151 |
+
"n": 480,
|
| 1152 |
+
"ci95": [
|
| 1153 |
+
0.3006451830920638,
|
| 1154 |
+
0.3852024166182546
|
| 1155 |
+
]
|
| 1156 |
+
},
|
| 1157 |
+
"pag": {
|
| 1158 |
+
"acc": 0.8229166666666666,
|
| 1159 |
+
"n": 480,
|
| 1160 |
+
"ci95": [
|
| 1161 |
+
0.7862412266386819,
|
| 1162 |
+
0.854464326583695
|
| 1163 |
+
]
|
| 1164 |
+
},
|
| 1165 |
+
"spa": {
|
| 1166 |
+
"acc": 0.85,
|
| 1167 |
+
"n": 480,
|
| 1168 |
+
"ci95": [
|
| 1169 |
+
0.8152829232916574,
|
| 1170 |
+
0.8791592247171123
|
| 1171 |
+
]
|
| 1172 |
+
},
|
| 1173 |
+
"tgl": {
|
| 1174 |
+
"acc": 0.8666666666666667,
|
| 1175 |
+
"n": 480,
|
| 1176 |
+
"ci95": [
|
| 1177 |
+
0.8333257856869319,
|
| 1178 |
+
0.8941850360365412
|
| 1179 |
+
]
|
| 1180 |
+
},
|
| 1181 |
+
"war": {
|
| 1182 |
+
"acc": 0.6333333333333333,
|
| 1183 |
+
"n": 480,
|
| 1184 |
+
"ci95": [
|
| 1185 |
+
0.5893222088010526,
|
| 1186 |
+
0.675227180916574
|
| 1187 |
+
]
|
| 1188 |
+
}
|
| 1189 |
+
}
|
| 1190 |
+
},
|
| 1191 |
+
"openlid3": {
|
| 1192 |
+
"acc": 0.7733796296296296,
|
| 1193 |
+
"ci95": [
|
| 1194 |
+
0.7606557433895734,
|
| 1195 |
+
0.7856177371190557
|
| 1196 |
+
],
|
| 1197 |
+
"by": {
|
| 1198 |
+
"ceb": {
|
| 1199 |
+
"acc": 0.8416666666666667,
|
| 1200 |
+
"n": 480,
|
| 1201 |
+
"ci95": [
|
| 1202 |
+
0.8063127326027417,
|
| 1203 |
+
0.8715950785486763
|
| 1204 |
+
]
|
| 1205 |
+
},
|
| 1206 |
+
"eng": {
|
| 1207 |
+
"acc": 0.7166666666666667,
|
| 1208 |
+
"n": 480,
|
| 1209 |
+
"ci95": [
|
| 1210 |
+
0.6747571073684372,
|
| 1211 |
+
0.755135650922706
|
| 1212 |
+
]
|
| 1213 |
+
},
|
| 1214 |
+
"ilo": {
|
| 1215 |
+
"acc": 0.9333333333333333,
|
| 1216 |
+
"n": 480,
|
| 1217 |
+
"ci95": [
|
| 1218 |
+
0.9074012300031347,
|
| 1219 |
+
0.9523842865791519
|
| 1220 |
+
]
|
| 1221 |
+
},
|
| 1222 |
+
"ind": {
|
| 1223 |
+
"acc": 0.8104166666666667,
|
| 1224 |
+
"n": 480,
|
| 1225 |
+
"ci95": [
|
| 1226 |
+
0.7729383775947093,
|
| 1227 |
+
0.84296567034164
|
| 1228 |
+
]
|
| 1229 |
+
},
|
| 1230 |
+
"len2": {
|
| 1231 |
+
"acc": 0.6212962962962963,
|
| 1232 |
+
"n": 1080,
|
| 1233 |
+
"ci95": [
|
| 1234 |
+
0.5919848236309152,
|
| 1235 |
+
0.6497479166514287
|
| 1236 |
+
]
|
| 1237 |
+
},
|
| 1238 |
+
"len3": {
|
| 1239 |
+
"acc": 0.75,
|
| 1240 |
+
"n": 1080,
|
| 1241 |
+
"ci95": [
|
| 1242 |
+
0.7233192092286217,
|
| 1243 |
+
0.7749085760861328
|
| 1244 |
+
]
|
| 1245 |
+
},
|
| 1246 |
+
"len4": {
|
| 1247 |
+
"acc": 0.8342592592592593,
|
| 1248 |
+
"n": 1080,
|
| 1249 |
+
"ci95": [
|
| 1250 |
+
0.8109048245345469,
|
| 1251 |
+
0.8552441773122175
|
| 1252 |
+
]
|
| 1253 |
+
},
|
| 1254 |
+
"len5": {
|
| 1255 |
+
"acc": 0.887962962962963,
|
| 1256 |
+
"n": 1080,
|
| 1257 |
+
"ci95": [
|
| 1258 |
+
0.8677594847017363,
|
| 1259 |
+
0.9054162265830124
|
| 1260 |
+
]
|
| 1261 |
+
},
|
| 1262 |
+
"msa": {
|
| 1263 |
+
"acc": 0.55,
|
| 1264 |
+
"n": 480,
|
| 1265 |
+
"ci95": [
|
| 1266 |
+
0.5052718108484551,
|
| 1267 |
+
0.5939342102956551
|
| 1268 |
+
]
|
| 1269 |
+
},
|
| 1270 |
+
"pag": {
|
| 1271 |
+
"acc": 0.7895833333333333,
|
| 1272 |
+
"n": 480,
|
| 1273 |
+
"ci95": [
|
| 1274 |
+
0.7508915928659468,
|
| 1275 |
+
0.8236766129270234
|
| 1276 |
+
]
|
| 1277 |
+
},
|
| 1278 |
+
"spa": {
|
| 1279 |
+
"acc": 0.6916666666666667,
|
| 1280 |
+
"n": 480,
|
| 1281 |
+
"ci95": [
|
| 1282 |
+
0.6489673795407492,
|
| 1283 |
+
0.7313223681783391
|
| 1284 |
+
]
|
| 1285 |
+
},
|
| 1286 |
+
"tgl": {
|
| 1287 |
+
"acc": 0.8875,
|
| 1288 |
+
"n": 480,
|
| 1289 |
+
"ci95": [
|
| 1290 |
+
0.8561001232465634,
|
| 1291 |
+
0.9127465406202887
|
| 1292 |
+
]
|
| 1293 |
+
},
|
| 1294 |
+
"war": {
|
| 1295 |
+
"acc": 0.7395833333333334,
|
| 1296 |
+
"n": 480,
|
| 1297 |
+
"ci95": [
|
| 1298 |
+
0.6985298417841903,
|
| 1299 |
+
0.77683234286467
|
| 1300 |
+
]
|
| 1301 |
+
}
|
| 1302 |
+
}
|
| 1303 |
+
},
|
| 1304 |
+
"lid176": {
|
| 1305 |
+
"acc": 0.5900462962962963,
|
| 1306 |
+
"ci95": [
|
| 1307 |
+
0.5753061534658956,
|
| 1308 |
+
0.6046264324086656
|
| 1309 |
+
],
|
| 1310 |
+
"by": {
|
| 1311 |
+
"ceb": {
|
| 1312 |
+
"acc": 0.68125,
|
| 1313 |
+
"n": 480,
|
| 1314 |
+
"ci95": [
|
| 1315 |
+
0.6382635469798807,
|
| 1316 |
+
0.721358279667518
|
| 1317 |
+
]
|
| 1318 |
+
},
|
| 1319 |
+
"eng": {
|
| 1320 |
+
"acc": 0.9916666666666667,
|
| 1321 |
+
"n": 480,
|
| 1322 |
+
"ci95": [
|
| 1323 |
+
0.9787711280476984,
|
| 1324 |
+
0.9967547465360497
|
| 1325 |
+
]
|
| 1326 |
+
},
|
| 1327 |
+
"ilo": {
|
| 1328 |
+
"acc": 0.5458333333333333,
|
| 1329 |
+
"n": 480,
|
| 1330 |
+
"ci95": [
|
| 1331 |
+
0.5011027665973042,
|
| 1332 |
+
0.5898360861181298
|
| 1333 |
+
]
|
| 1334 |
+
},
|
| 1335 |
+
"ind": {
|
| 1336 |
+
"acc": 0.75,
|
| 1337 |
+
"n": 480,
|
| 1338 |
+
"ci95": [
|
| 1339 |
+
0.709380219226088,
|
| 1340 |
+
0.7866498864944619
|
| 1341 |
+
]
|
| 1342 |
+
},
|
| 1343 |
+
"len2": {
|
| 1344 |
+
"acc": 0.5138888888888888,
|
| 1345 |
+
"n": 1080,
|
| 1346 |
+
"ci95": [
|
| 1347 |
+
0.4840835513937429,
|
| 1348 |
+
0.5435957700126323
|
| 1349 |
+
]
|
| 1350 |
+
},
|
| 1351 |
+
"len3": {
|
| 1352 |
+
"acc": 0.5472222222222223,
|
| 1353 |
+
"n": 1080,
|
| 1354 |
+
"ci95": [
|
| 1355 |
+
0.517419875518785,
|
| 1356 |
+
0.576689817262891
|
| 1357 |
+
]
|
| 1358 |
+
},
|
| 1359 |
+
"len4": {
|
| 1360 |
+
"acc": 0.6277777777777778,
|
| 1361 |
+
"n": 1080,
|
| 1362 |
+
"ci95": [
|
| 1363 |
+
0.5985422101910842,
|
| 1364 |
+
0.6561075467475681
|
| 1365 |
+
]
|
| 1366 |
+
},
|
| 1367 |
+
"len5": {
|
| 1368 |
+
"acc": 0.6712962962962963,
|
| 1369 |
+
"n": 1080,
|
| 1370 |
+
"ci95": [
|
| 1371 |
+
0.6427164149552664,
|
| 1372 |
+
0.6986618823900284
|
| 1373 |
+
]
|
| 1374 |
+
},
|
| 1375 |
+
"msa": {
|
| 1376 |
+
"acc": 0.3541666666666667,
|
| 1377 |
+
"n": 480,
|
| 1378 |
+
"ci95": [
|
| 1379 |
+
0.3126932319826662,
|
| 1380 |
+
0.39795587301367974
|
| 1381 |
+
]
|
| 1382 |
+
},
|
| 1383 |
+
"pag": {
|
| 1384 |
+
"acc": 0.0,
|
| 1385 |
+
"n": 480,
|
| 1386 |
+
"ci95": [
|
| 1387 |
+
-8.673617379884035e-19,
|
| 1388 |
+
0.00793978855890027
|
| 1389 |
+
]
|
| 1390 |
+
},
|
| 1391 |
+
"spa": {
|
| 1392 |
+
"acc": 0.9333333333333333,
|
| 1393 |
+
"n": 480,
|
| 1394 |
+
"ci95": [
|
| 1395 |
+
0.9074012300031347,
|
| 1396 |
+
0.9523842865791519
|
| 1397 |
+
]
|
| 1398 |
+
},
|
| 1399 |
+
"tgl": {
|
| 1400 |
+
"acc": 0.7291666666666666,
|
| 1401 |
+
"n": 480,
|
| 1402 |
+
"ci95": [
|
| 1403 |
+
0.687707743498603,
|
| 1404 |
+
0.7669865200785677
|
| 1405 |
+
]
|
| 1406 |
+
},
|
| 1407 |
+
"war": {
|
| 1408 |
+
"acc": 0.325,
|
| 1409 |
+
"n": 480,
|
| 1410 |
+
"ci95": [
|
| 1411 |
+
0.28463156886351215,
|
| 1412 |
+
0.368147357132103
|
| 1413 |
+
]
|
| 1414 |
+
}
|
| 1415 |
+
}
|
| 1416 |
+
}
|
| 1417 |
+
},
|
| 1418 |
+
"t_caps": {
|
| 1419 |
+
"n": 5388,
|
| 1420 |
+
"wikalid-1.0.1": {
|
| 1421 |
+
"acc": 0.9988864142538976,
|
| 1422 |
+
"ci95": [
|
| 1423 |
+
0.9975723876556302,
|
| 1424 |
+
0.9994895438373128
|
| 1425 |
+
],
|
| 1426 |
+
"by": {
|
| 1427 |
+
"ceb": {
|
| 1428 |
+
"acc": 0.9948914431673053,
|
| 1429 |
+
"n": 783,
|
| 1430 |
+
"ci95": [
|
| 1431 |
+
0.986938790959379,
|
| 1432 |
+
0.9980116742930938
|
| 1433 |
+
]
|
| 1434 |
+
},
|
| 1435 |
+
"eng": {
|
| 1436 |
+
"acc": 1.0,
|
| 1437 |
+
"n": 765,
|
| 1438 |
+
"ci95": [
|
| 1439 |
+
0.9950033921161395,
|
| 1440 |
+
1.0000000000000002
|
| 1441 |
+
]
|
| 1442 |
+
},
|
| 1443 |
+
"ilo": {
|
| 1444 |
+
"acc": 1.0,
|
| 1445 |
+
"n": 699,
|
| 1446 |
+
"ci95": [
|
| 1447 |
+
0.9945341880731021,
|
| 1448 |
+
1.0
|
| 1449 |
+
]
|
| 1450 |
+
},
|
| 1451 |
+
"lower": {
|
| 1452 |
+
"acc": 0.9983296213808464,
|
| 1453 |
+
"n": 1796,
|
| 1454 |
+
"ci95": [
|
| 1455 |
+
0.9951001914542256,
|
| 1456 |
+
0.9994317718029856
|
| 1457 |
+
]
|
| 1458 |
+
},
|
| 1459 |
+
"pag": {
|
| 1460 |
+
"acc": 0.9987080103359173,
|
| 1461 |
+
"n": 774,
|
| 1462 |
+
"ci95": [
|
| 1463 |
+
0.9927180861293878,
|
| 1464 |
+
0.9997719015493376
|
| 1465 |
+
]
|
| 1466 |
+
},
|
| 1467 |
+
"spa": {
|
| 1468 |
+
"acc": 1.0,
|
| 1469 |
+
"n": 786,
|
| 1470 |
+
"ci95": [
|
| 1471 |
+
0.9951362399752052,
|
| 1472 |
+
1.0
|
| 1473 |
+
]
|
| 1474 |
+
},
|
| 1475 |
+
"tgl": {
|
| 1476 |
+
"acc": 0.99867197875166,
|
| 1477 |
+
"n": 753,
|
| 1478 |
+
"ci95": [
|
| 1479 |
+
0.9925160690054392,
|
| 1480 |
+
0.9997655391936874
|
| 1481 |
+
]
|
| 1482 |
+
},
|
| 1483 |
+
"title": {
|
| 1484 |
+
"acc": 0.9988864142538976,
|
| 1485 |
+
"n": 1796,
|
| 1486 |
+
"ci95": [
|
| 1487 |
+
0.9959486042757345,
|
| 1488 |
+
0.9996945678789707
|
| 1489 |
+
]
|
| 1490 |
+
},
|
| 1491 |
+
"upper": {
|
| 1492 |
+
"acc": 0.9994432071269488,
|
| 1493 |
+
"n": 1796,
|
| 1494 |
+
"ci95": [
|
| 1495 |
+
0.9968526729003299,
|
| 1496 |
+
0.9999017081518694
|
| 1497 |
+
]
|
| 1498 |
+
},
|
| 1499 |
+
"war": {
|
| 1500 |
+
"acc": 1.0,
|
| 1501 |
+
"n": 828,
|
| 1502 |
+
"ci95": [
|
| 1503 |
+
0.9953818130759511,
|
| 1504 |
+
1.0
|
| 1505 |
+
]
|
| 1506 |
+
}
|
| 1507 |
+
}
|
| 1508 |
+
},
|
| 1509 |
+
"glotlid": {
|
| 1510 |
+
"acc": 0.9751299183370453,
|
| 1511 |
+
"ci95": [
|
| 1512 |
+
0.9706208513243705,
|
| 1513 |
+
0.9789619405736706
|
| 1514 |
+
],
|
| 1515 |
+
"by": {
|
| 1516 |
+
"ceb": {
|
| 1517 |
+
"acc": 0.9591315453384419,
|
| 1518 |
+
"n": 783,
|
| 1519 |
+
"ci95": [
|
| 1520 |
+
0.9428755574692792,
|
| 1521 |
+
0.970904293519789
|
| 1522 |
+
]
|
| 1523 |
+
},
|
| 1524 |
+
"eng": {
|
| 1525 |
+
"acc": 0.930718954248366,
|
| 1526 |
+
"n": 765,
|
| 1527 |
+
"ci95": [
|
| 1528 |
+
0.9104886883171216,
|
| 1529 |
+
0.9466449527345594
|
| 1530 |
+
]
|
| 1531 |
+
},
|
| 1532 |
+
"ilo": {
|
| 1533 |
+
"acc": 0.9985693848354793,
|
| 1534 |
+
"n": 699,
|
| 1535 |
+
"ci95": [
|
| 1536 |
+
0.9919411735524428,
|
| 1537 |
+
0.999747423138476
|
| 1538 |
+
]
|
| 1539 |
+
},
|
| 1540 |
+
"lower": {
|
| 1541 |
+
"acc": 0.9983296213808464,
|
| 1542 |
+
"n": 1796,
|
| 1543 |
+
"ci95": [
|
| 1544 |
+
0.9951001914542256,
|
| 1545 |
+
0.9994317718029856
|
| 1546 |
+
]
|
| 1547 |
+
},
|
| 1548 |
+
"pag": {
|
| 1549 |
+
"acc": 0.9935400516795866,
|
| 1550 |
+
"n": 774,
|
| 1551 |
+
"ci95": [
|
| 1552 |
+
0.9849674441294312,
|
| 1553 |
+
0.9972376732106547
|
| 1554 |
+
]
|
| 1555 |
+
},
|
| 1556 |
+
"spa": {
|
| 1557 |
+
"acc": 0.9491094147582697,
|
| 1558 |
+
"n": 786,
|
| 1559 |
+
"ci95": [
|
| 1560 |
+
0.9314429805910065,
|
| 1561 |
+
0.9624071280890125
|
| 1562 |
+
]
|
| 1563 |
+
},
|
| 1564 |
+
"tgl": {
|
| 1565 |
+
"acc": 0.9960159362549801,
|
| 1566 |
+
"n": 753,
|
| 1567 |
+
"ci95": [
|
| 1568 |
+
0.9883523084517515,
|
| 1569 |
+
0.9986441779993106
|
| 1570 |
+
]
|
| 1571 |
+
},
|
| 1572 |
+
"title": {
|
| 1573 |
+
"acc": 0.9899777282850779,
|
| 1574 |
+
"n": 1796,
|
| 1575 |
+
"ci95": [
|
| 1576 |
+
0.9842127020060506,
|
| 1577 |
+
0.9936511277887491
|
| 1578 |
+
]
|
| 1579 |
+
},
|
| 1580 |
+
"upper": {
|
| 1581 |
+
"acc": 0.9370824053452116,
|
| 1582 |
+
"n": 1796,
|
| 1583 |
+
"ci95": [
|
| 1584 |
+
0.9248928077217875,
|
| 1585 |
+
0.9474061768110735
|
| 1586 |
+
]
|
| 1587 |
+
},
|
| 1588 |
+
"war": {
|
| 1589 |
+
"acc": 1.0,
|
| 1590 |
+
"n": 828,
|
| 1591 |
+
"ci95": [
|
| 1592 |
+
0.9953818130759511,
|
| 1593 |
+
1.0
|
| 1594 |
+
]
|
| 1595 |
+
}
|
| 1596 |
+
}
|
| 1597 |
+
},
|
| 1598 |
+
"openlid3": {
|
| 1599 |
+
"acc": 0.5749814402375649,
|
| 1600 |
+
"ci95": [
|
| 1601 |
+
0.5617326363392661,
|
| 1602 |
+
0.5881233980071434
|
| 1603 |
+
],
|
| 1604 |
+
"by": {
|
| 1605 |
+
"ceb": {
|
| 1606 |
+
"acc": 0.6462324393358876,
|
| 1607 |
+
"n": 783,
|
| 1608 |
+
"ci95": [
|
| 1609 |
+
0.612101682005147,
|
| 1610 |
+
0.678935294179132
|
| 1611 |
+
]
|
| 1612 |
+
},
|
| 1613 |
+
"eng": {
|
| 1614 |
+
"acc": 0.4549019607843137,
|
| 1615 |
+
"n": 765,
|
| 1616 |
+
"ci95": [
|
| 1617 |
+
0.4199272911685178,
|
| 1618 |
+
0.4903273048366933
|
| 1619 |
+
]
|
| 1620 |
+
},
|
| 1621 |
+
"ilo": {
|
| 1622 |
+
"acc": 0.6609442060085837,
|
| 1623 |
+
"n": 699,
|
| 1624 |
+
"ci95": [
|
| 1625 |
+
0.6250552920839941,
|
| 1626 |
+
0.6950737384116398
|
| 1627 |
+
]
|
| 1628 |
+
},
|
| 1629 |
+
"lower": {
|
| 1630 |
+
"acc": 0.9988864142538976,
|
| 1631 |
+
"n": 1796,
|
| 1632 |
+
"ci95": [
|
| 1633 |
+
0.9959486042757345,
|
| 1634 |
+
0.9996945678789707
|
| 1635 |
+
]
|
| 1636 |
+
},
|
| 1637 |
+
"pag": {
|
| 1638 |
+
"acc": 0.6085271317829457,
|
| 1639 |
+
"n": 774,
|
| 1640 |
+
"ci95": [
|
| 1641 |
+
0.573686384289186,
|
| 1642 |
+
0.6422958928222464
|
| 1643 |
+
]
|
| 1644 |
+
},
|
| 1645 |
+
"spa": {
|
| 1646 |
+
"acc": 0.5458015267175572,
|
| 1647 |
+
"n": 786,
|
| 1648 |
+
"ci95": [
|
| 1649 |
+
0.5108543219550099,
|
| 1650 |
+
0.5803031962106577
|
| 1651 |
+
]
|
| 1652 |
+
},
|
| 1653 |
+
"tgl": {
|
| 1654 |
+
"acc": 0.6308100929614874,
|
| 1655 |
+
"n": 753,
|
| 1656 |
+
"ci95": [
|
| 1657 |
+
0.5957579765214207,
|
| 1658 |
+
0.6645342695707603
|
| 1659 |
+
]
|
| 1660 |
+
},
|
| 1661 |
+
"title": {
|
| 1662 |
+
"acc": 0.7260579064587973,
|
| 1663 |
+
"n": 1796,
|
| 1664 |
+
"ci95": [
|
| 1665 |
+
0.704965643486489,
|
| 1666 |
+
0.7461851688961116
|
| 1667 |
+
]
|
| 1668 |
+
},
|
| 1669 |
+
"upper": {
|
| 1670 |
+
"acc": 0.0,
|
| 1671 |
+
"n": 1796,
|
| 1672 |
+
"ci95": [
|
| 1673 |
+
0.0,
|
| 1674 |
+
0.002134410050306649
|
| 1675 |
+
]
|
| 1676 |
+
},
|
| 1677 |
+
"war": {
|
| 1678 |
+
"acc": 0.4915458937198068,
|
| 1679 |
+
"n": 828,
|
| 1680 |
+
"ci95": [
|
| 1681 |
+
0.4576111496341992,
|
| 1682 |
+
0.5255587230915697
|
| 1683 |
+
]
|
| 1684 |
+
}
|
| 1685 |
+
}
|
| 1686 |
+
},
|
| 1687 |
+
"lid176": {
|
| 1688 |
+
"acc": 0.5215293244246474,
|
| 1689 |
+
"ci95": [
|
| 1690 |
+
0.5081801578367945,
|
| 1691 |
+
0.5348478124210855
|
| 1692 |
+
],
|
| 1693 |
+
"by": {
|
| 1694 |
+
"ceb": {
|
| 1695 |
+
"acc": 0.4661558109833972,
|
| 1696 |
+
"n": 783,
|
| 1697 |
+
"ci95": [
|
| 1698 |
+
0.4314640406362553,
|
| 1699 |
+
0.501178056578737
|
| 1700 |
+
]
|
| 1701 |
+
},
|
| 1702 |
+
"eng": {
|
| 1703 |
+
"acc": 0.9647058823529412,
|
| 1704 |
+
"n": 765,
|
| 1705 |
+
"ci95": [
|
| 1706 |
+
0.9491356016719222,
|
| 1707 |
+
0.9756322568830781
|
| 1708 |
+
]
|
| 1709 |
+
},
|
| 1710 |
+
"ilo": {
|
| 1711 |
+
"acc": 0.43061516452074394,
|
| 1712 |
+
"n": 699,
|
| 1713 |
+
"ci95": [
|
| 1714 |
+
0.3943845333535883,
|
| 1715 |
+
0.46760428461051645
|
| 1716 |
+
]
|
| 1717 |
+
},
|
| 1718 |
+
"lower": {
|
| 1719 |
+
"acc": 0.7878619153674833,
|
| 1720 |
+
"n": 1796,
|
| 1721 |
+
"ci95": [
|
| 1722 |
+
0.7683500600977665,
|
| 1723 |
+
0.8061449399066783
|
| 1724 |
+
]
|
| 1725 |
+
},
|
| 1726 |
+
"pag": {
|
| 1727 |
+
"acc": 0.0,
|
| 1728 |
+
"n": 774,
|
| 1729 |
+
"ci95": [
|
| 1730 |
+
0.0,
|
| 1731 |
+
0.004938794736614755
|
| 1732 |
+
]
|
| 1733 |
+
},
|
| 1734 |
+
"spa": {
|
| 1735 |
+
"acc": 0.9936386768447837,
|
| 1736 |
+
"n": 786,
|
| 1737 |
+
"ci95": [
|
| 1738 |
+
0.9851955812525237,
|
| 1739 |
+
0.9972798923107831
|
| 1740 |
+
]
|
| 1741 |
+
},
|
| 1742 |
+
"tgl": {
|
| 1743 |
+
"acc": 0.5816733067729084,
|
| 1744 |
+
"n": 753,
|
| 1745 |
+
"ci95": [
|
| 1746 |
+
0.5461123229613584,
|
| 1747 |
+
0.6164051707916277
|
| 1748 |
+
]
|
| 1749 |
+
},
|
| 1750 |
+
"title": {
|
| 1751 |
+
"acc": 0.5066815144766147,
|
| 1752 |
+
"n": 1796,
|
| 1753 |
+
"ci95": [
|
| 1754 |
+
0.48356947361579444,
|
| 1755 |
+
0.5297650331541347
|
| 1756 |
+
]
|
| 1757 |
+
},
|
| 1758 |
+
"upper": {
|
| 1759 |
+
"acc": 0.2700445434298441,
|
| 1760 |
+
"n": 1796,
|
| 1761 |
+
"ci95": [
|
| 1762 |
+
0.2500176421747124,
|
| 1763 |
+
0.2910530831602282
|
| 1764 |
+
]
|
| 1765 |
+
},
|
| 1766 |
+
"war": {
|
| 1767 |
+
"acc": 0.22584541062801933,
|
| 1768 |
+
"n": 828,
|
| 1769 |
+
"ci95": [
|
| 1770 |
+
0.19866781968925123,
|
| 1771 |
+
0.25555519584639885
|
| 1772 |
+
]
|
| 1773 |
+
}
|
| 1774 |
+
}
|
| 1775 |
+
}
|
| 1776 |
+
},
|
| 1777 |
+
"t_bikol": {
|
| 1778 |
+
"n": 1800,
|
| 1779 |
+
"wikalid-1.0.1": {
|
| 1780 |
+
"acc": 0.94,
|
| 1781 |
+
"ci95": [
|
| 1782 |
+
0.9280633120585668,
|
| 1783 |
+
0.9500625721654137
|
| 1784 |
+
],
|
| 1785 |
+
"by": {
|
| 1786 |
+
"bcl": {
|
| 1787 |
+
"acc": 0.9611111111111111,
|
| 1788 |
+
"n": 900,
|
| 1789 |
+
"ci95": [
|
| 1790 |
+
0.9463957540769419,
|
| 1791 |
+
0.9719067449450107
|
| 1792 |
+
]
|
| 1793 |
+
},
|
| 1794 |
+
"ceb": {
|
| 1795 |
+
"acc": 0.9188888888888889,
|
| 1796 |
+
"n": 900,
|
| 1797 |
+
"ci95": [
|
| 1798 |
+
0.8992212323860316,
|
| 1799 |
+
0.934995736604995
|
| 1800 |
+
]
|
| 1801 |
+
},
|
| 1802 |
+
"story": {
|
| 1803 |
+
"acc": 0.94,
|
| 1804 |
+
"n": 1800,
|
| 1805 |
+
"ci95": [
|
| 1806 |
+
0.9280633120585668,
|
| 1807 |
+
0.9500625721654137
|
| 1808 |
+
]
|
| 1809 |
+
}
|
| 1810 |
+
}
|
| 1811 |
+
},
|
| 1812 |
+
"glotlid": {
|
| 1813 |
+
"acc": 0.8933333333333333,
|
| 1814 |
+
"ci95": [
|
| 1815 |
+
0.878225542898005,
|
| 1816 |
+
0.9067657778476745
|
| 1817 |
+
],
|
| 1818 |
+
"by": {
|
| 1819 |
+
"bcl": {
|
| 1820 |
+
"acc": 0.8944444444444445,
|
| 1821 |
+
"n": 900,
|
| 1822 |
+
"ci95": [
|
| 1823 |
+
0.8726657739209606,
|
| 1824 |
+
0.9128700987363724
|
| 1825 |
+
]
|
| 1826 |
+
},
|
| 1827 |
+
"ceb": {
|
| 1828 |
+
"acc": 0.8922222222222222,
|
| 1829 |
+
"n": 900,
|
| 1830 |
+
"ci95": [
|
| 1831 |
+
0.8702698070914948,
|
| 1832 |
+
0.910840511353684
|
| 1833 |
+
]
|
| 1834 |
+
},
|
| 1835 |
+
"story": {
|
| 1836 |
+
"acc": 0.8933333333333333,
|
| 1837 |
+
"n": 1800,
|
| 1838 |
+
"ci95": [
|
| 1839 |
+
0.878225542898005,
|
| 1840 |
+
0.9067657778476745
|
| 1841 |
+
]
|
| 1842 |
+
}
|
| 1843 |
+
}
|
| 1844 |
+
},
|
| 1845 |
+
"openlid3": {
|
| 1846 |
+
"acc": 0.4666666666666667,
|
| 1847 |
+
"ci95": [
|
| 1848 |
+
0.4437146676882942,
|
| 1849 |
+
0.48976064411291936
|
| 1850 |
+
],
|
| 1851 |
+
"by": {
|
| 1852 |
+
"bcl": {
|
| 1853 |
+
"acc": 0.0,
|
| 1854 |
+
"n": 900,
|
| 1855 |
+
"ci95": [
|
| 1856 |
+
4.336808689942018e-19,
|
| 1857 |
+
0.004250302265352689
|
| 1858 |
+
]
|
| 1859 |
+
},
|
| 1860 |
+
"ceb": {
|
| 1861 |
+
"acc": 0.9333333333333333,
|
| 1862 |
+
"n": 900,
|
| 1863 |
+
"ci95": [
|
| 1864 |
+
0.9151252450043573,
|
| 1865 |
+
0.9478578263656704
|
| 1866 |
+
]
|
| 1867 |
+
},
|
| 1868 |
+
"story": {
|
| 1869 |
+
"acc": 0.4666666666666667,
|
| 1870 |
+
"n": 1800,
|
| 1871 |
+
"ci95": [
|
| 1872 |
+
0.4437146676882942,
|
| 1873 |
+
0.48976064411291936
|
| 1874 |
+
]
|
| 1875 |
+
}
|
| 1876 |
+
}
|
| 1877 |
+
},
|
| 1878 |
+
"lid176": {
|
| 1879 |
+
"acc": 0.2911111111111111,
|
| 1880 |
+
"ci95": [
|
| 1881 |
+
0.27058719962290767,
|
| 1882 |
+
0.3125247543313642
|
| 1883 |
+
],
|
| 1884 |
+
"by": {
|
| 1885 |
+
"bcl": {
|
| 1886 |
+
"acc": 0.0033333333333333335,
|
| 1887 |
+
"n": 900,
|
| 1888 |
+
"ci95": [
|
| 1889 |
+
0.001134247210102744,
|
| 1890 |
+
0.009754386373480927
|
| 1891 |
+
]
|
| 1892 |
+
},
|
| 1893 |
+
"ceb": {
|
| 1894 |
+
"acc": 0.5788888888888889,
|
| 1895 |
+
"n": 900,
|
| 1896 |
+
"ci95": [
|
| 1897 |
+
0.5463629609744777,
|
| 1898 |
+
0.6107442135569888
|
| 1899 |
+
]
|
| 1900 |
+
},
|
| 1901 |
+
"story": {
|
| 1902 |
+
"acc": 0.2911111111111111,
|
| 1903 |
+
"n": 1800,
|
| 1904 |
+
"ci95": [
|
| 1905 |
+
0.27058719962290767,
|
| 1906 |
+
0.3125247543313642
|
| 1907 |
+
]
|
| 1908 |
+
}
|
| 1909 |
+
}
|
| 1910 |
+
}
|
| 1911 |
+
},
|
| 1912 |
+
"t_aklsgd": {
|
| 1913 |
+
"n": 110,
|
| 1914 |
+
"wikalid-1.0.1": {
|
| 1915 |
+
"acc": 0.34545454545454546,
|
| 1916 |
+
"ci95": [
|
| 1917 |
+
0.26316254590275023,
|
| 1918 |
+
0.4381768590247983
|
| 1919 |
+
],
|
| 1920 |
+
"by": {
|
| 1921 |
+
"incubator": {
|
| 1922 |
+
"acc": 0.3695652173913043,
|
| 1923 |
+
"n": 92,
|
| 1924 |
+
"ci95": [
|
| 1925 |
+
0.27801481252642296,
|
| 1926 |
+
0.4715720057236897
|
| 1927 |
+
]
|
| 1928 |
+
},
|
| 1929 |
+
"oth": {
|
| 1930 |
+
"acc": 0.34545454545454546,
|
| 1931 |
+
"n": 110,
|
| 1932 |
+
"ci95": [
|
| 1933 |
+
0.26316254590275023,
|
| 1934 |
+
0.4381768590247983
|
| 1935 |
+
]
|
| 1936 |
+
},
|
| 1937 |
+
"tatoeba": {
|
| 1938 |
+
"acc": 0.2222222222222222,
|
| 1939 |
+
"n": 18,
|
| 1940 |
+
"ci95": [
|
| 1941 |
+
0.09000781913382,
|
| 1942 |
+
0.4521502645230549
|
| 1943 |
+
]
|
| 1944 |
+
}
|
| 1945 |
+
}
|
| 1946 |
+
},
|
| 1947 |
+
"glotlid": {
|
| 1948 |
+
"acc": 0.3,
|
| 1949 |
+
"ci95": [
|
| 1950 |
+
0.22229769508028016,
|
| 1951 |
+
0.3912003583553708
|
| 1952 |
+
],
|
| 1953 |
+
"by": {
|
| 1954 |
+
"incubator": {
|
| 1955 |
+
"acc": 0.2717391304347826,
|
| 1956 |
+
"n": 92,
|
| 1957 |
+
"ci95": [
|
| 1958 |
+
0.19135648353357726,
|
| 1959 |
+
0.3704204484041199
|
| 1960 |
+
]
|
| 1961 |
+
},
|
| 1962 |
+
"oth": {
|
| 1963 |
+
"acc": 0.3,
|
| 1964 |
+
"n": 110,
|
| 1965 |
+
"ci95": [
|
| 1966 |
+
0.22229769508028016,
|
| 1967 |
+
0.3912003583553708
|
| 1968 |
+
]
|
| 1969 |
+
},
|
| 1970 |
+
"tatoeba": {
|
| 1971 |
+
"acc": 0.4444444444444444,
|
| 1972 |
+
"n": 18,
|
| 1973 |
+
"ci95": [
|
| 1974 |
+
0.24559231101389126,
|
| 1975 |
+
0.6628393057174837
|
| 1976 |
+
]
|
| 1977 |
+
}
|
| 1978 |
+
}
|
| 1979 |
+
},
|
| 1980 |
+
"openlid3": {
|
| 1981 |
+
"acc": 0.08181818181818182,
|
| 1982 |
+
"ci95": [
|
| 1983 |
+
0.04364015135431728,
|
| 1984 |
+
0.14821941492022558
|
| 1985 |
+
],
|
| 1986 |
+
"by": {
|
| 1987 |
+
"incubator": {
|
| 1988 |
+
"acc": 0.043478260869565216,
|
| 1989 |
+
"n": 92,
|
| 1990 |
+
"ci95": [
|
| 1991 |
+
0.01703544061158062,
|
| 1992 |
+
0.10651842326381375
|
| 1993 |
+
]
|
| 1994 |
+
},
|
| 1995 |
+
"oth": {
|
| 1996 |
+
"acc": 0.08181818181818182,
|
| 1997 |
+
"n": 110,
|
| 1998 |
+
"ci95": [
|
| 1999 |
+
0.04364015135431728,
|
| 2000 |
+
0.14821941492022558
|
| 2001 |
+
]
|
| 2002 |
+
},
|
| 2003 |
+
"tatoeba": {
|
| 2004 |
+
"acc": 0.2777777777777778,
|
| 2005 |
+
"n": 18,
|
| 2006 |
+
"ci95": [
|
| 2007 |
+
0.12499569146355474,
|
| 2008 |
+
0.5087307754619452
|
| 2009 |
+
]
|
| 2010 |
+
}
|
| 2011 |
+
}
|
| 2012 |
+
},
|
| 2013 |
+
"lid176": {
|
| 2014 |
+
"acc": 0.05454545454545454,
|
| 2015 |
+
"ci95": [
|
| 2016 |
+
0.025235483062071407,
|
| 2017 |
+
0.11391927231733298
|
| 2018 |
+
],
|
| 2019 |
+
"by": {
|
| 2020 |
+
"incubator": {
|
| 2021 |
+
"acc": 0.010869565217391304,
|
| 2022 |
+
"n": 92,
|
| 2023 |
+
"ci95": [
|
| 2024 |
+
0.0019212795763616508,
|
| 2025 |
+
0.05902928886156091
|
| 2026 |
+
]
|
| 2027 |
+
},
|
| 2028 |
+
"oth": {
|
| 2029 |
+
"acc": 0.05454545454545454,
|
| 2030 |
+
"n": 110,
|
| 2031 |
+
"ci95": [
|
| 2032 |
+
0.025235483062071407,
|
| 2033 |
+
0.11391927231733298
|
| 2034 |
+
]
|
| 2035 |
+
},
|
| 2036 |
+
"tatoeba": {
|
| 2037 |
+
"acc": 0.2777777777777778,
|
| 2038 |
+
"n": 18,
|
| 2039 |
+
"ci95": [
|
| 2040 |
+
0.12499569146355474,
|
| 2041 |
+
0.5087307754619452
|
| 2042 |
+
]
|
| 2043 |
+
}
|
| 2044 |
+
}
|
| 2045 |
+
}
|
| 2046 |
+
},
|
| 2047 |
+
"t_tshort": {
|
| 2048 |
+
"n": 543,
|
| 2049 |
+
"wikalid-1.0.1": {
|
| 2050 |
+
"acc": 0.856353591160221,
|
| 2051 |
+
"ci95": [
|
| 2052 |
+
0.824347035657435,
|
| 2053 |
+
0.8833533294940092
|
| 2054 |
+
],
|
| 2055 |
+
"by": {
|
| 2056 |
+
"cbk": {
|
| 2057 |
+
"acc": 0.9367088607594937,
|
| 2058 |
+
"n": 79,
|
| 2059 |
+
"ci95": [
|
| 2060 |
+
0.8602493258443956,
|
| 2061 |
+
0.9726655381852706
|
| 2062 |
+
]
|
| 2063 |
+
},
|
| 2064 |
+
"ceb": {
|
| 2065 |
+
"acc": 0.6666666666666666,
|
| 2066 |
+
"n": 39,
|
| 2067 |
+
"ci95": [
|
| 2068 |
+
0.5097710432024052,
|
| 2069 |
+
0.7936723389308484
|
| 2070 |
+
]
|
| 2071 |
+
},
|
| 2072 |
+
"eng": {
|
| 2073 |
+
"acc": 1.0,
|
| 2074 |
+
"n": 58,
|
| 2075 |
+
"ci95": [
|
| 2076 |
+
0.9378800031047062,
|
| 2077 |
+
0.9999999999999999
|
| 2078 |
+
]
|
| 2079 |
+
},
|
| 2080 |
+
"hil": {
|
| 2081 |
+
"acc": 1.0,
|
| 2082 |
+
"n": 1,
|
| 2083 |
+
"ci95": [
|
| 2084 |
+
0.20654329147389294,
|
| 2085 |
+
1.0
|
| 2086 |
+
]
|
| 2087 |
+
},
|
| 2088 |
+
"ilo": {
|
| 2089 |
+
"acc": 0.8222222222222222,
|
| 2090 |
+
"n": 45,
|
| 2091 |
+
"ci95": [
|
| 2092 |
+
0.6866989315634587,
|
| 2093 |
+
0.9070572107414617
|
| 2094 |
+
]
|
| 2095 |
+
},
|
| 2096 |
+
"ind": {
|
| 2097 |
+
"acc": 0.8410596026490066,
|
| 2098 |
+
"n": 151,
|
| 2099 |
+
"ci95": [
|
| 2100 |
+
0.7743902105949425,
|
| 2101 |
+
0.8908057057479524
|
| 2102 |
+
]
|
| 2103 |
+
},
|
| 2104 |
+
"pag": {
|
| 2105 |
+
"acc": 1.0,
|
| 2106 |
+
"n": 1,
|
| 2107 |
+
"ci95": [
|
| 2108 |
+
0.20654329147389294,
|
| 2109 |
+
1.0
|
| 2110 |
+
]
|
| 2111 |
+
},
|
| 2112 |
+
"pam": {
|
| 2113 |
+
"acc": 0.782608695652174,
|
| 2114 |
+
"n": 23,
|
| 2115 |
+
"ci95": [
|
| 2116 |
+
0.5809611981726508,
|
| 2117 |
+
0.9033616439828094
|
| 2118 |
+
]
|
| 2119 |
+
},
|
| 2120 |
+
"tatoeba": {
|
| 2121 |
+
"acc": 0.856353591160221,
|
| 2122 |
+
"n": 543,
|
| 2123 |
+
"ci95": [
|
| 2124 |
+
0.824347035657435,
|
| 2125 |
+
0.8833533294940092
|
| 2126 |
+
]
|
| 2127 |
+
},
|
| 2128 |
+
"tgl": {
|
| 2129 |
+
"acc": 0.9705882352941176,
|
| 2130 |
+
"n": 102,
|
| 2131 |
+
"ci95": [
|
| 2132 |
+
0.9170680330254163,
|
| 2133 |
+
0.9899477339319994
|
| 2134 |
+
]
|
| 2135 |
+
},
|
| 2136 |
+
"war": {
|
| 2137 |
+
"acc": 0.5454545454545454,
|
| 2138 |
+
"n": 44,
|
| 2139 |
+
"ci95": [
|
| 2140 |
+
0.4006593217065349,
|
| 2141 |
+
0.6829499221314639
|
| 2142 |
+
]
|
| 2143 |
+
}
|
| 2144 |
+
}
|
| 2145 |
+
},
|
| 2146 |
+
"glotlid": {
|
| 2147 |
+
"acc": 0.8655616942909761,
|
| 2148 |
+
"ci95": [
|
| 2149 |
+
0.8342870692237219,
|
| 2150 |
+
0.8917001270685865
|
| 2151 |
+
],
|
| 2152 |
+
"by": {
|
| 2153 |
+
"cbk": {
|
| 2154 |
+
"acc": 0.6455696202531646,
|
| 2155 |
+
"n": 79,
|
| 2156 |
+
"ci95": [
|
| 2157 |
+
0.5355906892335188,
|
| 2158 |
+
0.7420475987763698
|
| 2159 |
+
]
|
| 2160 |
+
},
|
| 2161 |
+
"ceb": {
|
| 2162 |
+
"acc": 0.7948717948717948,
|
| 2163 |
+
"n": 39,
|
| 2164 |
+
"ci95": [
|
| 2165 |
+
0.6446572703687138,
|
| 2166 |
+
0.8922040980208888
|
| 2167 |
+
]
|
| 2168 |
+
},
|
| 2169 |
+
"eng": {
|
| 2170 |
+
"acc": 0.9827586206896551,
|
| 2171 |
+
"n": 58,
|
| 2172 |
+
"ci95": [
|
| 2173 |
+
0.9085892732270167,
|
| 2174 |
+
0.9969500401154581
|
| 2175 |
+
]
|
| 2176 |
+
},
|
| 2177 |
+
"hil": {
|
| 2178 |
+
"acc": 0.0,
|
| 2179 |
+
"n": 1,
|
| 2180 |
+
"ci95": [
|
| 2181 |
+
0.0,
|
| 2182 |
+
0.7934567085261071
|
| 2183 |
+
]
|
| 2184 |
+
},
|
| 2185 |
+
"ilo": {
|
| 2186 |
+
"acc": 0.8888888888888888,
|
| 2187 |
+
"n": 45,
|
| 2188 |
+
"ci95": [
|
| 2189 |
+
0.7650062450910905,
|
| 2190 |
+
0.9515959956217445
|
| 2191 |
+
]
|
| 2192 |
+
},
|
| 2193 |
+
"ind": {
|
| 2194 |
+
"acc": 0.9139072847682119,
|
| 2195 |
+
"n": 151,
|
| 2196 |
+
"ci95": [
|
| 2197 |
+
0.8582784981751292,
|
| 2198 |
+
0.948998099328384
|
| 2199 |
+
]
|
| 2200 |
+
},
|
| 2201 |
+
"pag": {
|
| 2202 |
+
"acc": 1.0,
|
| 2203 |
+
"n": 1,
|
| 2204 |
+
"ci95": [
|
| 2205 |
+
0.20654329147389294,
|
| 2206 |
+
1.0
|
| 2207 |
+
]
|
| 2208 |
+
},
|
| 2209 |
+
"pam": {
|
| 2210 |
+
"acc": 0.8260869565217391,
|
| 2211 |
+
"n": 23,
|
| 2212 |
+
"ci95": [
|
| 2213 |
+
0.6286194521470433,
|
| 2214 |
+
0.9302145964938723
|
| 2215 |
+
]
|
| 2216 |
+
},
|
| 2217 |
+
"tatoeba": {
|
| 2218 |
+
"acc": 0.8655616942909761,
|
| 2219 |
+
"n": 543,
|
| 2220 |
+
"ci95": [
|
| 2221 |
+
0.8342870692237219,
|
| 2222 |
+
0.8917001270685865
|
| 2223 |
+
]
|
| 2224 |
+
},
|
| 2225 |
+
"tgl": {
|
| 2226 |
+
"acc": 0.9901960784313726,
|
| 2227 |
+
"n": 102,
|
| 2228 |
+
"ci95": [
|
| 2229 |
+
0.9465407803733753,
|
| 2230 |
+
0.9982673102072662
|
| 2231 |
+
]
|
| 2232 |
+
},
|
| 2233 |
+
"war": {
|
| 2234 |
+
"acc": 0.7272727272727273,
|
| 2235 |
+
"n": 44,
|
| 2236 |
+
"ci95": [
|
| 2237 |
+
0.581508453144907,
|
| 2238 |
+
0.8365377660450867
|
| 2239 |
+
]
|
| 2240 |
+
}
|
| 2241 |
+
}
|
| 2242 |
+
},
|
| 2243 |
+
"openlid3": {
|
| 2244 |
+
"acc": 0.6316758747697975,
|
| 2245 |
+
"ci95": [
|
| 2246 |
+
0.5903117707766475,
|
| 2247 |
+
0.6711899145377104
|
| 2248 |
+
],
|
| 2249 |
+
"by": {
|
| 2250 |
+
"cbk": {
|
| 2251 |
+
"acc": 0.0,
|
| 2252 |
+
"n": 79,
|
| 2253 |
+
"ci95": [
|
| 2254 |
+
-3.469446951953614e-18,
|
| 2255 |
+
0.04637283683559951
|
| 2256 |
+
]
|
| 2257 |
+
},
|
| 2258 |
+
"ceb": {
|
| 2259 |
+
"acc": 0.8461538461538461,
|
| 2260 |
+
"n": 39,
|
| 2261 |
+
"ci95": [
|
| 2262 |
+
0.7027023013426569,
|
| 2263 |
+
0.9275262615494854
|
| 2264 |
+
]
|
| 2265 |
+
},
|
| 2266 |
+
"eng": {
|
| 2267 |
+
"acc": 0.6551724137931034,
|
| 2268 |
+
"n": 58,
|
| 2269 |
+
"ci95": [
|
| 2270 |
+
0.5266753715786767,
|
| 2271 |
+
0.7643908362814045
|
| 2272 |
+
]
|
| 2273 |
+
},
|
| 2274 |
+
"hil": {
|
| 2275 |
+
"acc": 0.0,
|
| 2276 |
+
"n": 1,
|
| 2277 |
+
"ci95": [
|
| 2278 |
+
0.0,
|
| 2279 |
+
0.7934567085261071
|
| 2280 |
+
]
|
| 2281 |
+
},
|
| 2282 |
+
"ilo": {
|
| 2283 |
+
"acc": 0.8444444444444444,
|
| 2284 |
+
"n": 45,
|
| 2285 |
+
"ci95": [
|
| 2286 |
+
0.712158094887512,
|
| 2287 |
+
0.9225467468867133
|
| 2288 |
+
]
|
| 2289 |
+
},
|
| 2290 |
+
"ind": {
|
| 2291 |
+
"acc": 0.8609271523178808,
|
| 2292 |
+
"n": 151,
|
| 2293 |
+
"ci95": [
|
| 2294 |
+
0.7967393701331015,
|
| 2295 |
+
0.9072058228899619
|
| 2296 |
+
]
|
| 2297 |
+
},
|
| 2298 |
+
"pag": {
|
| 2299 |
+
"acc": 1.0,
|
| 2300 |
+
"n": 1,
|
| 2301 |
+
"ci95": [
|
| 2302 |
+
0.20654329147389294,
|
| 2303 |
+
1.0
|
| 2304 |
+
]
|
| 2305 |
+
},
|
| 2306 |
+
"pam": {
|
| 2307 |
+
"acc": 0.0,
|
| 2308 |
+
"n": 23,
|
| 2309 |
+
"ci95": [
|
| 2310 |
+
0.0,
|
| 2311 |
+
0.14312112541726274
|
| 2312 |
+
]
|
| 2313 |
+
},
|
| 2314 |
+
"tatoeba": {
|
| 2315 |
+
"acc": 0.6316758747697975,
|
| 2316 |
+
"n": 543,
|
| 2317 |
+
"ci95": [
|
| 2318 |
+
0.5903117707766475,
|
| 2319 |
+
0.6711899145377104
|
| 2320 |
+
]
|
| 2321 |
+
},
|
| 2322 |
+
"tgl": {
|
| 2323 |
+
"acc": 0.7941176470588235,
|
| 2324 |
+
"n": 102,
|
| 2325 |
+
"ci95": [
|
| 2326 |
+
0.7056727244152231,
|
| 2327 |
+
0.8612121299331615
|
| 2328 |
+
]
|
| 2329 |
+
},
|
| 2330 |
+
"war": {
|
| 2331 |
+
"acc": 0.5,
|
| 2332 |
+
"n": 44,
|
| 2333 |
+
"ci95": [
|
| 2334 |
+
0.35831521095931473,
|
| 2335 |
+
0.6416847890406853
|
| 2336 |
+
]
|
| 2337 |
+
}
|
| 2338 |
+
}
|
| 2339 |
+
},
|
| 2340 |
+
"lid176": {
|
| 2341 |
+
"acc": 0.6850828729281768,
|
| 2342 |
+
"ci95": [
|
| 2343 |
+
0.6448299972330791,
|
| 2344 |
+
0.7227353086982913
|
| 2345 |
+
],
|
| 2346 |
+
"by": {
|
| 2347 |
+
"cbk": {
|
| 2348 |
+
"acc": 0.20253164556962025,
|
| 2349 |
+
"n": 79,
|
| 2350 |
+
"ci95": [
|
| 2351 |
+
0.12869001243979505,
|
| 2352 |
+
0.303962181626954
|
| 2353 |
+
]
|
| 2354 |
+
},
|
| 2355 |
+
"ceb": {
|
| 2356 |
+
"acc": 0.5897435897435898,
|
| 2357 |
+
"n": 39,
|
| 2358 |
+
"ci95": [
|
| 2359 |
+
0.43418348473344903,
|
| 2360 |
+
0.7292091056459952
|
| 2361 |
+
]
|
| 2362 |
+
},
|
| 2363 |
+
"eng": {
|
| 2364 |
+
"acc": 1.0,
|
| 2365 |
+
"n": 58,
|
| 2366 |
+
"ci95": [
|
| 2367 |
+
0.9378800031047062,
|
| 2368 |
+
0.9999999999999999
|
| 2369 |
+
]
|
| 2370 |
+
},
|
| 2371 |
+
"hil": {
|
| 2372 |
+
"acc": 0.0,
|
| 2373 |
+
"n": 1,
|
| 2374 |
+
"ci95": [
|
| 2375 |
+
0.0,
|
| 2376 |
+
0.7934567085261071
|
| 2377 |
+
]
|
| 2378 |
+
},
|
| 2379 |
+
"ilo": {
|
| 2380 |
+
"acc": 0.5555555555555556,
|
| 2381 |
+
"n": 45,
|
| 2382 |
+
"ci95": [
|
| 2383 |
+
0.4117588341636533,
|
| 2384 |
+
0.6906129145096089
|
| 2385 |
+
]
|
| 2386 |
+
},
|
| 2387 |
+
"ind": {
|
| 2388 |
+
"acc": 0.8874172185430463,
|
| 2389 |
+
"n": 151,
|
| 2390 |
+
"ci95": [
|
| 2391 |
+
0.827099549136263,
|
| 2392 |
+
0.9285113461270252
|
| 2393 |
+
]
|
| 2394 |
+
},
|
| 2395 |
+
"pag": {
|
| 2396 |
+
"acc": 0.0,
|
| 2397 |
+
"n": 1,
|
| 2398 |
+
"ci95": [
|
| 2399 |
+
0.0,
|
| 2400 |
+
0.7934567085261071
|
| 2401 |
+
]
|
| 2402 |
+
},
|
| 2403 |
+
"pam": {
|
| 2404 |
+
"acc": 0.6521739130434783,
|
| 2405 |
+
"n": 23,
|
| 2406 |
+
"ci95": [
|
| 2407 |
+
0.4488997690689164,
|
| 2408 |
+
0.8118894536301776
|
| 2409 |
+
]
|
| 2410 |
+
},
|
| 2411 |
+
"tatoeba": {
|
| 2412 |
+
"acc": 0.6850828729281768,
|
| 2413 |
+
"n": 543,
|
| 2414 |
+
"ci95": [
|
| 2415 |
+
0.6448299972330791,
|
| 2416 |
+
0.7227353086982913
|
| 2417 |
+
]
|
| 2418 |
+
},
|
| 2419 |
+
"tgl": {
|
| 2420 |
+
"acc": 0.8235294117647058,
|
| 2421 |
+
"n": 102,
|
| 2422 |
+
"ci95": [
|
| 2423 |
+
0.7382155715042175,
|
| 2424 |
+
0.8853577682790058
|
| 2425 |
+
]
|
| 2426 |
+
},
|
| 2427 |
+
"war": {
|
| 2428 |
+
"acc": 0.38636363636363635,
|
| 2429 |
+
"n": 44,
|
| 2430 |
+
"ci95": [
|
| 2431 |
+
0.25720998886701985,
|
| 2432 |
+
0.5337669015379832
|
| 2433 |
+
]
|
| 2434 |
+
}
|
| 2435 |
+
}
|
| 2436 |
+
}
|
| 2437 |
+
}
|
| 2438 |
+
},
|
| 2439 |
+
"paired_vs_glotlid": {
|
| 2440 |
+
"crawl": {
|
| 2441 |
+
"delta": 0.07462686567164178,
|
| 2442 |
+
"ci95": [
|
| 2443 |
+
0.04470149253731343,
|
| 2444 |
+
0.11044776119402985
|
| 2445 |
+
]
|
| 2446 |
+
},
|
| 2447 |
+
"goldspan": {
|
| 2448 |
+
"delta": 0.039755351681957186,
|
| 2449 |
+
"ci95": [
|
| 2450 |
+
0.009174311926605505,
|
| 2451 |
+
0.07339449541284404
|
| 2452 |
+
]
|
| 2453 |
+
},
|
| 2454 |
+
"golddoc": {
|
| 2455 |
+
"delta": 0.02702702702702703,
|
| 2456 |
+
"ci95": [
|
| 2457 |
+
0.006756756756756757,
|
| 2458 |
+
0.05405405405405406
|
| 2459 |
+
]
|
| 2460 |
+
},
|
| 2461 |
+
"pld": {
|
| 2462 |
+
"delta": 0.08287961282516637,
|
| 2463 |
+
"ci95": [
|
| 2464 |
+
0.06654567453115548,
|
| 2465 |
+
0.09921355111917725
|
| 2466 |
+
]
|
| 2467 |
+
},
|
| 2468 |
+
"wiki": {
|
| 2469 |
+
"delta": 0.01833788529067499,
|
| 2470 |
+
"ci95": [
|
| 2471 |
+
0.009754194303550527,
|
| 2472 |
+
0.027311744049941473
|
| 2473 |
+
]
|
| 2474 |
+
},
|
| 2475 |
+
"hard": {
|
| 2476 |
+
"delta": 0.07164132309139364,
|
| 2477 |
+
"ci95": [
|
| 2478 |
+
0.06317604640225741,
|
| 2479 |
+
0.08026336416366202
|
| 2480 |
+
]
|
| 2481 |
+
},
|
| 2482 |
+
"basaha": {
|
| 2483 |
+
"delta": 0.1561712846347607,
|
| 2484 |
+
"ci95": [
|
| 2485 |
+
0.12342569269521411,
|
| 2486 |
+
0.1889168765743073
|
| 2487 |
+
]
|
| 2488 |
+
},
|
| 2489 |
+
"tato": {
|
| 2490 |
+
"delta": -0.026859504132231406,
|
| 2491 |
+
"ci95": [
|
| 2492 |
+
-0.04648760330578512,
|
| 2493 |
+
-0.009297520661157025
|
| 2494 |
+
]
|
| 2495 |
+
}
|
| 2496 |
+
},
|
| 2497 |
+
"gate_sealed": {
|
| 2498 |
+
"crawl": {
|
| 2499 |
+
"coverage": 0.5104477611940299,
|
| 2500 |
+
"precision_accepted_corrected": 1.0,
|
| 2501 |
+
"n": 335
|
| 2502 |
+
},
|
| 2503 |
+
"goldspan": {
|
| 2504 |
+
"coverage": 0.5137614678899083,
|
| 2505 |
+
"precision_accepted_corrected": 1.0,
|
| 2506 |
+
"n": 327
|
| 2507 |
+
},
|
| 2508 |
+
"golddoc": {
|
| 2509 |
+
"coverage": 0.7837837837837838,
|
| 2510 |
+
"precision_accepted_corrected": 0.9913793103448276,
|
| 2511 |
+
"n": 148
|
| 2512 |
+
},
|
| 2513 |
+
"pld": {
|
| 2514 |
+
"coverage": 0.6581972171808832,
|
| 2515 |
+
"precision_accepted_corrected": 0.9852941176470589,
|
| 2516 |
+
"n": 1653
|
| 2517 |
+
},
|
| 2518 |
+
"wiki": {
|
| 2519 |
+
"coverage": 0.5657432696059306,
|
| 2520 |
+
"precision_accepted_corrected": 0.9889655172413793,
|
| 2521 |
+
"n": 2563
|
| 2522 |
+
},
|
| 2523 |
+
"hard": {
|
| 2524 |
+
"coverage": 0.5101113027120239,
|
| 2525 |
+
"precision_accepted_corrected": 0.994161032575292,
|
| 2526 |
+
"n": 6379
|
| 2527 |
+
},
|
| 2528 |
+
"basaha": {
|
| 2529 |
+
"coverage": 0.35768261964735515,
|
| 2530 |
+
"precision_accepted_corrected": 0.971830985915493,
|
| 2531 |
+
"n": 794
|
| 2532 |
+
},
|
| 2533 |
+
"tato": {
|
| 2534 |
+
"coverage": 0.39462809917355374,
|
| 2535 |
+
"precision_accepted_corrected": 0.9947643979057592,
|
| 2536 |
+
"n": 968
|
| 2537 |
+
}
|
| 2538 |
+
},
|
| 2539 |
+
"consistency_vs_research_c7r": {
|
| 2540 |
+
"crawl": {
|
| 2541 |
+
"n": 661,
|
| 2542 |
+
"same_label": 661
|
| 2543 |
+
},
|
| 2544 |
+
"goldspan": {
|
| 2545 |
+
"n": 593,
|
| 2546 |
+
"same_label": 593
|
| 2547 |
+
},
|
| 2548 |
+
"golddoc": {
|
| 2549 |
+
"n": 294,
|
| 2550 |
+
"same_label": 294
|
| 2551 |
+
},
|
| 2552 |
+
"pld": {
|
| 2553 |
+
"n": 3222,
|
| 2554 |
+
"same_label": 3222
|
| 2555 |
+
},
|
| 2556 |
+
"wiki": {
|
| 2557 |
+
"n": 5037,
|
| 2558 |
+
"same_label": 5037
|
| 2559 |
+
},
|
| 2560 |
+
"hard": {
|
| 2561 |
+
"n": 12520,
|
| 2562 |
+
"same_label": 12520
|
| 2563 |
+
},
|
| 2564 |
+
"basaha": {
|
| 2565 |
+
"n": 1600,
|
| 2566 |
+
"same_label": 1600
|
| 2567 |
+
},
|
| 2568 |
+
"tato": {
|
| 2569 |
+
"n": 1847,
|
| 2570 |
+
"same_label": 1847
|
| 2571 |
+
},
|
| 2572 |
+
"t_sib": {
|
| 2573 |
+
"n": 9036,
|
| 2574 |
+
"same_label": 9036
|
| 2575 |
+
},
|
| 2576 |
+
"t_frag": {
|
| 2577 |
+
"n": 4320,
|
| 2578 |
+
"same_label": 4320
|
| 2579 |
+
},
|
| 2580 |
+
"t_caps": {
|
| 2581 |
+
"n": 5388,
|
| 2582 |
+
"same_label": 5388
|
| 2583 |
+
},
|
| 2584 |
+
"t_bikol": {
|
| 2585 |
+
"n": 1799,
|
| 2586 |
+
"same_label": 1799
|
| 2587 |
+
},
|
| 2588 |
+
"t_aklsgd": {
|
| 2589 |
+
"n": 110,
|
| 2590 |
+
"same_label": 110
|
| 2591 |
+
},
|
| 2592 |
+
"t_tshort": {
|
| 2593 |
+
"n": 543,
|
| 2594 |
+
"same_label": 543
|
| 2595 |
+
}
|
| 2596 |
+
}
|
| 2597 |
+
}
|
eval/RESULTS_v1.0.1.md
ADDED
|
@@ -0,0 +1,117 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# WikaLID 1.0.1 evaluation (generated by eval/evaluate.py; do not edit by hand)
|
| 2 |
+
|
| 3 |
+
## Sealed halves (never used for any decision): accuracy raw / corrected gold (macro-F1 corrected)
|
| 4 |
+
|
| 5 |
+
| system | crawl (n=335) | goldspan (n=327) | golddoc (n=148) | pld (n=1653) | wiki* (n=2563) | hard* (n=6379) | basaha (n=794) | tato* (n=968) | mean corrected |
|
| 6 |
+
|---|---|---|---|---|---|---|---|---|---|
|
| 7 |
+
| wikalid-1.0.1 | 0.949 / 0.949 (0.95) | 0.939 / 0.939 (0.93) | 0.986 / 0.986 (1.00) | 0.916 / 0.917 (0.93) | 0.899 / 0.925 (0.93) | 0.879 / 0.901 (0.88) | 0.739 / 0.739 (0.79) | 0.931 / 0.928 (0.94) | **0.911** |
|
| 8 |
+
| glotlid | 0.875 / 0.875 (0.93) | 0.899 / 0.899 (0.92) | 0.959 / 0.959 (1.00) | 0.830 / 0.834 (0.84) | 0.869 / 0.907 (0.91) | 0.799 / 0.829 (0.76) | 0.583 / 0.583 (0.64) | 0.949 / 0.955 (0.97) | **0.855** |
|
| 9 |
+
| openlid3 | 0.785 / 0.785 (0.62) | 0.881 / 0.881 (0.87) | 0.919 / 0.919 (0.99) | 0.417 / 0.417 (0.40) | 0.676 / 0.710 (0.63) | 0.619 / 0.641 (0.55) | 0.026 / 0.026 (0.01) | 0.754 / 0.758 (0.68) | **0.642** |
|
| 10 |
+
| lid176 | 0.734 / 0.734 (0.57) | 0.783 / 0.783 (0.74) | 0.818 / 0.818 (0.90) | 0.337 / 0.341 (0.29) | 0.620 / 0.664 (0.55) | 0.567 / 0.599 (0.45) | 0.031 / 0.031 (0.03) | 0.801 / 0.805 (0.80) | **0.597** |
|
| 11 |
+
|
| 12 |
+
\* public set (ships in eval/data, anyone can re-run). Others are private (news gold, crawl realism, PLD read speech, BasahaCorpus stories).
|
| 13 |
+
|
| 14 |
+
## WikaLID 1.0.1 minus GlotLID v3, sealed, corrected gold (paired bootstrap, 2,000 resamples)
|
| 15 |
+
|
| 16 |
+
| set | delta | 95% CI |
|
| 17 |
+
|---|---|---|
|
| 18 |
+
| crawl | +0.075 | [+0.045, +0.110] |
|
| 19 |
+
| goldspan | +0.040 | [+0.009, +0.073] |
|
| 20 |
+
| golddoc | +0.027 | [+0.007, +0.054] |
|
| 21 |
+
| pld | +0.083 | [+0.067, +0.099] |
|
| 22 |
+
| wiki | +0.018 | [+0.010, +0.027] |
|
| 23 |
+
| hard | +0.072 | [+0.063, +0.080] |
|
| 24 |
+
| basaha | +0.156 | [+0.123, +0.189] |
|
| 25 |
+
| tato | -0.027 | [-0.046, -0.009] |
|
| 26 |
+
|
| 27 |
+
## Gate v2 on sealed halves (accept = accept / accept_eng / accept_ph_other)
|
| 28 |
+
|
| 29 |
+
| set | coverage | precision of accepted (corrected gold) |
|
| 30 |
+
|---|---|---|
|
| 31 |
+
| crawl | 0.510 | 1.000 |
|
| 32 |
+
| goldspan | 0.514 | 1.000 |
|
| 33 |
+
| golddoc | 0.784 | 0.991 |
|
| 34 |
+
| pld | 0.658 | 0.985 |
|
| 35 |
+
| wiki | 0.566 | 0.989 |
|
| 36 |
+
| hard | 0.510 | 0.994 |
|
| 37 |
+
| basaha | 0.358 | 0.972 |
|
| 38 |
+
| tato | 0.395 | 0.995 |
|
| 39 |
+
|
| 40 |
+
## Human-labelled external trust sets (all rows; accuracy with 95% Wilson interval)
|
| 41 |
+
|
| 42 |
+
### t_sib (n=9036)
|
| 43 |
+
|
| 44 |
+
| system | overall | ceb | eng | ilo | ind | msa | pag | sib200 | spa | tgl | war |
|
| 45 |
+
|---|---|---|---|---|---|---|---|---|---|---|---|
|
| 46 |
+
| wikalid-1.0.1 | 0.989 [0.99-0.99] | 0.997 | 1.000 | 1.000 | 0.959 | 0.953 | 0.997 | 0.989 | 0.997 | 1.000 | 1.000 |
|
| 47 |
+
| glotlid | 0.986 [0.98-0.99] | 0.991 | 1.000 | 0.999 | 0.995 | 0.894 | 1.000 | 0.986 | 0.998 | 1.000 | 1.000 |
|
| 48 |
+
| openlid3 | 0.985 [0.98-0.99] | 1.000 | 0.998 | 1.000 | 0.918 | 0.963 | 0.996 | 0.985 | 0.986 | 1.000 | 1.000 |
|
| 49 |
+
| lid176 | 0.774 [0.77-0.78] | 0.922 | 1.000 | 0.985 | 0.969 | 0.449 | 0.000 | 0.774 | 1.000 | 0.998 | 0.640 |
|
| 50 |
+
|
| 51 |
+
### t_frag (n=4320)
|
| 52 |
+
|
| 53 |
+
| system | overall | ceb | eng | ilo | ind | len2 | len3 | len4 | len5 | msa | pag | spa | tgl | war |
|
| 54 |
+
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| 55 |
+
| wikalid-1.0.1 | 0.832 [0.82-0.84] | 0.815 | 0.977 | 0.942 | 0.821 | 0.704 | 0.815 | 0.878 | 0.931 | 0.640 | 0.873 | 0.871 | 0.844 | 0.706 |
|
| 56 |
+
| glotlid | 0.771 [0.76-0.78] | 0.760 | 0.908 | 0.892 | 0.867 | 0.624 | 0.739 | 0.835 | 0.887 | 0.342 | 0.823 | 0.850 | 0.867 | 0.633 |
|
| 57 |
+
| openlid3 | 0.773 [0.76-0.79] | 0.842 | 0.717 | 0.933 | 0.810 | 0.621 | 0.750 | 0.834 | 0.888 | 0.550 | 0.790 | 0.692 | 0.887 | 0.740 |
|
| 58 |
+
| lid176 | 0.590 [0.58-0.60] | 0.681 | 0.992 | 0.546 | 0.750 | 0.514 | 0.547 | 0.628 | 0.671 | 0.354 | 0.000 | 0.933 | 0.729 | 0.325 |
|
| 59 |
+
|
| 60 |
+
### t_caps (n=5388)
|
| 61 |
+
|
| 62 |
+
| system | overall | ceb | eng | ilo | lower | pag | spa | tgl | title | upper | war |
|
| 63 |
+
|---|---|---|---|---|---|---|---|---|---|---|---|
|
| 64 |
+
| wikalid-1.0.1 | 0.999 [1.00-1.00] | 0.995 | 1.000 | 1.000 | 0.998 | 0.999 | 1.000 | 0.999 | 0.999 | 0.999 | 1.000 |
|
| 65 |
+
| glotlid | 0.975 [0.97-0.98] | 0.959 | 0.931 | 0.999 | 0.998 | 0.994 | 0.949 | 0.996 | 0.990 | 0.937 | 1.000 |
|
| 66 |
+
| openlid3 | 0.575 [0.56-0.59] | 0.646 | 0.455 | 0.661 | 0.999 | 0.609 | 0.546 | 0.631 | 0.726 | 0.000 | 0.492 |
|
| 67 |
+
| lid176 | 0.522 [0.51-0.53] | 0.466 | 0.965 | 0.431 | 0.788 | 0.000 | 0.994 | 0.582 | 0.507 | 0.270 | 0.226 |
|
| 68 |
+
|
| 69 |
+
### t_bikol (n=1800)
|
| 70 |
+
|
| 71 |
+
| system | overall | bcl | ceb | story |
|
| 72 |
+
|---|---|---|---|---|
|
| 73 |
+
| wikalid-1.0.1 | 0.940 [0.93-0.95] | 0.961 | 0.919 | 0.940 |
|
| 74 |
+
| glotlid | 0.893 [0.88-0.91] | 0.894 | 0.892 | 0.893 |
|
| 75 |
+
| openlid3 | 0.467 [0.44-0.49] | 0.000 | 0.933 | 0.467 |
|
| 76 |
+
| lid176 | 0.291 [0.27-0.31] | 0.003 | 0.579 | 0.291 |
|
| 77 |
+
|
| 78 |
+
### t_aklsgd (n=110)
|
| 79 |
+
|
| 80 |
+
| system | overall | incubator | oth | tatoeba |
|
| 81 |
+
|---|---|---|---|---|
|
| 82 |
+
| wikalid-1.0.1 | 0.345 [0.26-0.44] | 0.370 | 0.345 | 0.222 |
|
| 83 |
+
| glotlid | 0.300 [0.22-0.39] | 0.272 | 0.300 | 0.444 |
|
| 84 |
+
| openlid3 | 0.082 [0.04-0.15] | 0.043 | 0.082 | 0.278 |
|
| 85 |
+
| lid176 | 0.055 [0.03-0.11] | 0.011 | 0.055 | 0.278 |
|
| 86 |
+
|
| 87 |
+
### t_tshort (n=543)
|
| 88 |
+
|
| 89 |
+
| system | overall | cbk | ceb | eng | hil | ilo | ind | pag | pam | tatoeba | tgl | war |
|
| 90 |
+
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| 91 |
+
| wikalid-1.0.1 | 0.856 [0.82-0.88] | 0.937 | 0.667 | 1.000 | 1.000 | 0.822 | 0.841 | 1.000 | 0.783 | 0.856 | 0.971 | 0.545 |
|
| 92 |
+
| glotlid | 0.866 [0.83-0.89] | 0.646 | 0.795 | 0.983 | 0.000 | 0.889 | 0.914 | 1.000 | 0.826 | 0.866 | 0.990 | 0.727 |
|
| 93 |
+
| openlid3 | 0.632 [0.59-0.67] | 0.000 | 0.846 | 0.655 | 0.000 | 0.844 | 0.861 | 1.000 | 0.000 | 0.632 | 0.794 | 0.500 |
|
| 94 |
+
| lid176 | 0.685 [0.64-0.72] | 0.203 | 0.590 | 1.000 | 0.000 | 0.556 | 0.887 | 0.000 | 0.652 | 0.685 | 0.824 | 0.386 |
|
| 95 |
+
|
| 96 |
+
## Consistency with the research model that labels Himay (c7 + rules, cached predictions)
|
| 97 |
+
|
| 98 |
+
| set | n | same label |
|
| 99 |
+
|---|---|---|
|
| 100 |
+
| crawl | 661 | 661 |
|
| 101 |
+
| goldspan | 593 | 593 |
|
| 102 |
+
| golddoc | 294 | 294 |
|
| 103 |
+
| pld | 3222 | 3222 |
|
| 104 |
+
| wiki | 5037 | 5037 |
|
| 105 |
+
| hard | 12520 | 12520 |
|
| 106 |
+
| basaha | 1600 | 1600 |
|
| 107 |
+
| tato | 1847 | 1847 |
|
| 108 |
+
| t_sib | 9036 | 9036 |
|
| 109 |
+
| t_frag | 4320 | 4320 |
|
| 110 |
+
| t_caps | 5388 | 5388 |
|
| 111 |
+
| t_bikol | 1799 | 1799 |
|
| 112 |
+
| t_aklsgd | 110 | 110 |
|
| 113 |
+
| t_tshort | 543 | 543 |
|
| 114 |
+
|
| 115 |
+
## Throughput during this run (items/s, all workers, shared box)
|
| 116 |
+
|
| 117 |
+
crawl: 260 | goldspan: 280 | golddoc: 83 | pld: 361 | wiki: 412 | hard: 418 | basaha: 341 | tato: 416 | t_sib: 434 | t_frag: 414 | t_caps: 394 | t_bikol: 390 | t_aklsgd: 90 | t_tshort: 312
|
eval/evaluate.py
CHANGED
|
@@ -1,8 +1,8 @@
|
|
| 1 |
-
"""Reproducible evaluation of the installed `
|
| 2 |
|
| 3 |
usage: python eval/evaluate.py [--workers 4] [--skip-predict]
|
| 4 |
needs: eval/out/preds/{glotlid,openlid3,lid176}__<set>.json from eval/run_baselines.py (live fastText runs)
|
| 5 |
-
writes: eval/out/preds/
|
| 6 |
|
| 7 |
Protocol (unchanged from the research loop): every set is split 50/50 by md5 of a group key into dev (all tuning
|
| 8 |
decisions were taken there) and sealed (never used for a decision). Headline = sealed halves. "corrected" gold fixes known
|
|
@@ -15,15 +15,15 @@ from multiprocessing import get_context
|
|
| 15 |
import numpy as np
|
| 16 |
HERE = os.path.dirname(os.path.abspath(__file__)); sys.path.insert(0, HERE)
|
| 17 |
from sets_v1 import CORE, TRUST, PUBLIC, available, rows, ok, part_of
|
| 18 |
-
PRED = os.path.join(HERE, "out", "preds"); TAG = "
|
| 19 |
CBKM = set(json.load(open(os.path.join(HERE, "data", "cbk_markers.json"))))
|
| 20 |
_P = _G = None
|
| 21 |
|
| 22 |
|
| 23 |
def _init():
|
| 24 |
global _P, _G
|
| 25 |
-
import ourlid
|
| 26 |
-
from
|
| 27 |
_P = ourlid.load(rules=True); _G = Gate(ourlid.model_dir())
|
| 28 |
|
| 29 |
|
|
@@ -36,7 +36,7 @@ def _work(batch):
|
|
| 36 |
|
| 37 |
|
| 38 |
def predict(workers, resume=False):
|
| 39 |
-
import ourlid
|
| 40 |
os.makedirs(PRED, exist_ok=True); speed = {}
|
| 41 |
for n in available():
|
| 42 |
if resume and os.path.exists(f"{PRED}/{TAG}__{n}.json"): continue
|
|
@@ -118,12 +118,12 @@ def main():
|
|
| 118 |
p = f"/workspace/ourlid/eval/preds/c7r__{n}.json"
|
| 119 |
A, B = load(TAG, n), (json.load(open(p))["pred"] if os.path.exists(p) else None)
|
| 120 |
if A and B: res["consistency_vs_research_c7r"][n] = {"n": len(A), "same_label": sum(A[i][0] == B[i][0] for i in A if i in B)}
|
| 121 |
-
json.dump(res, open(os.path.join(HERE, "RESULTS_v1.0.
|
| 122 |
write_md(res, tools)
|
| 123 |
|
| 124 |
|
| 125 |
def write_md(res, tools):
|
| 126 |
-
L = ["#
|
| 127 |
S = res["core"]["sealed"]; sets = list(S)
|
| 128 |
L += ["## Sealed halves (never used for any decision): accuracy raw / corrected gold (macro-F1 corrected)", "",
|
| 129 |
"| system | " + " | ".join(f"{n}{'*' if S[n]['public'] else ''} (n={S[n]['n']})" for n in sets) + " | mean corrected |", "|---|" + "---|" * (len(sets) + 1)]
|
|
@@ -135,7 +135,7 @@ def write_md(res, tools):
|
|
| 135 |
cells.append(f"{c['acc_raw']:.3f} / {c['acc_corrected']:.3f} ({c['macro_f1_corrected']:.2f})"); ms.append(c["acc_corrected"])
|
| 136 |
L.append(f"| {t} | " + " | ".join(cells) + f" | **{np.mean(ms):.3f}** |")
|
| 137 |
L += ["", "\\* public set (ships in eval/data, anyone can re-run). Others are private (news gold, crawl realism, PLD read speech, BasahaCorpus stories).", "",
|
| 138 |
-
"##
|
| 139 |
for n, d in res["paired_vs_glotlid"].items(): L.append(f"| {n} | {d['delta']:+.3f} | [{d['ci95'][0]:+.3f}, {d['ci95'][1]:+.3f}] |")
|
| 140 |
L += ["", "## Gate v2 on sealed halves (accept = accept / accept_eng / accept_ph_other)", "", "| set | coverage | precision of accepted (corrected gold) |", "|---|---|---|"]
|
| 141 |
for n, d in res["gate_sealed"].items(): L.append(f"| {n} | {d['coverage']:.3f} | {d['precision_accepted_corrected']:.3f} |")
|
|
@@ -150,7 +150,7 @@ def write_md(res, tools):
|
|
| 150 |
for n, d in res["consistency_vs_research_c7r"].items(): L.append(f"| {n} | {d['n']} | {d['same_label']} |")
|
| 151 |
if res["speed_items_per_s"]:
|
| 152 |
L += ["", "## Throughput during this run (items/s, all workers, shared box)", "", " | ".join(f"{n}: {v:.0f}" for n, v in res["speed_items_per_s"].items())]
|
| 153 |
-
open(os.path.join(HERE, "RESULTS_v1.0.
|
| 154 |
|
| 155 |
|
| 156 |
if __name__ == "__main__":
|
|
|
|
| 1 |
+
"""Reproducible evaluation of the installed `wikalid` package (1.0.1) against GlotLID v3, OpenLID-v3 and fastText lid.176.
|
| 2 |
|
| 3 |
usage: python eval/evaluate.py [--workers 4] [--skip-predict]
|
| 4 |
needs: eval/out/preds/{glotlid,openlid3,lid176}__<set>.json from eval/run_baselines.py (live fastText runs)
|
| 5 |
+
writes: eval/out/preds/wikalid-1.0.1__<set>.json, eval/RESULTS_v1.0.1.json, eval/RESULTS_v1.0.1.md
|
| 6 |
|
| 7 |
Protocol (unchanged from the research loop): every set is split 50/50 by md5 of a group key into dev (all tuning
|
| 8 |
decisions were taken there) and sealed (never used for a decision). Headline = sealed halves. "corrected" gold fixes known
|
|
|
|
| 15 |
import numpy as np
|
| 16 |
HERE = os.path.dirname(os.path.abspath(__file__)); sys.path.insert(0, HERE)
|
| 17 |
from sets_v1 import CORE, TRUST, PUBLIC, available, rows, ok, part_of
|
| 18 |
+
PRED = os.path.join(HERE, "out", "preds"); TAG = "wikalid-1.0.1"; BASE = ["glotlid", "openlid3", "lid176"]
|
| 19 |
CBKM = set(json.load(open(os.path.join(HERE, "data", "cbk_markers.json"))))
|
| 20 |
_P = _G = None
|
| 21 |
|
| 22 |
|
| 23 |
def _init():
|
| 24 |
global _P, _G
|
| 25 |
+
import wikalid as ourlid
|
| 26 |
+
from wikalid.gate import Gate
|
| 27 |
_P = ourlid.load(rules=True); _G = Gate(ourlid.model_dir())
|
| 28 |
|
| 29 |
|
|
|
|
| 36 |
|
| 37 |
|
| 38 |
def predict(workers, resume=False):
|
| 39 |
+
import wikalid as ourlid
|
| 40 |
os.makedirs(PRED, exist_ok=True); speed = {}
|
| 41 |
for n in available():
|
| 42 |
if resume and os.path.exists(f"{PRED}/{TAG}__{n}.json"): continue
|
|
|
|
| 118 |
p = f"/workspace/ourlid/eval/preds/c7r__{n}.json"
|
| 119 |
A, B = load(TAG, n), (json.load(open(p))["pred"] if os.path.exists(p) else None)
|
| 120 |
if A and B: res["consistency_vs_research_c7r"][n] = {"n": len(A), "same_label": sum(A[i][0] == B[i][0] for i in A if i in B)}
|
| 121 |
+
json.dump(res, open(os.path.join(HERE, "RESULTS_v1.0.1.json"), "w"), indent=1, default=float)
|
| 122 |
write_md(res, tools)
|
| 123 |
|
| 124 |
|
| 125 |
def write_md(res, tools):
|
| 126 |
+
L = ["# WikaLID 1.0.1 evaluation (generated by eval/evaluate.py; do not edit by hand)", ""]
|
| 127 |
S = res["core"]["sealed"]; sets = list(S)
|
| 128 |
L += ["## Sealed halves (never used for any decision): accuracy raw / corrected gold (macro-F1 corrected)", "",
|
| 129 |
"| system | " + " | ".join(f"{n}{'*' if S[n]['public'] else ''} (n={S[n]['n']})" for n in sets) + " | mean corrected |", "|---|" + "---|" * (len(sets) + 1)]
|
|
|
|
| 135 |
cells.append(f"{c['acc_raw']:.3f} / {c['acc_corrected']:.3f} ({c['macro_f1_corrected']:.2f})"); ms.append(c["acc_corrected"])
|
| 136 |
L.append(f"| {t} | " + " | ".join(cells) + f" | **{np.mean(ms):.3f}** |")
|
| 137 |
L += ["", "\\* public set (ships in eval/data, anyone can re-run). Others are private (news gold, crawl realism, PLD read speech, BasahaCorpus stories).", "",
|
| 138 |
+
"## WikaLID 1.0.1 minus GlotLID v3, sealed, corrected gold (paired bootstrap, 2,000 resamples)", "", "| set | delta | 95% CI |", "|---|---|---|"]
|
| 139 |
for n, d in res["paired_vs_glotlid"].items(): L.append(f"| {n} | {d['delta']:+.3f} | [{d['ci95'][0]:+.3f}, {d['ci95'][1]:+.3f}] |")
|
| 140 |
L += ["", "## Gate v2 on sealed halves (accept = accept / accept_eng / accept_ph_other)", "", "| set | coverage | precision of accepted (corrected gold) |", "|---|---|---|"]
|
| 141 |
for n, d in res["gate_sealed"].items(): L.append(f"| {n} | {d['coverage']:.3f} | {d['precision_accepted_corrected']:.3f} |")
|
|
|
|
| 150 |
for n, d in res["consistency_vs_research_c7r"].items(): L.append(f"| {n} | {d['n']} | {d['same_label']} |")
|
| 151 |
if res["speed_items_per_s"]:
|
| 152 |
L += ["", "## Throughput during this run (items/s, all workers, shared box)", "", " | ".join(f"{n}: {v:.0f}" for n, v in res["speed_items_per_s"].items())]
|
| 153 |
+
open(os.path.join(HERE, "RESULTS_v1.0.1.md"), "w").write("\n".join(L) + "\n")
|
| 154 |
|
| 155 |
|
| 156 |
if __name__ == "__main__":
|
eval/parity_himay.py
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
"""Parity check:
|
| 2 |
usage: parity_himay.py <documents.parquet> <shipped sidecar.parquet> <workdir> [n=3000] [workers=3]
|
| 3 |
Samples n docs (seed 20261115; every non-tgl label is over-sampled), runs `ourlid sidecar`, compares every column."""
|
| 4 |
import json, os, subprocess, sys
|
|
@@ -16,7 +16,7 @@ for i in range(f.num_row_groups):
|
|
| 16 |
sub = pa.concat_tables(parts); src = os.path.join(wd, "parity_docs.parquet"); pq.write_table(sub, src, row_group_size=250)
|
| 17 |
out = os.path.join(wd, "parity_sidecar.parquet")
|
| 18 |
if os.path.exists(out): os.remove(out)
|
| 19 |
-
exe = os.path.join(os.path.dirname(sys.executable), "
|
| 20 |
subprocess.run([exe, "sidecar", src, out, "--workers", workers], check=True, stdout=subprocess.DEVNULL)
|
| 21 |
N = pq.read_table(out).to_pandas().set_index("document_id"); O = S.filter(pc.is_in(S.column("document_id"), pa.array(sorted(pick)))).to_pandas().set_index("document_id").loc[N.index]
|
| 22 |
rep = {"n_docs": len(N), "labels_sampled": O.ourlid_label_final.value_counts().to_dict(), "columns": {}}
|
|
|
|
| 1 |
+
"""Parity check: wikalid (installed package; formerly ourlid) vs the sidecar that labels Himay (built by the research-repo c7 scripts).
|
| 2 |
usage: parity_himay.py <documents.parquet> <shipped sidecar.parquet> <workdir> [n=3000] [workers=3]
|
| 3 |
Samples n docs (seed 20261115; every non-tgl label is over-sampled), runs `ourlid sidecar`, compares every column."""
|
| 4 |
import json, os, subprocess, sys
|
|
|
|
| 16 |
sub = pa.concat_tables(parts); src = os.path.join(wd, "parity_docs.parquet"); pq.write_table(sub, src, row_group_size=250)
|
| 17 |
out = os.path.join(wd, "parity_sidecar.parquet")
|
| 18 |
if os.path.exists(out): os.remove(out)
|
| 19 |
+
exe = os.path.join(os.path.dirname(sys.executable), "wikalid")
|
| 20 |
subprocess.run([exe, "sidecar", src, out, "--workers", workers], check=True, stdout=subprocess.DEVNULL)
|
| 21 |
N = pq.read_table(out).to_pandas().set_index("document_id"); O = S.filter(pc.is_in(S.column("document_id"), pa.array(sorted(pick)))).to_pandas().set_index("document_id").loc[N.index]
|
| 22 |
rep = {"n_docs": len(N), "labels_sampled": O.ourlid_label_final.value_counts().to_dict(), "columns": {}}
|
pyproject.toml
CHANGED
|
@@ -3,9 +3,9 @@ requires = ["setuptools>=77", "wheel"]
|
|
| 3 |
build-backend = "setuptools.build_meta"
|
| 4 |
|
| 5 |
[project]
|
| 6 |
-
name = "
|
| 7 |
-
version = "1.0.
|
| 8 |
-
description = "
|
| 9 |
readme = "README.md"
|
| 10 |
requires-python = ">=3.10"
|
| 11 |
license = "Apache-2.0"
|
|
@@ -17,10 +17,11 @@ parquet = ["pyarrow>=15"]
|
|
| 17 |
test = ["pytest", "pyarrow>=15"]
|
| 18 |
|
| 19 |
[project.scripts]
|
| 20 |
-
|
|
|
|
| 21 |
|
| 22 |
[tool.setuptools.packages.find]
|
| 23 |
where = ["src"]
|
| 24 |
|
| 25 |
[tool.setuptools.package-data]
|
| 26 |
-
|
|
|
|
| 3 |
build-backend = "setuptools.build_meta"
|
| 4 |
|
| 5 |
[project]
|
| 6 |
+
name = "wikalid"
|
| 7 |
+
version = "1.0.1"
|
| 8 |
+
description = "WikaLID: mixture-aware, calibrated language identification for Philippine languages (Tagalog/Taglish, Cebuano, Hiligaynon, Waray, Ilocano, Bikol, Kapampangan, Pangasinan, Chavacano, ...)"
|
| 9 |
readme = "README.md"
|
| 10 |
requires-python = ">=3.10"
|
| 11 |
license = "Apache-2.0"
|
|
|
|
| 17 |
test = ["pytest", "pyarrow>=15"]
|
| 18 |
|
| 19 |
[project.scripts]
|
| 20 |
+
wikalid = "wikalid.cli:main"
|
| 21 |
+
ourlid = "wikalid.cli:main" # former name, kept as an alias
|
| 22 |
|
| 23 |
[tool.setuptools.packages.find]
|
| 24 |
where = ["src"]
|
| 25 |
|
| 26 |
[tool.setuptools.package-data]
|
| 27 |
+
wikalid = ["cbk_vs_spa.json", "model/*"]
|
src/ourlid/__init__.py
CHANGED
|
@@ -1,50 +1,11 @@
|
|
| 1 |
-
"""
|
|
|
|
|
|
|
| 2 |
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
ourlid.classify("Indi ko mahibal-an kon ngaa wala sila nagkadto.") # analysis + gate decision
|
| 7 |
|
| 8 |
-
|
| 9 |
-
"""
|
| 10 |
-
|
| 11 |
-
import os
|
| 12 |
-
|
| 13 |
-
__version__ = "1.0.0"
|
| 14 |
-
MODEL_ID = "ourlid-1.0.0"
|
| 15 |
-
GATE_ID = "gate_v2"
|
| 16 |
-
TOOL_ID = f"{MODEL_ID}+{GATE_ID}" # value written to Himay's `lang_tool` column
|
| 17 |
-
LABELS = ["tgl", "ceb", "hil", "war", "ilo", "bcl", "pam", "pag", "cbk", "krj", "tsg", "eng", "spa", "ind", "msa", "oth", "nol"]
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
def model_dir() -> str:
|
| 21 |
-
"""Directory of the bundled ourlid-1.0.0 weights (override with env OURLID_MODEL)."""
|
| 22 |
-
return os.environ.get("OURLID_MODEL") or os.path.join(os.path.dirname(os.path.abspath(__file__)), "model")
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
def load(model: str | None = None, alpha: str = "0.1", rules: bool = True):
|
| 26 |
-
"""-> predictor with .analyze(text, max_tokens=None) -> dict.
|
| 27 |
-
|
| 28 |
-
alpha: conformal level for `label_set`/`abstain` ("0.01" | "0.05" | "0.1"; 0.1 is the shipped default).
|
| 29 |
-
rules: apply the unsupported-Philippine-language marker bucket (Aklanon/Surigaonon -> label 'oth', key 'rule')."""
|
| 30 |
-
import json
|
| 31 |
-
from .r3 import PredictorR3
|
| 32 |
-
md = model or model_dir()
|
| 33 |
-
opts = json.load(open(os.path.join(md, "r3opts.json")))
|
| 34 |
-
opts.pop("alpha", None)
|
| 35 |
-
return PredictorR3(md, alpha, rules=3.0 if rules else None, **opts)
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
_DEFAULT = {}
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
def classify(text: str, model: str | None = None) -> dict:
|
| 42 |
-
"""Analysis + gate v2 decision for one text: adds gate ('accept'|'accept_eng'|'accept_ph_other'|'review'|'junk'),
|
| 43 |
-
gate_reason, gate_p_correct, label_final, oth_sub. Same decision as the batch sidecar."""
|
| 44 |
-
from .gate import Gate
|
| 45 |
-
key = model or model_dir()
|
| 46 |
-
if key not in _DEFAULT:
|
| 47 |
-
_DEFAULT[key] = (load(key, rules=False), Gate(key))
|
| 48 |
-
P, G = _DEFAULT[key]
|
| 49 |
-
a = P.analyze(text)
|
| 50 |
-
return dict(a, **G.decide_one(text, a))
|
|
|
|
| 1 |
+
"""Compatibility alias: `ourlid` is the former name of WikaLID. `import ourlid` and `ourlid.<submodule>` resolve to wikalid."""
|
| 2 |
+
import importlib as _il
|
| 3 |
+
import sys as _sys
|
| 4 |
|
| 5 |
+
import wikalid as _w
|
| 6 |
+
from wikalid import * # noqa: F401,F403
|
| 7 |
+
from wikalid import __version__, MODEL_ID, GATE_ID, TOOL_ID, LABELS, model_dir, load, classify # noqa: F401
|
|
|
|
| 8 |
|
| 9 |
+
for _m in ("cli", "doc", "features", "gate", "head", "infer", "lexicon", "predictor", "r3", "rules", "sidecar", "synth", "textutil"):
|
| 10 |
+
_sys.modules[f"{__name__}.{_m}"] = _il.import_module(f"wikalid.{_m}")
|
| 11 |
+
globals()[_m] = _sys.modules[f"{__name__}.{_m}"]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
src/ourlid/__main__.py
CHANGED
|
@@ -1,2 +1,2 @@
|
|
| 1 |
-
from .cli import main
|
| 2 |
main()
|
|
|
|
| 1 |
+
from wikalid.cli import main
|
| 2 |
main()
|
src/wikalid/__init__.py
ADDED
|
@@ -0,0 +1,50 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""WikaLID: mixture-aware, calibrated language identification for Philippine languages ("wika" = language).
|
| 2 |
+
|
| 3 |
+
import wikalid
|
| 4 |
+
lid = wikalid.load() # bundled model (two-tagger ensemble + head + marker rules)
|
| 5 |
+
lid.analyze("Nag-submit na ako ng report pero hindi pa daw na-approve.")
|
| 6 |
+
wikalid.classify("Indi ko mahibal-an kon ngaa wala sila nagkadto.") # analysis + gate decision
|
| 7 |
+
|
| 8 |
+
Numpy-only inference, deterministic, CPU. Formerly released as OurLID 1.0.0 (same weights); `import ourlid` still works.
|
| 9 |
+
"""
|
| 10 |
+
from __future__ import annotations
|
| 11 |
+
import os
|
| 12 |
+
|
| 13 |
+
__version__ = "1.0.1"
|
| 14 |
+
MODEL_ID = "wikalid-1.0.1"
|
| 15 |
+
GATE_ID = "gate_v2"
|
| 16 |
+
TOOL_ID = f"{MODEL_ID}+{GATE_ID}" # value written to Himay's `lang_tool` column
|
| 17 |
+
LABELS = ["tgl", "ceb", "hil", "war", "ilo", "bcl", "pam", "pag", "cbk", "krj", "tsg", "eng", "spa", "ind", "msa", "oth", "nol"]
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def model_dir() -> str:
|
| 21 |
+
"""Directory of the bundled weights (override with env WIKALID_MODEL; OURLID_MODEL is still honoured)."""
|
| 22 |
+
return os.environ.get("WIKALID_MODEL") or os.environ.get("OURLID_MODEL") or os.path.join(os.path.dirname(os.path.abspath(__file__)), "model")
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def load(model: str | None = None, alpha: str = "0.1", rules: bool = True):
|
| 26 |
+
"""-> predictor with .analyze(text, max_tokens=None) -> dict.
|
| 27 |
+
|
| 28 |
+
alpha: conformal level for `label_set`/`abstain` ("0.01" | "0.05" | "0.1"; 0.1 is the shipped default).
|
| 29 |
+
rules: apply the unsupported-Philippine-language marker bucket (Aklanon/Surigaonon -> label 'oth', key 'rule')."""
|
| 30 |
+
import json
|
| 31 |
+
from .r3 import PredictorR3
|
| 32 |
+
md = model or model_dir()
|
| 33 |
+
opts = json.load(open(os.path.join(md, "r3opts.json")))
|
| 34 |
+
opts.pop("alpha", None)
|
| 35 |
+
return PredictorR3(md, alpha, rules=3.0 if rules else None, **opts)
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
_DEFAULT = {}
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
def classify(text: str, model: str | None = None) -> dict:
|
| 42 |
+
"""Analysis + gate v2 decision for one text: adds gate ('accept'|'accept_eng'|'accept_ph_other'|'review'|'junk'),
|
| 43 |
+
gate_reason, gate_p_correct, label_final, oth_sub. Same decision as the batch sidecar."""
|
| 44 |
+
from .gate import Gate
|
| 45 |
+
key = model or model_dir()
|
| 46 |
+
if key not in _DEFAULT:
|
| 47 |
+
_DEFAULT[key] = (load(key, rules=False), Gate(key))
|
| 48 |
+
P, G = _DEFAULT[key]
|
| 49 |
+
a = P.analyze(text)
|
| 50 |
+
return dict(a, **G.decide_one(text, a))
|
src/wikalid/__main__.py
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from .cli import main
|
| 2 |
+
main()
|
src/{ourlid → wikalid}/cbk_vs_spa.json
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/cli.py
RENAMED
|
@@ -1,10 +1,10 @@
|
|
| 1 |
-
"""`
|
| 2 |
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
"""
|
| 9 |
from __future__ import annotations
|
| 10 |
import argparse, hashlib, json, os, sys
|
|
@@ -20,9 +20,9 @@ def _sha(p):
|
|
| 20 |
|
| 21 |
def main(argv=None):
|
| 22 |
from . import __version__, MODEL_ID, TOOL_ID, LABELS, model_dir
|
| 23 |
-
ap = argparse.ArgumentParser(prog="ourlid", description=f"
|
| 24 |
-
ap.add_argument("--version", action="version", version=f"
|
| 25 |
-
ap.add_argument("--model", default=None, help="model directory (default: bundled
|
| 26 |
sub = ap.add_subparsers(dest="cmd", required=True)
|
| 27 |
p = sub.add_parser("predict", help="identify the language of texts (args, --jsonl file, or stdin lines)")
|
| 28 |
p.add_argument("text", nargs="*"); p.add_argument("--jsonl"); p.add_argument("--alpha", default="0.1", choices=["0.01", "0.05", "0.1"])
|
|
|
|
| 1 |
+
"""`wikalid` command line (`ourlid` is kept as an alias).
|
| 2 |
|
| 3 |
+
wikalid predict "text" ... [--jsonl FILE|-] [--alpha 0.1] [--no-rules] [--gate] -> one JSON line per text
|
| 4 |
+
wikalid score DOCS.parquet RAW.parquet [--workers N] [--max-tokens K] -> model columns per document
|
| 5 |
+
wikalid gate RAW.parquet DOCS.parquet OUT.parquet -> + gate v2 columns
|
| 6 |
+
wikalid sidecar DOCS.parquet OUT.parquet [--workers N] -> score + gate
|
| 7 |
+
wikalid info | wikalid verify -> version, labels, weight hashes
|
| 8 |
"""
|
| 9 |
from __future__ import annotations
|
| 10 |
import argparse, hashlib, json, os, sys
|
|
|
|
| 20 |
|
| 21 |
def main(argv=None):
|
| 22 |
from . import __version__, MODEL_ID, TOOL_ID, LABELS, model_dir
|
| 23 |
+
ap = argparse.ArgumentParser(prog=os.path.basename(sys.argv[0]) if os.path.basename(sys.argv[0]) in ("wikalid", "ourlid") else "wikalid", description=f"WikaLID {__version__}: Philippine-language identification")
|
| 24 |
+
ap.add_argument("--version", action="version", version=f"wikalid {__version__} ({MODEL_ID})")
|
| 25 |
+
ap.add_argument("--model", default=None, help="model directory (default: bundled weights, or $WIKALID_MODEL)")
|
| 26 |
sub = ap.add_subparsers(dest="cmd", required=True)
|
| 27 |
p = sub.add_parser("predict", help="identify the language of texts (args, --jsonl file, or stdin lines)")
|
| 28 |
p.add_argument("text", nargs="*"); p.add_argument("--jsonl"); p.add_argument("--alpha", default="0.1", choices=["0.01", "0.05", "0.1"])
|
src/{ourlid → wikalid}/doc.py
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/features.py
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/gate.py
RENAMED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
"""Gate v2: named rules + a small gradient-boosted tree head over
|
| 2 |
|
| 3 |
The tree head was fit with scikit-learn (HistGradientBoostingClassifier, dev halves only) and exported to plain arrays
|
| 4 |
(`model/gate_head.npz`), so inference needs numpy only and never unpickles anything. `tests/test_gate_parity.py` checks the
|
|
|
|
| 1 |
+
"""Gate v2: named rules + a small gradient-boosted tree head over WikaLID outputs -> accept / review / junk.
|
| 2 |
|
| 3 |
The tree head was fit with scikit-learn (HistGradientBoostingClassifier, dev halves only) and exported to plain arrays
|
| 4 |
(`model/gate_head.npz`), so inference needs numpy only and never unpickles anything. `tests/test_gate_parity.py` checks the
|
src/{ourlid → wikalid}/head.py
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/infer.py
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/lexicon.py
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/model/MANIFEST.json
RENAMED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
{
|
| 2 |
-
"model": "
|
| 3 |
"built_from": "research repo /workspace/ourlid @ 680798d, models/ourlid-c7 (tok_c7a + tok_c7b) + eval/gate_v2_head.pkl exported to gate_head.npz",
|
| 4 |
"files": {
|
| 5 |
"conformal.json": "c389f66562e1e74bf6675fc3f3b81a08fa85040a1fdff4f3ac1a561b140f2bb0",
|
|
@@ -13,5 +13,6 @@
|
|
| 13 |
"model_int8_b.npz": "d3bb7d051f9576266b00b24551d99698f9e1c5d0bd497d415a9f0a3666c0b2d0",
|
| 14 |
"r3opts.json": "c7ea93a8c6e1fef34be26f812367ea02773fe22f97d57d743de114222a980b7d",
|
| 15 |
"train_cfg.json": "6442e9af7754497da62cec822c5884993eba601ec2110a519e7cf43b61a2b613"
|
| 16 |
-
}
|
|
|
|
| 17 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"model": "wikalid-1.0.1",
|
| 3 |
"built_from": "research repo /workspace/ourlid @ 680798d, models/ourlid-c7 (tok_c7a + tok_c7b) + eval/gate_v2_head.pkl exported to gate_head.npz",
|
| 4 |
"files": {
|
| 5 |
"conformal.json": "c389f66562e1e74bf6675fc3f3b81a08fa85040a1fdff4f3ac1a561b140f2bb0",
|
|
|
|
| 13 |
"model_int8_b.npz": "d3bb7d051f9576266b00b24551d99698f9e1c5d0bd497d415a9f0a3666c0b2d0",
|
| 14 |
"r3opts.json": "c7ea93a8c6e1fef34be26f812367ea02773fe22f97d57d743de114222a980b7d",
|
| 15 |
"train_cfg.json": "6442e9af7754497da62cec822c5884993eba601ec2110a519e7cf43b61a2b613"
|
| 16 |
+
},
|
| 17 |
+
"note": "Weights identical to OurLID 1.0.0 (renamed to WikaLID in 1.0.1)."
|
| 18 |
}
|
src/{ourlid → wikalid}/model/conformal.json
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/model/ensemble.txt
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/model/gate_head.npz
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/model/gate_thresholds.json
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/model/head.npz
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/model/head_report.json
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/model/lexicon.json
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/model/model_int8.npz
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/model/model_int8_b.npz
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/model/r3opts.json
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/model/train_cfg.json
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/predictor.py
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/r3.py
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/rules.py
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/sidecar.py
RENAMED
|
@@ -1,8 +1,8 @@
|
|
| 1 |
"""Batch sidecar for datasets: score a documents parquet (columns document_id, source_name, text) and apply gate v2.
|
| 2 |
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
|
| 7 |
Column names and values are identical to the research-repo scripts (scripts/run_release.py + scripts/gate_v2_apply.py)
|
| 8 |
that built every Himay sidecar since v0.2.9.5; `phc release lid` validates the result against the release."""
|
|
|
|
| 1 |
"""Batch sidecar for datasets: score a documents parquet (columns document_id, source_name, text) and apply gate v2.
|
| 2 |
|
| 3 |
+
wikalid score docs.parquet raw.parquet --workers 3 # model columns (ourlid_label, ourlid_conf, ...), resumable
|
| 4 |
+
wikalid gate raw.parquet docs.parquet gated.parquet # + ourlid_label_final, ourlid_oth_sub, gate_v2, gate_v2_reason, gate_v2_p_correct
|
| 5 |
+
wikalid sidecar docs.parquet gated.parquet # both
|
| 6 |
|
| 7 |
Column names and values are identical to the research-repo scripts (scripts/run_release.py + scripts/gate_v2_apply.py)
|
| 8 |
that built every Himay sidecar since v0.2.9.5; `phc release lid` validates the result against the release."""
|
src/{ourlid → wikalid}/synth.py
RENAMED
|
File without changes
|
src/{ourlid → wikalid}/textutil.py
RENAMED
|
File without changes
|
tests/test_package.py
CHANGED
|
@@ -1,9 +1,9 @@
|
|
| 1 |
-
"""
|
| 2 |
import json, os, subprocess, sys
|
| 3 |
import numpy as np
|
| 4 |
import pytest
|
| 5 |
-
import
|
| 6 |
-
from
|
| 7 |
|
| 8 |
EX = {"tgl": "Nag-submit na ako ng report pero hindi pa daw na-approve ng boss namin kahapon.",
|
| 9 |
"hil": "Indi ko mahibal-an kon ngaa wala sila nagkadto sa balay ni Lola sang Domingo.",
|
|
@@ -14,18 +14,18 @@ EX = {"tgl": "Nag-submit na ako ng report pero hindi pa daw na-approve ng boss n
|
|
| 14 |
|
| 15 |
@pytest.fixture(scope="module")
|
| 16 |
def P():
|
| 17 |
-
return
|
| 18 |
|
| 19 |
|
| 20 |
def test_version_and_labels():
|
| 21 |
-
assert
|
| 22 |
-
assert len(
|
| 23 |
|
| 24 |
|
| 25 |
def test_manifest_matches_bundled_weights():
|
| 26 |
-
r = subprocess.run([sys.executable, "-m", "
|
| 27 |
assert r.returncode == 0, r.stdout + r.stderr
|
| 28 |
-
bad = subprocess.run([sys.executable, "-m", "
|
| 29 |
assert bad.returncode == 1
|
| 30 |
|
| 31 |
|
|
@@ -41,19 +41,19 @@ def test_taglish_is_a_statistic_not_a_class(P):
|
|
| 41 |
|
| 42 |
|
| 43 |
def test_deterministic(P):
|
| 44 |
-
a, b = P.analyze(EX["tgl"]),
|
| 45 |
assert a == b
|
| 46 |
|
| 47 |
|
| 48 |
def test_classify_gate_fields():
|
| 49 |
-
r =
|
| 50 |
assert r["gate"] == "accept" and r["gate_reason"] == "accepted" and 0 < r["gate_p_correct"] <= 1 and r["label_final"] == "tgl"
|
| 51 |
-
j =
|
| 52 |
assert j["gate"] in ("junk", "review")
|
| 53 |
|
| 54 |
|
| 55 |
def test_aklanon_rule_bucket():
|
| 56 |
-
r =
|
| 57 |
assert r["label_final"] == "oth" and r["oth_sub"] == "akl" and r["gate"] == "accept_ph_other"
|
| 58 |
|
| 59 |
|
|
@@ -63,19 +63,28 @@ def test_tree_head_matches_sklearn_when_available():
|
|
| 63 |
if not os.path.exists(pkl):
|
| 64 |
pytest.skip("research gate pickle not on this machine")
|
| 65 |
import pickle
|
| 66 |
-
h = pickle.load(open(pkl, "rb")); T = TreeHead(os.path.join(
|
| 67 |
X = np.random.default_rng(0).random((2000, 20)) * 5; X[::40, 5] = np.nan
|
| 68 |
assert (h.predict_proba(X)[:, 1] == T.p_correct(X)).all()
|
| 69 |
|
| 70 |
|
| 71 |
def test_sidecar_roundtrip(tmp_path):
|
| 72 |
pa = pytest.importorskip("pyarrow"); import pyarrow.parquet as pq
|
| 73 |
-
from
|
| 74 |
ids = sorted(EX)
|
| 75 |
pq.write_table(pa.table({"document_id": ids, "source_name": ["s"] * len(ids), "text": [EX[i] for i in ids]}), tmp_path / "d.parquet")
|
| 76 |
rep = sidecar(str(tmp_path / "d.parquet"), str(tmp_path / "o.parquet"), workers=1, log=lambda *a: None)
|
| 77 |
t = pq.read_table(tmp_path / "o.parquet")
|
| 78 |
assert t.column("ourlid_label_final").to_pylist() == ids
|
| 79 |
-
assert t.schema.metadata[b"ourlid_version"] == b"1.0.
|
| 80 |
for c in ("gate_v2", "gate_v2_reason", "gate_v2_p_correct", "ourlid_oth_sub", "text_sha256"):
|
| 81 |
assert c in t.column_names
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""WikaLID 1.0.1 package tests: version pin, weight manifest, labels on known texts, gate decisions, determinism, sidecar."""
|
| 2 |
import json, os, subprocess, sys
|
| 3 |
import numpy as np
|
| 4 |
import pytest
|
| 5 |
+
import wikalid
|
| 6 |
+
from wikalid.gate import Gate, TreeHead
|
| 7 |
|
| 8 |
EX = {"tgl": "Nag-submit na ako ng report pero hindi pa daw na-approve ng boss namin kahapon.",
|
| 9 |
"hil": "Indi ko mahibal-an kon ngaa wala sila nagkadto sa balay ni Lola sang Domingo.",
|
|
|
|
| 14 |
|
| 15 |
@pytest.fixture(scope="module")
|
| 16 |
def P():
|
| 17 |
+
return wikalid.load()
|
| 18 |
|
| 19 |
|
| 20 |
def test_version_and_labels():
|
| 21 |
+
assert wikalid.__version__ == "1.0.1" and wikalid.TOOL_ID == "wikalid-1.0.1+gate_v2"
|
| 22 |
+
assert len(wikalid.LABELS) == 17 and "nol" in wikalid.LABELS
|
| 23 |
|
| 24 |
|
| 25 |
def test_manifest_matches_bundled_weights():
|
| 26 |
+
r = subprocess.run([sys.executable, "-m", "wikalid", "verify", "--expect-version", "1.0.1"], capture_output=True, text=True)
|
| 27 |
assert r.returncode == 0, r.stdout + r.stderr
|
| 28 |
+
bad = subprocess.run([sys.executable, "-m", "wikalid", "verify", "--expect-version", "9.9.9"], capture_output=True, text=True)
|
| 29 |
assert bad.returncode == 1
|
| 30 |
|
| 31 |
|
|
|
|
| 41 |
|
| 42 |
|
| 43 |
def test_deterministic(P):
|
| 44 |
+
a, b = P.analyze(EX["tgl"]), wikalid.load().analyze(EX["tgl"])
|
| 45 |
assert a == b
|
| 46 |
|
| 47 |
|
| 48 |
def test_classify_gate_fields():
|
| 49 |
+
r = wikalid.classify(EX["tgl"])
|
| 50 |
assert r["gate"] == "accept" and r["gate_reason"] == "accepted" and 0 < r["gate_p_correct"] <= 1 and r["label_final"] == "tgl"
|
| 51 |
+
j = wikalid.classify("1. Juan Dela Cruz 2. Pedro Penduko 3. Maria Clara 4. Jose Rizal")
|
| 52 |
assert j["gate"] in ("junk", "review")
|
| 53 |
|
| 54 |
|
| 55 |
def test_aklanon_rule_bucket():
|
| 56 |
+
r = wikalid.classify("Ro Altavas, a opisyal nga Banwa it Altavas, hay sangka banwa sa probinsya it Aklan. Eon ro mayad-ayad nga baeay.")
|
| 57 |
assert r["label_final"] == "oth" and r["oth_sub"] == "akl" and r["gate"] == "accept_ph_other"
|
| 58 |
|
| 59 |
|
|
|
|
| 63 |
if not os.path.exists(pkl):
|
| 64 |
pytest.skip("research gate pickle not on this machine")
|
| 65 |
import pickle
|
| 66 |
+
h = pickle.load(open(pkl, "rb")); T = TreeHead(os.path.join(wikalid.model_dir(), "gate_head.npz"))
|
| 67 |
X = np.random.default_rng(0).random((2000, 20)) * 5; X[::40, 5] = np.nan
|
| 68 |
assert (h.predict_proba(X)[:, 1] == T.p_correct(X)).all()
|
| 69 |
|
| 70 |
|
| 71 |
def test_sidecar_roundtrip(tmp_path):
|
| 72 |
pa = pytest.importorskip("pyarrow"); import pyarrow.parquet as pq
|
| 73 |
+
from wikalid.sidecar import sidecar
|
| 74 |
ids = sorted(EX)
|
| 75 |
pq.write_table(pa.table({"document_id": ids, "source_name": ["s"] * len(ids), "text": [EX[i] for i in ids]}), tmp_path / "d.parquet")
|
| 76 |
rep = sidecar(str(tmp_path / "d.parquet"), str(tmp_path / "o.parquet"), workers=1, log=lambda *a: None)
|
| 77 |
t = pq.read_table(tmp_path / "o.parquet")
|
| 78 |
assert t.column("ourlid_label_final").to_pylist() == ids
|
| 79 |
+
assert t.schema.metadata[b"ourlid_version"] == b"1.0.1" and sum(rep["gate"].values()) == len(ids)
|
| 80 |
for c in ("gate_v2", "gate_v2_reason", "gate_v2_p_correct", "ourlid_oth_sub", "text_sha256"):
|
| 81 |
assert c in t.column_names
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
def test_ourlid_alias_still_works():
|
| 85 |
+
import ourlid
|
| 86 |
+
from ourlid.gate import Gate as G2
|
| 87 |
+
from ourlid.r3 import PredictorR3 # noqa: F401
|
| 88 |
+
assert ourlid.__version__ == wikalid.__version__ and G2 is Gate and ourlid.classify(EX["tgl"])["label_final"] == "tgl"
|
| 89 |
+
r = subprocess.run([sys.executable, "-m", "ourlid", "verify", "--expect-version", "1.0.1"], capture_output=True, text=True)
|
| 90 |
+
assert r.returncode == 0, r.stdout + r.stderr
|
tests/test_rules.py
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
import sys
|
| 2 |
-
from
|
| 3 |
AKL_POS = ["Gusto ko ro maeamig nga tubi, owa ako't kwarta.", "Ro uwa gatan-aw sa anang ginhalinan hay indi makaabut sa anang ginapaeangpan.", "Hin-uno ka gapanaw, ro baeay mo hay siin?", "Mayad-ayad nga agahon, ro ngaean ko hay Maria.", "Ro ngaean ko si Tom.", "Ro banwa hay ginabug-usan it mga barangay ag ro eugta hay mayad.", "Ro Altavas hay sangka banwa sa probinsya it Aklan.", "Guinakalipay ko ro pakipagkita kimo, imaw ro ginsiling ni sanda eon.", "Saeamat sa imo, eon ako maeakat ag imaw ro nagpauna."]
|
| 4 |
SGD_POS = ["An Alegria usa ka lungsod nan Surigao del Norte sanan may gilapdon ini na 65 ka kilometro kwadrado.", "Haman ka paingon sija sanan oman an ginhimo nan lungsod?", "Gitudlo-an ini nan 86 na area code, sanan pinaka-ulahi na sensus."]
|
| 5 |
NEG = ["Ang bata ay naglalaro sa labas kasama ang kanyang mga kaibigan.", "Gusto ko nga mokaon ug saging kay gutom na ko.", "Ang akon mga kaibigan nagapanguli na sa balay kag nagakaon.",
|
|
|
|
| 1 |
import sys
|
| 2 |
+
from wikalid import rules
|
| 3 |
AKL_POS = ["Gusto ko ro maeamig nga tubi, owa ako't kwarta.", "Ro uwa gatan-aw sa anang ginhalinan hay indi makaabut sa anang ginapaeangpan.", "Hin-uno ka gapanaw, ro baeay mo hay siin?", "Mayad-ayad nga agahon, ro ngaean ko hay Maria.", "Ro ngaean ko si Tom.", "Ro banwa hay ginabug-usan it mga barangay ag ro eugta hay mayad.", "Ro Altavas hay sangka banwa sa probinsya it Aklan.", "Guinakalipay ko ro pakipagkita kimo, imaw ro ginsiling ni sanda eon.", "Saeamat sa imo, eon ako maeakat ag imaw ro nagpauna."]
|
| 4 |
SGD_POS = ["An Alegria usa ka lungsod nan Surigao del Norte sanan may gilapdon ini na 65 ka kilometro kwadrado.", "Haman ka paingon sija sanan oman an ginhimo nan lungsod?", "Gitudlo-an ini nan 86 na area code, sanan pinaka-ulahi na sensus."]
|
| 5 |
NEG = ["Ang bata ay naglalaro sa labas kasama ang kanyang mga kaibigan.", "Gusto ko nga mokaon ug saging kay gutom na ko.", "Ang akon mga kaibigan nagapanguli na sa balay kag nagakaon.",
|