OurLID 1.0.0 (private preview): package, weights, model card, reproducible eval
863c520 verified
OurLID 1.0.0 evaluation (generated by eval/evaluate.py; do not edit by hand)
Sealed halves (never used for any decision): accuracy raw / corrected gold (macro-F1 corrected)
| system |
crawl (n=335) |
goldspan (n=327) |
golddoc (n=148) |
pld (n=1653) |
wiki* (n=2563) |
hard* (n=6379) |
basaha (n=794) |
tato* (n=968) |
mean corrected |
| ourlid-1.0.0 |
0.949 / 0.949 (0.95) |
0.939 / 0.939 (0.93) |
0.986 / 0.986 (1.00) |
0.916 / 0.917 (0.93) |
0.899 / 0.925 (0.93) |
0.879 / 0.901 (0.88) |
0.739 / 0.739 (0.79) |
0.931 / 0.928 (0.94) |
0.911 |
| glotlid |
0.875 / 0.875 (0.93) |
0.899 / 0.899 (0.92) |
0.959 / 0.959 (1.00) |
0.830 / 0.834 (0.84) |
0.869 / 0.907 (0.91) |
0.799 / 0.829 (0.76) |
0.583 / 0.583 (0.64) |
0.949 / 0.955 (0.97) |
0.855 |
| openlid3 |
0.785 / 0.785 (0.62) |
0.881 / 0.881 (0.87) |
0.919 / 0.919 (0.99) |
0.417 / 0.417 (0.40) |
0.676 / 0.710 (0.63) |
0.619 / 0.641 (0.55) |
0.026 / 0.026 (0.01) |
0.754 / 0.758 (0.68) |
0.642 |
| lid176 |
0.734 / 0.734 (0.57) |
0.783 / 0.783 (0.74) |
0.818 / 0.818 (0.90) |
0.337 / 0.341 (0.29) |
0.620 / 0.664 (0.55) |
0.567 / 0.599 (0.45) |
0.031 / 0.031 (0.03) |
0.801 / 0.805 (0.80) |
0.597 |
* public set (ships in eval/data, anyone can re-run). Others are private (news gold, crawl realism, PLD read speech, BasahaCorpus stories).
OurLID 1.0.0 minus GlotLID v3, sealed, corrected gold (paired bootstrap, 2,000 resamples)
| set |
delta |
95% CI |
| crawl |
+0.075 |
[+0.045, +0.110] |
| goldspan |
+0.040 |
[+0.009, +0.073] |
| golddoc |
+0.027 |
[+0.007, +0.054] |
| pld |
+0.083 |
[+0.067, +0.099] |
| wiki |
+0.018 |
[+0.010, +0.027] |
| hard |
+0.072 |
[+0.063, +0.080] |
| basaha |
+0.156 |
[+0.123, +0.189] |
| tato |
-0.027 |
[-0.046, -0.009] |
Gate v2 on sealed halves (accept = accept / accept_eng / accept_ph_other)
| set |
coverage |
precision of accepted (corrected gold) |
| crawl |
0.510 |
1.000 |
| goldspan |
0.514 |
1.000 |
| golddoc |
0.784 |
0.991 |
| pld |
0.658 |
0.985 |
| wiki |
0.566 |
0.989 |
| hard |
0.510 |
0.994 |
| basaha |
0.358 |
0.972 |
| tato |
0.395 |
0.995 |
Human-labelled external trust sets (all rows; accuracy with 95% Wilson interval)
t_sib (n=9036)
| system |
overall |
ceb |
eng |
ilo |
ind |
msa |
pag |
sib200 |
spa |
tgl |
war |
| ourlid-1.0.0 |
0.989 [0.99-0.99] |
0.997 |
1.000 |
1.000 |
0.959 |
0.953 |
0.997 |
0.989 |
0.997 |
1.000 |
1.000 |
| glotlid |
0.986 [0.98-0.99] |
0.991 |
1.000 |
0.999 |
0.995 |
0.894 |
1.000 |
0.986 |
0.998 |
1.000 |
1.000 |
| openlid3 |
0.985 [0.98-0.99] |
1.000 |
0.998 |
1.000 |
0.918 |
0.963 |
0.996 |
0.985 |
0.986 |
1.000 |
1.000 |
| lid176 |
0.774 [0.77-0.78] |
0.922 |
1.000 |
0.985 |
0.969 |
0.449 |
0.000 |
0.774 |
1.000 |
0.998 |
0.640 |
t_frag (n=4320)
| system |
overall |
ceb |
eng |
ilo |
ind |
len2 |
len3 |
len4 |
len5 |
msa |
pag |
spa |
tgl |
war |
| ourlid-1.0.0 |
0.832 [0.82-0.84] |
0.815 |
0.977 |
0.942 |
0.821 |
0.704 |
0.815 |
0.878 |
0.931 |
0.640 |
0.873 |
0.871 |
0.844 |
0.706 |
| glotlid |
0.771 [0.76-0.78] |
0.760 |
0.908 |
0.892 |
0.867 |
0.624 |
0.739 |
0.835 |
0.887 |
0.342 |
0.823 |
0.850 |
0.867 |
0.633 |
| openlid3 |
0.773 [0.76-0.79] |
0.842 |
0.717 |
0.933 |
0.810 |
0.621 |
0.750 |
0.834 |
0.888 |
0.550 |
0.790 |
0.692 |
0.887 |
0.740 |
| lid176 |
0.590 [0.58-0.60] |
0.681 |
0.992 |
0.546 |
0.750 |
0.514 |
0.547 |
0.628 |
0.671 |
0.354 |
0.000 |
0.933 |
0.729 |
0.325 |
t_caps (n=5388)
| system |
overall |
ceb |
eng |
ilo |
lower |
pag |
spa |
tgl |
title |
upper |
war |
| ourlid-1.0.0 |
0.999 [1.00-1.00] |
0.995 |
1.000 |
1.000 |
0.998 |
0.999 |
1.000 |
0.999 |
0.999 |
0.999 |
1.000 |
| glotlid |
0.975 [0.97-0.98] |
0.959 |
0.931 |
0.999 |
0.998 |
0.994 |
0.949 |
0.996 |
0.990 |
0.937 |
1.000 |
| openlid3 |
0.575 [0.56-0.59] |
0.646 |
0.455 |
0.661 |
0.999 |
0.609 |
0.546 |
0.631 |
0.726 |
0.000 |
0.492 |
| lid176 |
0.522 [0.51-0.53] |
0.466 |
0.965 |
0.431 |
0.788 |
0.000 |
0.994 |
0.582 |
0.507 |
0.270 |
0.226 |
t_bikol (n=1800)
| system |
overall |
bcl |
ceb |
story |
| ourlid-1.0.0 |
0.940 [0.93-0.95] |
0.961 |
0.919 |
0.940 |
| glotlid |
0.893 [0.88-0.91] |
0.894 |
0.892 |
0.893 |
| openlid3 |
0.467 [0.44-0.49] |
0.000 |
0.933 |
0.467 |
| lid176 |
0.291 [0.27-0.31] |
0.003 |
0.579 |
0.291 |
t_aklsgd (n=110)
| system |
overall |
incubator |
oth |
tatoeba |
| ourlid-1.0.0 |
0.345 [0.26-0.44] |
0.370 |
0.345 |
0.222 |
| glotlid |
0.300 [0.22-0.39] |
0.272 |
0.300 |
0.444 |
| openlid3 |
0.082 [0.04-0.15] |
0.043 |
0.082 |
0.278 |
| lid176 |
0.055 [0.03-0.11] |
0.011 |
0.055 |
0.278 |
t_tshort (n=543)
| system |
overall |
cbk |
ceb |
eng |
hil |
ilo |
ind |
pag |
pam |
tatoeba |
tgl |
war |
| ourlid-1.0.0 |
0.856 [0.82-0.88] |
0.937 |
0.667 |
1.000 |
1.000 |
0.822 |
0.841 |
1.000 |
0.783 |
0.856 |
0.971 |
0.545 |
| glotlid |
0.866 [0.83-0.89] |
0.646 |
0.795 |
0.983 |
0.000 |
0.889 |
0.914 |
1.000 |
0.826 |
0.866 |
0.990 |
0.727 |
| openlid3 |
0.632 [0.59-0.67] |
0.000 |
0.846 |
0.655 |
0.000 |
0.844 |
0.861 |
1.000 |
0.000 |
0.632 |
0.794 |
0.500 |
| lid176 |
0.685 [0.64-0.72] |
0.203 |
0.590 |
1.000 |
0.000 |
0.556 |
0.887 |
0.000 |
0.652 |
0.685 |
0.824 |
0.386 |
Consistency with the research model that labels Himay (c7 + rules, cached predictions)
| set |
n |
same label |
| crawl |
661 |
661 |
| goldspan |
593 |
593 |
| golddoc |
294 |
294 |
| pld |
3222 |
3222 |
| wiki |
5037 |
5037 |
| hard |
12520 |
12520 |
| basaha |
1600 |
1600 |
| tato |
1847 |
1847 |
| t_sib |
9036 |
9036 |
| t_frag |
4320 |
4320 |
| t_caps |
5388 |
5388 |
| t_bikol |
1799 |
1799 |
| t_aklsgd |
110 |
110 |
| t_tshort |
543 |
543 |
Throughput during this run (items/s, all workers, shared box)
t_sib: 609 | t_frag: 589 | t_caps: 576 | t_bikol: 459 | t_aklsgd: 99 | t_tshort: 337