wikalid / eval /RESULTS_v1.0.0.md
jpaulpoliquit's picture
OurLID 1.0.0 (private preview): package, weights, model card, reproducible eval
863c520 verified
|
Raw History Blame Contribute Delete
6.12 kB

OurLID 1.0.0 evaluation (generated by eval/evaluate.py; do not edit by hand)

Sealed halves (never used for any decision): accuracy raw / corrected gold (macro-F1 corrected)

system crawl (n=335) goldspan (n=327) golddoc (n=148) pld (n=1653) wiki* (n=2563) hard* (n=6379) basaha (n=794) tato* (n=968) mean corrected
ourlid-1.0.0 0.949 / 0.949 (0.95) 0.939 / 0.939 (0.93) 0.986 / 0.986 (1.00) 0.916 / 0.917 (0.93) 0.899 / 0.925 (0.93) 0.879 / 0.901 (0.88) 0.739 / 0.739 (0.79) 0.931 / 0.928 (0.94) 0.911
glotlid 0.875 / 0.875 (0.93) 0.899 / 0.899 (0.92) 0.959 / 0.959 (1.00) 0.830 / 0.834 (0.84) 0.869 / 0.907 (0.91) 0.799 / 0.829 (0.76) 0.583 / 0.583 (0.64) 0.949 / 0.955 (0.97) 0.855
openlid3 0.785 / 0.785 (0.62) 0.881 / 0.881 (0.87) 0.919 / 0.919 (0.99) 0.417 / 0.417 (0.40) 0.676 / 0.710 (0.63) 0.619 / 0.641 (0.55) 0.026 / 0.026 (0.01) 0.754 / 0.758 (0.68) 0.642
lid176 0.734 / 0.734 (0.57) 0.783 / 0.783 (0.74) 0.818 / 0.818 (0.90) 0.337 / 0.341 (0.29) 0.620 / 0.664 (0.55) 0.567 / 0.599 (0.45) 0.031 / 0.031 (0.03) 0.801 / 0.805 (0.80) 0.597

* public set (ships in eval/data, anyone can re-run). Others are private (news gold, crawl realism, PLD read speech, BasahaCorpus stories).

OurLID 1.0.0 minus GlotLID v3, sealed, corrected gold (paired bootstrap, 2,000 resamples)

set delta 95% CI
crawl +0.075 [+0.045, +0.110]
goldspan +0.040 [+0.009, +0.073]
golddoc +0.027 [+0.007, +0.054]
pld +0.083 [+0.067, +0.099]
wiki +0.018 [+0.010, +0.027]
hard +0.072 [+0.063, +0.080]
basaha +0.156 [+0.123, +0.189]
tato -0.027 [-0.046, -0.009]

Gate v2 on sealed halves (accept = accept / accept_eng / accept_ph_other)

set coverage precision of accepted (corrected gold)
crawl 0.510 1.000
goldspan 0.514 1.000
golddoc 0.784 0.991
pld 0.658 0.985
wiki 0.566 0.989
hard 0.510 0.994
basaha 0.358 0.972
tato 0.395 0.995

Human-labelled external trust sets (all rows; accuracy with 95% Wilson interval)

t_sib (n=9036)

system overall ceb eng ilo ind msa pag sib200 spa tgl war
ourlid-1.0.0 0.989 [0.99-0.99] 0.997 1.000 1.000 0.959 0.953 0.997 0.989 0.997 1.000 1.000
glotlid 0.986 [0.98-0.99] 0.991 1.000 0.999 0.995 0.894 1.000 0.986 0.998 1.000 1.000
openlid3 0.985 [0.98-0.99] 1.000 0.998 1.000 0.918 0.963 0.996 0.985 0.986 1.000 1.000
lid176 0.774 [0.77-0.78] 0.922 1.000 0.985 0.969 0.449 0.000 0.774 1.000 0.998 0.640

t_frag (n=4320)

system overall ceb eng ilo ind len2 len3 len4 len5 msa pag spa tgl war
ourlid-1.0.0 0.832 [0.82-0.84] 0.815 0.977 0.942 0.821 0.704 0.815 0.878 0.931 0.640 0.873 0.871 0.844 0.706
glotlid 0.771 [0.76-0.78] 0.760 0.908 0.892 0.867 0.624 0.739 0.835 0.887 0.342 0.823 0.850 0.867 0.633
openlid3 0.773 [0.76-0.79] 0.842 0.717 0.933 0.810 0.621 0.750 0.834 0.888 0.550 0.790 0.692 0.887 0.740
lid176 0.590 [0.58-0.60] 0.681 0.992 0.546 0.750 0.514 0.547 0.628 0.671 0.354 0.000 0.933 0.729 0.325

t_caps (n=5388)

system overall ceb eng ilo lower pag spa tgl title upper war
ourlid-1.0.0 0.999 [1.00-1.00] 0.995 1.000 1.000 0.998 0.999 1.000 0.999 0.999 0.999 1.000
glotlid 0.975 [0.97-0.98] 0.959 0.931 0.999 0.998 0.994 0.949 0.996 0.990 0.937 1.000
openlid3 0.575 [0.56-0.59] 0.646 0.455 0.661 0.999 0.609 0.546 0.631 0.726 0.000 0.492
lid176 0.522 [0.51-0.53] 0.466 0.965 0.431 0.788 0.000 0.994 0.582 0.507 0.270 0.226

t_bikol (n=1800)

system overall bcl ceb story
ourlid-1.0.0 0.940 [0.93-0.95] 0.961 0.919 0.940
glotlid 0.893 [0.88-0.91] 0.894 0.892 0.893
openlid3 0.467 [0.44-0.49] 0.000 0.933 0.467
lid176 0.291 [0.27-0.31] 0.003 0.579 0.291

t_aklsgd (n=110)

system overall incubator oth tatoeba
ourlid-1.0.0 0.345 [0.26-0.44] 0.370 0.345 0.222
glotlid 0.300 [0.22-0.39] 0.272 0.300 0.444
openlid3 0.082 [0.04-0.15] 0.043 0.082 0.278
lid176 0.055 [0.03-0.11] 0.011 0.055 0.278

t_tshort (n=543)

system overall cbk ceb eng hil ilo ind pag pam tatoeba tgl war
ourlid-1.0.0 0.856 [0.82-0.88] 0.937 0.667 1.000 1.000 0.822 0.841 1.000 0.783 0.856 0.971 0.545
glotlid 0.866 [0.83-0.89] 0.646 0.795 0.983 0.000 0.889 0.914 1.000 0.826 0.866 0.990 0.727
openlid3 0.632 [0.59-0.67] 0.000 0.846 0.655 0.000 0.844 0.861 1.000 0.000 0.632 0.794 0.500
lid176 0.685 [0.64-0.72] 0.203 0.590 1.000 0.000 0.556 0.887 0.000 0.652 0.685 0.824 0.386

Consistency with the research model that labels Himay (c7 + rules, cached predictions)

set n same label
crawl 661 661
goldspan 593 593
golddoc 294 294
pld 3222 3222
wiki 5037 5037
hard 12520 12520
basaha 1600 1600
tato 1847 1847
t_sib 9036 9036
t_frag 4320 4320
t_caps 5388 5388
t_bikol 1799 1799
t_aklsgd 110 110
t_tshort 543 543

Throughput during this run (items/s, all workers, shared box)

t_sib: 609 | t_frag: 589 | t_caps: 576 | t_bikol: 459 | t_aklsgd: 99 | t_tshort: 337