# OurLID 1.0.0 evaluation (generated by eval/evaluate.py; do not edit by hand) ## Sealed halves (never used for any decision): accuracy raw / corrected gold (macro-F1 corrected) | system | crawl (n=335) | goldspan (n=327) | golddoc (n=148) | pld (n=1653) | wiki* (n=2563) | hard* (n=6379) | basaha (n=794) | tato* (n=968) | mean corrected | |---|---|---|---|---|---|---|---|---|---| | ourlid-1.0.0 | 0.949 / 0.949 (0.95) | 0.939 / 0.939 (0.93) | 0.986 / 0.986 (1.00) | 0.916 / 0.917 (0.93) | 0.899 / 0.925 (0.93) | 0.879 / 0.901 (0.88) | 0.739 / 0.739 (0.79) | 0.931 / 0.928 (0.94) | **0.911** | | glotlid | 0.875 / 0.875 (0.93) | 0.899 / 0.899 (0.92) | 0.959 / 0.959 (1.00) | 0.830 / 0.834 (0.84) | 0.869 / 0.907 (0.91) | 0.799 / 0.829 (0.76) | 0.583 / 0.583 (0.64) | 0.949 / 0.955 (0.97) | **0.855** | | openlid3 | 0.785 / 0.785 (0.62) | 0.881 / 0.881 (0.87) | 0.919 / 0.919 (0.99) | 0.417 / 0.417 (0.40) | 0.676 / 0.710 (0.63) | 0.619 / 0.641 (0.55) | 0.026 / 0.026 (0.01) | 0.754 / 0.758 (0.68) | **0.642** | | lid176 | 0.734 / 0.734 (0.57) | 0.783 / 0.783 (0.74) | 0.818 / 0.818 (0.90) | 0.337 / 0.341 (0.29) | 0.620 / 0.664 (0.55) | 0.567 / 0.599 (0.45) | 0.031 / 0.031 (0.03) | 0.801 / 0.805 (0.80) | **0.597** | \* public set (ships in eval/data, anyone can re-run). Others are private (news gold, crawl realism, PLD read speech, BasahaCorpus stories). ## OurLID 1.0.0 minus GlotLID v3, sealed, corrected gold (paired bootstrap, 2,000 resamples) | set | delta | 95% CI | |---|---|---| | crawl | +0.075 | [+0.045, +0.110] | | goldspan | +0.040 | [+0.009, +0.073] | | golddoc | +0.027 | [+0.007, +0.054] | | pld | +0.083 | [+0.067, +0.099] | | wiki | +0.018 | [+0.010, +0.027] | | hard | +0.072 | [+0.063, +0.080] | | basaha | +0.156 | [+0.123, +0.189] | | tato | -0.027 | [-0.046, -0.009] | ## Gate v2 on sealed halves (accept = accept / accept_eng / accept_ph_other) | set | coverage | precision of accepted (corrected gold) | |---|---|---| | crawl | 0.510 | 1.000 | | goldspan | 0.514 | 1.000 | | golddoc | 0.784 | 0.991 | | pld | 0.658 | 0.985 | | wiki | 0.566 | 0.989 | | hard | 0.510 | 0.994 | | basaha | 0.358 | 0.972 | | tato | 0.395 | 0.995 | ## Human-labelled external trust sets (all rows; accuracy with 95% Wilson interval) ### t_sib (n=9036) | system | overall | ceb | eng | ilo | ind | msa | pag | sib200 | spa | tgl | war | |---|---|---|---|---|---|---|---|---|---|---|---| | ourlid-1.0.0 | 0.989 [0.99-0.99] | 0.997 | 1.000 | 1.000 | 0.959 | 0.953 | 0.997 | 0.989 | 0.997 | 1.000 | 1.000 | | glotlid | 0.986 [0.98-0.99] | 0.991 | 1.000 | 0.999 | 0.995 | 0.894 | 1.000 | 0.986 | 0.998 | 1.000 | 1.000 | | openlid3 | 0.985 [0.98-0.99] | 1.000 | 0.998 | 1.000 | 0.918 | 0.963 | 0.996 | 0.985 | 0.986 | 1.000 | 1.000 | | lid176 | 0.774 [0.77-0.78] | 0.922 | 1.000 | 0.985 | 0.969 | 0.449 | 0.000 | 0.774 | 1.000 | 0.998 | 0.640 | ### t_frag (n=4320) | system | overall | ceb | eng | ilo | ind | len2 | len3 | len4 | len5 | msa | pag | spa | tgl | war | |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | ourlid-1.0.0 | 0.832 [0.82-0.84] | 0.815 | 0.977 | 0.942 | 0.821 | 0.704 | 0.815 | 0.878 | 0.931 | 0.640 | 0.873 | 0.871 | 0.844 | 0.706 | | glotlid | 0.771 [0.76-0.78] | 0.760 | 0.908 | 0.892 | 0.867 | 0.624 | 0.739 | 0.835 | 0.887 | 0.342 | 0.823 | 0.850 | 0.867 | 0.633 | | openlid3 | 0.773 [0.76-0.79] | 0.842 | 0.717 | 0.933 | 0.810 | 0.621 | 0.750 | 0.834 | 0.888 | 0.550 | 0.790 | 0.692 | 0.887 | 0.740 | | lid176 | 0.590 [0.58-0.60] | 0.681 | 0.992 | 0.546 | 0.750 | 0.514 | 0.547 | 0.628 | 0.671 | 0.354 | 0.000 | 0.933 | 0.729 | 0.325 | ### t_caps (n=5388) | system | overall | ceb | eng | ilo | lower | pag | spa | tgl | title | upper | war | |---|---|---|---|---|---|---|---|---|---|---|---| | ourlid-1.0.0 | 0.999 [1.00-1.00] | 0.995 | 1.000 | 1.000 | 0.998 | 0.999 | 1.000 | 0.999 | 0.999 | 0.999 | 1.000 | | glotlid | 0.975 [0.97-0.98] | 0.959 | 0.931 | 0.999 | 0.998 | 0.994 | 0.949 | 0.996 | 0.990 | 0.937 | 1.000 | | openlid3 | 0.575 [0.56-0.59] | 0.646 | 0.455 | 0.661 | 0.999 | 0.609 | 0.546 | 0.631 | 0.726 | 0.000 | 0.492 | | lid176 | 0.522 [0.51-0.53] | 0.466 | 0.965 | 0.431 | 0.788 | 0.000 | 0.994 | 0.582 | 0.507 | 0.270 | 0.226 | ### t_bikol (n=1800) | system | overall | bcl | ceb | story | |---|---|---|---|---| | ourlid-1.0.0 | 0.940 [0.93-0.95] | 0.961 | 0.919 | 0.940 | | glotlid | 0.893 [0.88-0.91] | 0.894 | 0.892 | 0.893 | | openlid3 | 0.467 [0.44-0.49] | 0.000 | 0.933 | 0.467 | | lid176 | 0.291 [0.27-0.31] | 0.003 | 0.579 | 0.291 | ### t_aklsgd (n=110) | system | overall | incubator | oth | tatoeba | |---|---|---|---|---| | ourlid-1.0.0 | 0.345 [0.26-0.44] | 0.370 | 0.345 | 0.222 | | glotlid | 0.300 [0.22-0.39] | 0.272 | 0.300 | 0.444 | | openlid3 | 0.082 [0.04-0.15] | 0.043 | 0.082 | 0.278 | | lid176 | 0.055 [0.03-0.11] | 0.011 | 0.055 | 0.278 | ### t_tshort (n=543) | system | overall | cbk | ceb | eng | hil | ilo | ind | pag | pam | tatoeba | tgl | war | |---|---|---|---|---|---|---|---|---|---|---|---|---| | ourlid-1.0.0 | 0.856 [0.82-0.88] | 0.937 | 0.667 | 1.000 | 1.000 | 0.822 | 0.841 | 1.000 | 0.783 | 0.856 | 0.971 | 0.545 | | glotlid | 0.866 [0.83-0.89] | 0.646 | 0.795 | 0.983 | 0.000 | 0.889 | 0.914 | 1.000 | 0.826 | 0.866 | 0.990 | 0.727 | | openlid3 | 0.632 [0.59-0.67] | 0.000 | 0.846 | 0.655 | 0.000 | 0.844 | 0.861 | 1.000 | 0.000 | 0.632 | 0.794 | 0.500 | | lid176 | 0.685 [0.64-0.72] | 0.203 | 0.590 | 1.000 | 0.000 | 0.556 | 0.887 | 0.000 | 0.652 | 0.685 | 0.824 | 0.386 | ## Consistency with the research model that labels Himay (c7 + rules, cached predictions) | set | n | same label | |---|---|---| | crawl | 661 | 661 | | goldspan | 593 | 593 | | golddoc | 294 | 294 | | pld | 3222 | 3222 | | wiki | 5037 | 5037 | | hard | 12520 | 12520 | | basaha | 1600 | 1600 | | tato | 1847 | 1847 | | t_sib | 9036 | 9036 | | t_frag | 4320 | 4320 | | t_caps | 5388 | 5388 | | t_bikol | 1799 | 1799 | | t_aklsgd | 110 | 110 | | t_tshort | 543 | 543 | ## Throughput during this run (items/s, all workers, shared box) t_sib: 609 | t_frag: 589 | t_caps: 576 | t_bikol: 459 | t_aklsgd: 99 | t_tshort: 337