wikalid / docs /BENCHMARK_CARD.md
jpaulpoliquit's picture
OurLID 1.0.0 (private preview): package, weights, model card, reproducible eval
863c520 verified
|
Raw History Blame Contribute Delete
3.18 kB

PH-LID-Hard v0 - benchmark card (DRAFT)

What: 12,880 short texts with a single gold label for language-identification stress testing on Philippine text. File: eval/phlid_hard_v0.jsonl (id, cat, label, text, n_words, cs). Built by scripts/build_benchmark.py with fixed seed 20261010.

Source text: only Tier-R test-split sentences/paragraphs (split key = source article, so no text from the training split): Wikipedia dumps (CC BY-SA 4.0) and Tatoeba (CC BY 2.0 FR / CC0). No scraped news, no private data. Licence of the benchmark: CC BY-SA 4.0 (inherits share-alike from Wikipedia); attribution list = article titles are not retained in v0 (TODO before release: keep doc titles as attribution manifest).

Labels (15): tgl ceb hil war ilo bcl pam pag cbk eng spa ind msa oth nol. Label = language of the source sentence (known, not annotated), except quoted_english where label = the narrative frame language (convention, see below) and listing = nol.

category n construction
clean_sent 3,822 30-300 char sentences, 14 languages x ~300
paragraph 858 up to 8 consecutive sentences (<=1500 chars)
short_1_5w 2,597 random 1-5 word windows of clean sentences
allcaps / lowercase_nopunct / texting / typos 840 each upper-casing; lower+no punctuation; rule-based SMS spelling (vowel dropping, kc/nmn/tlga abbreviations); random character deletion
html_noise 700 line-level web boilerplate ("Share on Facebook", "Read more") around the sentence
taglish_light / taglish_heavy 460 / 423 Tagalog sentence with 1-2 / 6 inserted English nouns/verbs, incl. Tagalog-affixed English (pina-comment, nag-submit). Template-generated, not natural Taglish: ungrammatical in places; it tests robustness to English insertions, not realism. Gold label = tgl; cs = light/heavy.
quoted_english 360 Tagalog (etc.) frame + English quote. Convention-dependent: label = frame language. Tools that label by majority token language legitimately answer eng; headline numbers therefore EXCLUDE this category and report it separately.
listing 300 numbered lists / name rosters / score tables, label nol (not a language). Only OpenLID's zxx and our nol can be right.

Known weaknesses (do not over-claim)

  • synthetic transformations: the benchmark measures robustness to kinds of noise, not the real-world frequency of each;
  • only 120 hil items (no redistributable Hiligaynon text exists), none for krj/tsg;
  • Wikipedia pam/pag/cbk sentences contain label noise (English/Spanish sentences inside those wikis); we did not hand-clean v0;
  • baselines were trained on Wikipedia/Tatoeba, so the clean categories favour them; and our own model trains on the Tier-R train split of the same sources (disjoint articles) - the two share generation families (we deliberately use different mechanisms in build_benchmark.py than in synth.py, but the benchmark is not independent of our design choices). Real-text evidence (gold news, PLD) is reported alongside and should be weighted higher.
  • single-annotator real-text sets are NOT part of the benchmark (private).