wikalid / docs /BENCHMARK_CARD.md
jpaulpoliquit's picture
OurLID 1.0.0 (private preview): package, weights, model card, reproducible eval
863c520 verified
|
Raw History Blame Contribute Delete
3.18 kB
# PH-LID-Hard v0 - benchmark card (DRAFT)
**What**: 12,880 short texts with a single gold label for language-identification stress testing on Philippine text. File: `eval/phlid_hard_v0.jsonl` (`id, cat, label, text, n_words, cs`). Built by `scripts/build_benchmark.py` with fixed seed 20261010.
**Source text**: only Tier-R *test-split* sentences/paragraphs (split key = source article, so no text from the training split): Wikipedia dumps (CC BY-SA 4.0) and Tatoeba (CC BY 2.0 FR / CC0). No scraped news, no private data. **Licence of the benchmark: CC BY-SA 4.0** (inherits share-alike from Wikipedia); attribution list = article titles are not retained in v0 (TODO before release: keep `doc` titles as attribution manifest).
**Labels (15)**: tgl ceb hil war ilo bcl pam pag cbk eng spa ind msa oth nol. Label = language of the *source sentence* (known, not annotated), except `quoted_english` where label = the narrative frame language (convention, see below) and `listing` = `nol`.
| category | n | construction |
|---|---|---|
| clean_sent | 3,822 | 30-300 char sentences, 14 languages x ~300 |
| paragraph | 858 | up to 8 consecutive sentences (<=1500 chars) |
| short_1_5w | 2,597 | random 1-5 word windows of clean sentences |
| allcaps / lowercase_nopunct / texting / typos | 840 each | upper-casing; lower+no punctuation; rule-based SMS spelling (vowel dropping, kc/nmn/tlga abbreviations); random character deletion |
| html_noise | 700 | line-level web boilerplate ("Share on Facebook", "Read more") around the sentence |
| taglish_light / taglish_heavy | 460 / 423 | Tagalog sentence with 1-2 / 6 inserted English nouns/verbs, incl. Tagalog-affixed English (pina-comment, nag-submit). **Template-generated, not natural Taglish**: ungrammatical in places; it tests robustness to English insertions, not realism. Gold label = tgl; `cs` = light/heavy. |
| quoted_english | 360 | Tagalog (etc.) frame + English quote. **Convention-dependent**: label = frame language. Tools that label by majority token language legitimately answer `eng`; headline numbers therefore EXCLUDE this category and report it separately. |
| listing | 300 | numbered lists / name rosters / score tables, label `nol` (not a language). Only OpenLID's `zxx` and our `nol` can be right. |
**Known weaknesses (do not over-claim)**
* synthetic transformations: the benchmark measures robustness to *kinds* of noise, not the real-world frequency of each;
* only 120 hil items (no redistributable Hiligaynon text exists), none for krj/tsg;
* Wikipedia `pam`/`pag`/`cbk` sentences contain label noise (English/Spanish sentences inside those wikis); we did not hand-clean v0;
* baselines were trained on Wikipedia/Tatoeba, so the clean categories favour them; and our own model trains on the Tier-R *train* split of the same sources (disjoint articles) - the two share generation *families* (we deliberately use different mechanisms in `build_benchmark.py` than in `synth.py`, but the benchmark is not independent of our design choices). Real-text evidence (gold news, PLD) is reported alongside and should be weighted higher.
* single-annotator real-text sets are NOT part of the benchmark (private).