# PH-LID-Hard v0 - benchmark card (DRAFT) **What**: 12,880 short texts with a single gold label for language-identification stress testing on Philippine text. File: `eval/phlid_hard_v0.jsonl` (`id, cat, label, text, n_words, cs`). Built by `scripts/build_benchmark.py` with fixed seed 20261010. **Source text**: only Tier-R *test-split* sentences/paragraphs (split key = source article, so no text from the training split): Wikipedia dumps (CC BY-SA 4.0) and Tatoeba (CC BY 2.0 FR / CC0). No scraped news, no private data. **Licence of the benchmark: CC BY-SA 4.0** (inherits share-alike from Wikipedia); attribution list = article titles are not retained in v0 (TODO before release: keep `doc` titles as attribution manifest). **Labels (15)**: tgl ceb hil war ilo bcl pam pag cbk eng spa ind msa oth nol. Label = language of the *source sentence* (known, not annotated), except `quoted_english` where label = the narrative frame language (convention, see below) and `listing` = `nol`. | category | n | construction | |---|---|---| | clean_sent | 3,822 | 30-300 char sentences, 14 languages x ~300 | | paragraph | 858 | up to 8 consecutive sentences (<=1500 chars) | | short_1_5w | 2,597 | random 1-5 word windows of clean sentences | | allcaps / lowercase_nopunct / texting / typos | 840 each | upper-casing; lower+no punctuation; rule-based SMS spelling (vowel dropping, kc/nmn/tlga abbreviations); random character deletion | | html_noise | 700 | line-level web boilerplate ("Share on Facebook", "Read more") around the sentence | | taglish_light / taglish_heavy | 460 / 423 | Tagalog sentence with 1-2 / 6 inserted English nouns/verbs, incl. Tagalog-affixed English (pina-comment, nag-submit). **Template-generated, not natural Taglish**: ungrammatical in places; it tests robustness to English insertions, not realism. Gold label = tgl; `cs` = light/heavy. | | quoted_english | 360 | Tagalog (etc.) frame + English quote. **Convention-dependent**: label = frame language. Tools that label by majority token language legitimately answer `eng`; headline numbers therefore EXCLUDE this category and report it separately. | | listing | 300 | numbered lists / name rosters / score tables, label `nol` (not a language). Only OpenLID's `zxx` and our `nol` can be right. | **Known weaknesses (do not over-claim)** * synthetic transformations: the benchmark measures robustness to *kinds* of noise, not the real-world frequency of each; * only 120 hil items (no redistributable Hiligaynon text exists), none for krj/tsg; * Wikipedia `pam`/`pag`/`cbk` sentences contain label noise (English/Spanish sentences inside those wikis); we did not hand-clean v0; * baselines were trained on Wikipedia/Tatoeba, so the clean categories favour them; and our own model trains on the Tier-R *train* split of the same sources (disjoint articles) - the two share generation *families* (we deliberately use different mechanisms in `build_benchmark.py` than in `synth.py`, but the benchmark is not independent of our design choices). Real-text evidence (gold news, PLD) is reported alongside and should be weighted higher. * single-annotator real-text sets are NOT part of the benchmark (private).