Download docs/BENCHMARK_CARD.md from sapinsapin/wikalid: direct link, hf CLI and curl.
- Browser
- Download file 3.18 kB
-
https://huggingface.co/sapinsapin/wikalid/resolve/main/docs/BENCHMARK_CARD.md
- Command line
-
hf download hf://sapinsapin/wikalid/docs/BENCHMARK_CARD.md
-
curl -L -o BENCHMARK_CARD.md https://huggingface.co/sapinsapin/wikalid/resolve/main/docs/BENCHMARK_CARD.md
PH-LID-Hard v0 - benchmark card (DRAFT)
What: 12,880 short texts with a single gold label for language-identification stress testing on Philippine text. File: eval/phlid_hard_v0.jsonl (id, cat, label, text, n_words, cs). Built by scripts/build_benchmark.py with fixed seed 20261010.
Source text: only Tier-R test-split sentences/paragraphs (split key = source article, so no text from the training split): Wikipedia dumps (CC BY-SA 4.0) and Tatoeba (CC BY 2.0 FR / CC0). No scraped news, no private data. Licence of the benchmark: CC BY-SA 4.0 (inherits share-alike from Wikipedia); attribution list = article titles are not retained in v0 (TODO before release: keep doc titles as attribution manifest).
Labels (15): tgl ceb hil war ilo bcl pam pag cbk eng spa ind msa oth nol. Label = language of the source sentence (known, not annotated), except quoted_english where label = the narrative frame language (convention, see below) and listing = nol.
| category | n | construction |
|---|---|---|
| clean_sent | 3,822 | 30-300 char sentences, 14 languages x ~300 |
| paragraph | 858 | up to 8 consecutive sentences (<=1500 chars) |
| short_1_5w | 2,597 | random 1-5 word windows of clean sentences |
| allcaps / lowercase_nopunct / texting / typos | 840 each | upper-casing; lower+no punctuation; rule-based SMS spelling (vowel dropping, kc/nmn/tlga abbreviations); random character deletion |
| html_noise | 700 | line-level web boilerplate ("Share on Facebook", "Read more") around the sentence |
| taglish_light / taglish_heavy | 460 / 423 | Tagalog sentence with 1-2 / 6 inserted English nouns/verbs, incl. Tagalog-affixed English (pina-comment, nag-submit). Template-generated, not natural Taglish: ungrammatical in places; it tests robustness to English insertions, not realism. Gold label = tgl; cs = light/heavy. |
| quoted_english | 360 | Tagalog (etc.) frame + English quote. Convention-dependent: label = frame language. Tools that label by majority token language legitimately answer eng; headline numbers therefore EXCLUDE this category and report it separately. |
| listing | 300 | numbered lists / name rosters / score tables, label nol (not a language). Only OpenLID's zxx and our nol can be right. |
Known weaknesses (do not over-claim)
- synthetic transformations: the benchmark measures robustness to kinds of noise, not the real-world frequency of each;
- only 120 hil items (no redistributable Hiligaynon text exists), none for krj/tsg;
- Wikipedia
pam/pag/cbksentences contain label noise (English/Spanish sentences inside those wikis); we did not hand-clean v0; - baselines were trained on Wikipedia/Tatoeba, so the clean categories favour them; and our own model trains on the Tier-R train split of the same sources (disjoint articles) - the two share generation families (we deliberately use different mechanisms in
build_benchmark.pythan insynth.py, but the benchmark is not independent of our design choices). Real-text evidence (gold news, PLD) is reported alongside and should be weighted higher. - single-annotator real-text sets are NOT part of the benchmark (private).