|
Download docs/BENCHMARK_CARD.md from sapinsapin/wikalid: direct link, hf CLI and curl.
- Browser
- Download file 3.18 kB
-
https://huggingface.co/sapinsapin/wikalid/resolve/main/docs/BENCHMARK_CARD.md
- Command line
-
hf download hf://sapinsapin/wikalid/docs/BENCHMARK_CARD.md
-
curl -L -o BENCHMARK_CARD.md https://huggingface.co/sapinsapin/wikalid/resolve/main/docs/BENCHMARK_CARD.md
3.18 kB
| # PH-LID-Hard v0 - benchmark card (DRAFT) | |
| **What**: 12,880 short texts with a single gold label for language-identification stress testing on Philippine text. File: `eval/phlid_hard_v0.jsonl` (`id, cat, label, text, n_words, cs`). Built by `scripts/build_benchmark.py` with fixed seed 20261010. | |
| **Source text**: only Tier-R *test-split* sentences/paragraphs (split key = source article, so no text from the training split): Wikipedia dumps (CC BY-SA 4.0) and Tatoeba (CC BY 2.0 FR / CC0). No scraped news, no private data. **Licence of the benchmark: CC BY-SA 4.0** (inherits share-alike from Wikipedia); attribution list = article titles are not retained in v0 (TODO before release: keep `doc` titles as attribution manifest). | |
| **Labels (15)**: tgl ceb hil war ilo bcl pam pag cbk eng spa ind msa oth nol. Label = language of the *source sentence* (known, not annotated), except `quoted_english` where label = the narrative frame language (convention, see below) and `listing` = `nol`. | |
| | category | n | construction | | |
| |---|---|---| | |
| | clean_sent | 3,822 | 30-300 char sentences, 14 languages x ~300 | | |
| | paragraph | 858 | up to 8 consecutive sentences (<=1500 chars) | | |
| | short_1_5w | 2,597 | random 1-5 word windows of clean sentences | | |
| | allcaps / lowercase_nopunct / texting / typos | 840 each | upper-casing; lower+no punctuation; rule-based SMS spelling (vowel dropping, kc/nmn/tlga abbreviations); random character deletion | | |
| | html_noise | 700 | line-level web boilerplate ("Share on Facebook", "Read more") around the sentence | | |
| | taglish_light / taglish_heavy | 460 / 423 | Tagalog sentence with 1-2 / 6 inserted English nouns/verbs, incl. Tagalog-affixed English (pina-comment, nag-submit). **Template-generated, not natural Taglish**: ungrammatical in places; it tests robustness to English insertions, not realism. Gold label = tgl; `cs` = light/heavy. | | |
| | quoted_english | 360 | Tagalog (etc.) frame + English quote. **Convention-dependent**: label = frame language. Tools that label by majority token language legitimately answer `eng`; headline numbers therefore EXCLUDE this category and report it separately. | | |
| | listing | 300 | numbered lists / name rosters / score tables, label `nol` (not a language). Only OpenLID's `zxx` and our `nol` can be right. | | |
| **Known weaknesses (do not over-claim)** | |
| * synthetic transformations: the benchmark measures robustness to *kinds* of noise, not the real-world frequency of each; | |
| * only 120 hil items (no redistributable Hiligaynon text exists), none for krj/tsg; | |
| * Wikipedia `pam`/`pag`/`cbk` sentences contain label noise (English/Spanish sentences inside those wikis); we did not hand-clean v0; | |
| * baselines were trained on Wikipedia/Tatoeba, so the clean categories favour them; and our own model trains on the Tier-R *train* split of the same sources (disjoint articles) - the two share generation *families* (we deliberately use different mechanisms in `build_benchmark.py` than in `synth.py`, but the benchmark is not independent of our design choices). Real-text evidence (gold news, PLD) is reported alongside and should be weighted higher. | |
| * single-annotator real-text sets are NOT part of the benchmark (private). | |