|
Download docs/DATA_CARD.md from sapinsapin/wikalid: direct link, hf CLI and curl.
- Browser
- Download file 4.04 kB
-
https://huggingface.co/sapinsapin/wikalid/resolve/main/docs/DATA_CARD.md
- Command line
-
hf download hf://sapinsapin/wikalid/docs/DATA_CARD.md
-
curl -L -o DATA_CARD.md https://huggingface.co/sapinsapin/wikalid/resolve/main/docs/DATA_CARD.md
4.04 kB
OurLID training data card (DRAFT v0 - 2026-10-10)
Composition (sentences, data/work/corpus_v0.parquet, split by hash of source doc/domain so test domains are unseen)
| lang | R (redistributable) | W (web, pointer-only) | G (.gov.ph web, pointer-only) | total |
|---|---|---|---|---|
| tgl | 219,729 | 370,875 | 4,185 | 594,789 |
| ceb | 156,739 | 248,125 | 11,493 | 416,357 |
| ilo | 77,590 | 194,250 | 4,558 | 276,398 |
| war | 255,616 | 7,885 | 2,838 | 266,339 |
| bcl | 103,451 | 82,622 | 2,952 | 189,025 |
| hil | 42 | 105,267 | 11,050 | 116,359 |
| pam | 47,182 | 44,212 | 835 | 92,229 |
| cbk | 10,436 | 2,837 | 9 | 13,282 |
| pag | 3,482 | 8,474 | 752 | 12,708 |
| krj (experimental) | 0 | 3,048 | 191 | 3,239 |
| tsg (experimental) | 0 | 1,990 | 0 | 1,990 |
| eng / spa / ind / msa | 237,717 / 36,515 / 75,079 / 59,596 | 0 / 59,592 / 73,033 / 59,557 | - | |
| oth (30+ languages, Wikipedia) | 333,610 | 0 | 0 | 333,610 |
| Splits: train 2,641,543 / dev 124,324 / test 151,547 (split key = source article/domain, so no sentence from a test domain is in train). | ||||
| Important: Hiligaynon has no redistributable text (no Wikipedia edition); every hil training sentence is web-derived. Same for krj/tsg. These classes can only ever ship as "weights trained on pointer-only data", never as data. |
Sources and licences
| source | what | licence / terms | redistribute text? | use |
|---|---|---|---|---|
Wikipedia dumps via wikimedia/wikipedia 20231101 (tl, ceb, ilo, war, pam, bcl, pag, cbk-zam, en, es, id, ms + ~30 oth languages) |
encyclopedic paragraphs split into sentences | CC BY-SA 4.0 (+GFDL) per Wikimedia terms | yes, with attribution + share-alike | train, dev, test, benchmark (Tier R) |
| Tatoeba (tgl, ceb, ilo, war, ... sentence exports) | short sentences | CC BY 2.0 FR (some CC0) per Tatoeba terms | yes with attribution | train, benchmark (Tier R) |
FineWeb-2 (HuggingFaceFW/fineweb-2: hil, war, ilo, bcl, pam, pag, cbk, tsg, krj, ceb, fil, ind, zsm, spa, eng) |
CommonCrawl web text; its own language label = GlotLID-v3 (Apache-2.0) | ODC-By 1.0 + CommonCrawl ToU; page copyrights stay with publishers | no (publish URL + doc id + our label only) | train (Tier W / G) after multi-view purity filtering (seed lexicon + domain deny-list) |
| .gov.ph pages inside FineWeb-2 | Philippine government text | RA 8293 s.176 (govt works not copyrightable but some require approval for commercial use) - treated as W until reviewed | no | train (Tier G) |
| deny-list | jw.org, bible.is, biblegateway, bibles.org, adult/spam, machine-translated farms | copyrighted translations / unsafe | n/a | excluded from every tier |
| private evaluation only | our news gold (299 docs/593 spans, single annotator); PLD (UP read-speech prompts); halo-hil/halo-bcl/BantayWika/halohalo (sapinsapin); our news crawl; FLORES+ (CC BY-SA 4.0, gated, eval-only policy) | NC / unclear / research-only | no | evaluation; never in training |
| teachers | GlotLID-v3 (Apache-2.0) labels via FineWeb-2 only. Not used: halo-lid (CC-BY-NC-4.0), OpenLID (GPL-3.0), lid201 (CC-BY-NC), MMS | baselines only |
Filters
Sentence split + length 20-400 chars; alpha ratio >= 0.5; exact dedup; Tier-W rows dropped when domain deny-list hits, or when the marker-lexicon purity test says the row is more consistent with another PH language (scripts/filter_corpus.py); cross-tier dedup so test text is never in train; Wikipedia-bot boilerplate capped per template prefix.
Known issues / to verify
- Tier-W hil/ceb labels come from GlotLID-v3 and are noisy;
purityfiltering reduces but does not remove errors (halo-hil card: original label 44 % English). A native-speaker audit of 200 rows per class is required before release. - Wikipedia-derived ceb/war are dominated by Lsjbot article templates (species, communes): the model must not learn those topics as language (see lexicon cross-domain filter).
- Wikipedia
tlis more formal than tabloid/social text; hence the augmentation + synthetic CS design.