wikalid / docs /DATA_CARD.md
jpaulpoliquit's picture
OurLID 1.0.0 (private preview): package, weights, model card, reproducible eval
863c520 verified
|
Raw History Blame Contribute Delete
4.04 kB
# OurLID training data card (DRAFT v0 - 2026-10-10)
## Composition (sentences, `data/work/corpus_v0.parquet`, split by hash of source doc/domain so test domains are unseen)
| lang | R (redistributable) | W (web, pointer-only) | G (.gov.ph web, pointer-only) | total |
|---|---|---|---|---|
| tgl | 219,729 | 370,875 | 4,185 | 594,789 |
| ceb | 156,739 | 248,125 | 11,493 | 416,357 |
| ilo | 77,590 | 194,250 | 4,558 | 276,398 |
| war | 255,616 | 7,885 | 2,838 | 266,339 |
| bcl | 103,451 | 82,622 | 2,952 | 189,025 |
| hil | **42** | 105,267 | 11,050 | 116,359 |
| pam | 47,182 | 44,212 | 835 | 92,229 |
| cbk | 10,436 | 2,837 | 9 | 13,282 |
| pag | 3,482 | 8,474 | 752 | 12,708 |
| krj (experimental) | 0 | 3,048 | 191 | 3,239 |
| tsg (experimental) | 0 | 1,990 | 0 | 1,990 |
| eng / spa / ind / msa | 237,717 / 36,515 / 75,079 / 59,596 | 0 / 59,592 / 73,033 / 59,557 | - | |
| oth (30+ languages, Wikipedia) | 333,610 | 0 | 0 | 333,610 |
Splits: train 2,641,543 / dev 124,324 / test 151,547 (split key = source article/domain, so no sentence from a test domain is in train).
**Important**: Hiligaynon has *no* redistributable text (no Wikipedia edition); every hil training sentence is web-derived. Same for krj/tsg. These classes can only ever ship as "weights trained on pointer-only data", never as data.
## Sources and licences
| source | what | licence / terms | redistribute text? | use |
|---|---|---|---|---|
| Wikipedia dumps via `wikimedia/wikipedia` 20231101 (tl, ceb, ilo, war, pam, bcl, pag, cbk-zam, en, es, id, ms + ~30 `oth` languages) | encyclopedic paragraphs split into sentences | CC BY-SA 4.0 (+GFDL) per Wikimedia terms | yes, with attribution + share-alike | train, dev, test, benchmark (Tier R) |
| Tatoeba (tgl, ceb, ilo, war, ... sentence exports) | short sentences | CC BY 2.0 FR (some CC0) per Tatoeba terms | yes with attribution | train, benchmark (Tier R) |
| FineWeb-2 (`HuggingFaceFW/fineweb-2`: hil, war, ilo, bcl, pam, pag, cbk, tsg, krj, ceb, fil, ind, zsm, spa, eng) | CommonCrawl web text; its own language label = GlotLID-v3 (Apache-2.0) | ODC-By 1.0 + CommonCrawl ToU; **page copyrights stay with publishers** | **no** (publish URL + doc id + our label only) | train (Tier W / G) after multi-view purity filtering (seed lexicon + domain deny-list) |
| .gov.ph pages inside FineWeb-2 | Philippine government text | RA 8293 s.176 (govt works not copyrightable but some require approval for commercial use) - **treated as W until reviewed** | no | train (Tier G) |
| deny-list | jw.org, bible.is, biblegateway, bibles.org, adult/spam, machine-translated farms | copyrighted translations / unsafe | n/a | excluded from every tier |
| **private evaluation only** | our news gold (299 docs/593 spans, single annotator); PLD (UP read-speech prompts); halo-hil/halo-bcl/BantayWika/halohalo (sapinsapin); our news crawl; FLORES+ (CC BY-SA 4.0, gated, eval-only policy) | NC / unclear / research-only | no | evaluation; never in training |
| teachers | GlotLID-v3 (Apache-2.0) labels via FineWeb-2 only. **Not used**: halo-lid (CC-BY-NC-4.0), OpenLID (GPL-3.0), lid201 (CC-BY-NC), MMS | | | baselines only |
## Filters
Sentence split + length 20-400 chars; alpha ratio >= 0.5; exact dedup; Tier-W rows dropped when domain deny-list hits, or when the marker-lexicon purity test says the row is more consistent with another PH language (`scripts/filter_corpus.py`); cross-tier dedup so test text is never in train; Wikipedia-bot boilerplate capped per template prefix.
## Known issues / to verify
* Tier-W hil/ceb labels come from GlotLID-v3 and are noisy; `purity` filtering reduces but does not remove errors (halo-hil card: original label 44 % English). A native-speaker audit of 200 rows per class is required before release.
* Wikipedia-derived ceb/war are dominated by Lsjbot article templates (species, communes): the model must not learn those topics as language (see lexicon cross-domain filter).
* Wikipedia `tl` is more formal than tabloid/social text; hence the augmentation + synthetic CS design.