wikalid / docs /DATA_CARD.md
jpaulpoliquit's picture
OurLID 1.0.0 (private preview): package, weights, model card, reproducible eval
863c520 verified
|
Raw History Blame Contribute Delete
4.04 kB

OurLID training data card (DRAFT v0 - 2026-10-10)

Composition (sentences, data/work/corpus_v0.parquet, split by hash of source doc/domain so test domains are unseen)

lang R (redistributable) W (web, pointer-only) G (.gov.ph web, pointer-only) total
tgl 219,729 370,875 4,185 594,789
ceb 156,739 248,125 11,493 416,357
ilo 77,590 194,250 4,558 276,398
war 255,616 7,885 2,838 266,339
bcl 103,451 82,622 2,952 189,025
hil 42 105,267 11,050 116,359
pam 47,182 44,212 835 92,229
cbk 10,436 2,837 9 13,282
pag 3,482 8,474 752 12,708
krj (experimental) 0 3,048 191 3,239
tsg (experimental) 0 1,990 0 1,990
eng / spa / ind / msa 237,717 / 36,515 / 75,079 / 59,596 0 / 59,592 / 73,033 / 59,557 -
oth (30+ languages, Wikipedia) 333,610 0 0 333,610
Splits: train 2,641,543 / dev 124,324 / test 151,547 (split key = source article/domain, so no sentence from a test domain is in train).
Important: Hiligaynon has no redistributable text (no Wikipedia edition); every hil training sentence is web-derived. Same for krj/tsg. These classes can only ever ship as "weights trained on pointer-only data", never as data.

Sources and licences

source what licence / terms redistribute text? use
Wikipedia dumps via wikimedia/wikipedia 20231101 (tl, ceb, ilo, war, pam, bcl, pag, cbk-zam, en, es, id, ms + ~30 oth languages) encyclopedic paragraphs split into sentences CC BY-SA 4.0 (+GFDL) per Wikimedia terms yes, with attribution + share-alike train, dev, test, benchmark (Tier R)
Tatoeba (tgl, ceb, ilo, war, ... sentence exports) short sentences CC BY 2.0 FR (some CC0) per Tatoeba terms yes with attribution train, benchmark (Tier R)
FineWeb-2 (HuggingFaceFW/fineweb-2: hil, war, ilo, bcl, pam, pag, cbk, tsg, krj, ceb, fil, ind, zsm, spa, eng) CommonCrawl web text; its own language label = GlotLID-v3 (Apache-2.0) ODC-By 1.0 + CommonCrawl ToU; page copyrights stay with publishers no (publish URL + doc id + our label only) train (Tier W / G) after multi-view purity filtering (seed lexicon + domain deny-list)
.gov.ph pages inside FineWeb-2 Philippine government text RA 8293 s.176 (govt works not copyrightable but some require approval for commercial use) - treated as W until reviewed no train (Tier G)
deny-list jw.org, bible.is, biblegateway, bibles.org, adult/spam, machine-translated farms copyrighted translations / unsafe n/a excluded from every tier
private evaluation only our news gold (299 docs/593 spans, single annotator); PLD (UP read-speech prompts); halo-hil/halo-bcl/BantayWika/halohalo (sapinsapin); our news crawl; FLORES+ (CC BY-SA 4.0, gated, eval-only policy) NC / unclear / research-only no evaluation; never in training
teachers GlotLID-v3 (Apache-2.0) labels via FineWeb-2 only. Not used: halo-lid (CC-BY-NC-4.0), OpenLID (GPL-3.0), lid201 (CC-BY-NC), MMS baselines only

Filters

Sentence split + length 20-400 chars; alpha ratio >= 0.5; exact dedup; Tier-W rows dropped when domain deny-list hits, or when the marker-lexicon purity test says the row is more consistent with another PH language (scripts/filter_corpus.py); cross-tier dedup so test text is never in train; Wikipedia-bot boilerplate capped per template prefix.

Known issues / to verify

  • Tier-W hil/ceb labels come from GlotLID-v3 and are noisy; purity filtering reduces but does not remove errors (halo-hil card: original label 44 % English). A native-speaker audit of 200 rows per class is required before release.
  • Wikipedia-derived ceb/war are dominated by Lsjbot article templates (species, communes): the model must not learn those topics as language (see lexicon cross-domain filter).
  • Wikipedia tl is more formal than tabloid/social text; hence the augmentation + synthetic CS design.