wikalid / docs /LABELING_GUIDE_NATIVE.md
jpaulpoliquit's picture
OurLID 1.0.0 (private preview): package, weights, model card, reproducible eval
863c520 verified
|
Raw History Blame Contribute Delete
7.37 kB

Native-speaker language labeling guide

You will see about 675 short texts (at most 300 characters each). For each one, say which language it is written in. There are no right-or-wrong "tricks": we want your honest native judgement. About 50 rows are repeated on purpose (to measure consistency) - just label every row on its own, as if you had never seen it.

Time: about 1-1.5 minutes per 10 rows, so roughly 1.5 hours in total. You can stop and continue later; leave unfinished rows empty.

What to fill in (columns E-H of the sheet; columns A-C are the item)

column what to put
annotator_label one code from the list below (dropdown in the Excel file)
code_switched yes if a second language also appears in the text (e.g. Taglish), otherwise no
confidence high (sure), medium, or low (guessing)
note optional; required for oth_ph and oth (write the language name), helpful for anything odd

Codes

code use it for
tgl Tagalog / Filipino, including Taglish whose sentence frame is Tagalog
eng English, including Taglish whose frame is English
ceb Cebuano / Bisaya
hil Hiligaynon / Ilonggo
war Waray
ilo Ilocano
bcl Bikol (any variety: Naga, Legazpi, Rinconada, Albay, Sorsogon...)
pam Kapampangan
pag Pangasinan
cbk Chavacano (Zamboanga, Cavite, Ternate, Davao...)
krj Kinaray-a
akl Aklanon
sgd Surigaonon
tsg Tausug
oth_ph another Philippine language (write which in note)
spa Spanish
ind Indonesian
msa Malay
oth any other language (write which in note)
mixed two languages in clearly separate parts and no single frame language
not_language only names, numbers, codes, symbols, gibberish, or a list of proper nouns
unsure you really cannot tell (this is a fine answer - please use it instead of guessing)

Edge cases

  1. Taglish (code-switching). Pick the language that supplies the sentence frame - the grammar words (Tagalog: ang, ng, sa, na, ay, mga, ko, mo, niya, hindi, para sa...; English: the, is, to, of, and, for...). "Nag-meeting kami kanina about the project" -> tgl, code_switched = yes. "The mayor said na hindi sila aalis" -> eng if the English part carries the frame, otherwise tgl; use your judgement and mark confidence medium if it is half-half. Use mixed only if two separate chunks (e.g. a headline in English followed by a Tagalog sentence) have no single frame.
  2. Names, numbers, places, acronyms (Juan dela Cruz, PNP, P20,000, Kalibo) do not decide the language. Look at the rest of the text. If nothing else is left, use not_language.
  3. Quotations. If a Tagalog sentence quotes English words, or an English sentence quotes a Tagalog phrase, label the language of the whole text frame (the part outside the quotation marks), and set code_switched = yes. If the whole text is only the quote, label the quote.
  4. Very short strings (2-4 words, headlines): label what a native reader would say it is. If it is equally valid as English and Tagalog (e.g. "Ok na", "Bahay"), use your best guess and low.
  5. Close languages. Cebuano vs Hiligaynon vs Waray, Tagalog vs Bikol, Indonesian vs Malay, Spanish vs Chavacano: answer from your own knowledge, and use low if the text is too short or too standard to tell. Chavacano uses Spanish words with Philippine grammar (ta, ya, di, mi, na, e.g. "Ya mira yo con el mujer"); plain Spanish with Spanish grammar is spa.
  6. Old/formal/religious spelling, accents (e.g. Cebuano with stress accents) - label the language normally.
  7. Mixed junk (OCR errors, broken words): label the language if you can read it, otherwise unsure.
  8. Do not search the internet or use a translator. Do not change the text column. Do not sort or delete rows.

What happens with your labels

They become a private evaluation set and training candidates for the OurLID language identifier. They are only used to measure and improve the model; the excerpts themselves are not published.


Gabay sa pag-label ng mga katutubong tagapagsalita

Makakakita ka ng humigit-kumulang 675 maiikling teksto (hindi hihigit sa 300 character bawat isa). Para sa bawat isa, sabihin kung anong wika ang pagkakasulat nito. Gusto namin ang tapat mong pagkaintindi bilang katutubong tagapagsalita. May mga 50 hilera na sadyang inulit para masukat ang pagkakapare-pareho - i-label ang bawat hilera na parang ngayon mo lang nakita.

Oras: mga 1-1.5 minuto bawat 10 hilera, kaya mga 1.5 oras lahat. Puwedeng huminto at ipagpatuloy mamaya; iwanang blangko ang hindi pa nagagawa.

Ano ang pupunan (mga column E-H; ang A-C ay ang aytem)

column ilalagay
annotator_label isang code mula sa listahan sa itaas (may dropdown sa Excel)
code_switched yes kung may ikalawang wika rin sa teksto (hal. Taglish), kung hindi no
confidence high (sigurado), medium, o low (hula lang)
note opsyonal; kailangan para sa oth_ph at oth (isulat ang pangalan ng wika)

Mga espesyal na kaso

  1. Taglish. Piliin ang wikang nagbibigay ng balangkas ng pangungusap - ang mga salitang pambalarila (Tagalog: ang, ng, sa, na, ay, mga, ko, mo, niya, hindi, para sa...; English: the, is, to, of, and, for...). "Nag-meeting kami kanina about the project" -> tgl, code_switched = yes. Kung kalahati-kalahati, pumili ayon sa pakiramdam mo at lagyan ng medium. Gamitin ang mixed lang kung magkahiwalay ang dalawang bahagi (hal. English na headline na sinundan ng Tagalog na pangungusap) at walang iisang balangkas.
  2. Mga pangalan, numero, lugar, akronim (Juan dela Cruz, PNP, P20,000, Kalibo) ay hindi nagpapasya ng wika. Tingnan ang natitirang teksto. Kung wala nang natira, not_language.
  3. Mga sipi (quotes). Kung ang pangungusap na Tagalog ay may sipi sa English (o kabaligtaran), i-label ang wika ng buong balangkas (ang labas ng panipi), at code_switched = yes. Kung sipi lang ang buong teksto, i-label ang sipi.
  4. Napakaikling teksto (2-4 salita, headline): sagutin kung ano ang sasabihin ng isang katutubong bumabasa. Kung pantay na puwedeng English o Tagalog, magbigay ng pinakamahusay na hula at lagyan ng low.
  5. Magkakalapit na wika. Cebuano / Hiligaynon / Waray, Tagalog / Bikol, Indonesian / Malay, Spanish / Chavacano: sumagot ayon sa sarili mong kaalaman; low kung masyadong maikli o masyadong karaniwan. Ang Chavacano ay gumagamit ng salitang Espanyol na may balarilang Pilipino (ta, ya, di, mi, na, hal. "Ya mira yo con el mujer"); ang purong Espanyol na may balarilang Espanyol ay spa.
  6. Lumang/pormal/panrelihiyong baybay, may accent (hal. Cebuano na may tuldik) - i-label ang wika gaya ng dati.
  7. Sirang teksto (OCR, putol na salita): i-label kung nababasa mo, kung hindi unsure.
  8. Huwag mag-search sa internet o gumamit ng translator. Huwag baguhin ang text column. Huwag mag-sort o magbura ng hilera.

Ang unsure ay katanggap-tanggap na sagot - mas mabuti ito kaysa manghula.

Ano ang mangyayari sa mga label mo

Gagamitin ang mga ito bilang pribadong evaluation set at training candidates ng OurLID na language identifier. Ginagamit lamang ang mga ito para sukatin at pahusayin ang model; hindi ilalathala ang mga sipi.