You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

Zo classifiers: four small models that label one line of text

Four small classifiers for Zo / Kuki-Chin language text. Each reads one line and gives it a label. They are tables of letter-runs (every run of 1 to 5 characters in a line votes for a label), not neural networks: a few MB each, trained in about a minute on a laptop CPU, read with numpy alone.

Folder What it says Labels Size
lang/ which language a line is in Paite pck, Tedim ctd, Zou zom, Simte smt, Vaiphei vap, Gangte gnb, Thadou tcz, Hmar hmr, Mizo lus, Hakha cnh, Falam cfm, Siyin csy, English eng, Hindi hin, Burmese mya, Meitei mni, other 7.9 MB
machine/ natural Zo text or machine-made Zo text natural, machine 0.7 MB
clean/ a real sentence, or junk to drop ok, noise, furniture, fragment, mixed 2.2 MB
type/ what kind of text bible, news, chat, song, dictionary 2.1 MB

Trained 19 November 2026.

How to use

Download the repository, then (Python 3.9+ and numpy):

python zo_classify.py lang mixed.txt                      label every line -> mixed.lang.tsv
python zo_classify.py lang mixed.txt --split sorted/      also one file per language + unsure.txt
python zo_classify.py clean book.txt --split cleaned/     cleaned/ok.txt = the real sentences
python zo_classify.py machine pairs.jsonl --field zo      a .jsonl file, text taken from one field

The .tsv has three columns: label, how sure (0 to 1), the line. With --split, a line the model is less than 50% sure of goes to unsure.txt with its guess (change with --min 0.7).

from zo_classify import load
model = load("lang")
model.predict(["A tung in Pathian in lei leh van a siam.", "Thank you very much."])
# [('pck', 0.66), ('eng', ...)]

Test results

Every figure is from lines kept out of training; each test also holds back whole websites and books the model never saw. No test line is in training (compared with capitals, digits and punctuation ignored).

Language: 92.6% of 21,338 lines right (the twelve Zo languages alone: 90.7%)

Bible lines 99.2%; other text from sources used in training 94.1%; sources never seen 88.8%.

Words in the line (Zo languages) 1-3 4-6 7-12 13-25 26+
Right 56.3% 71.7% 85.0% 96.0% 97.6%

When it is 90% sure or more (63% of lines) it is right 99.7% of the time.

Language Precision Recall Test lines
Paite pck 80.3% 86.3% 1,486
Tedim ctd 91.6% 91.1% 1,476
Zou zom 87.5% 89.3% 1,481
Simte smt 96.7% 77.2% 1,356
Vaiphei vap 93.0% 90.7% 1,470
Gangte gnb 93.8% 93.4% 1,191
Thadou tcz 93.9% 95.2% 1,494
Hmar hmr 93.2% 90.7% 1,484
Mizo lus 89.6% 94.6% 1,199
Hakha cnh 91.3% 96.5% 1,192
Falam cfm 96.4% 89.1% 1,225
Siyin csy 95.3% 96.1% 1,154
English eng 91.4% 99.1% 1,500
Hindi hin 100% 100% 1,191
Burmese mya 100% 100% 1,160
Meitei mni 100% 100% 600
other 88.0% 90.7% 679

Close cousins, lines of 7 words or more: Paite 93.2%, Tedim 95.1%, Zou 95.2%, Vaiphei 95.3%, Simte 81.3% right. Simte is mostly mistaken for Paite and Zou.

Limits:

  • Lines of one to three words, names and numbers cannot be told apart between close languages.
  • Simte outside Bible and school books is weak: 47% on the one Simte website held out.
  • Hindi, Burmese and Meitei are recognised in their own scripts only (Meitei in Bengali script). Written in Latin letters they are not recognised. Not tested against other languages in the same scripts (Bengali, Assamese, Nepali, Marathi).
  • other was trained on eleven small Kuki-Chin languages (Khumi, Bawm, Matu, Ngawn, Zyphe, Lamkang, Daai, Zotung, Biate, Kom, Aimol) and scan debris. A language it was never shown is mostly given the nearest known one: of 600 Mara lines, 202 went to other and 268 were called Mizo.

Machine-made detector: 86.0% of 10,485 lines right

Machine: precision 82.6%, recall 83.1%. Natural: 88.4%, 88.0%. Sources used in training 90.8%; sources never seen 79.1%. Trained on Paite, Tedim and Mizo only; on natural text of nine other languages about one line in ten is wrongly flagged.

It calls natural short everyday phrases machine-made ("Na dam maw?"), and it does not carry over to a kind of machine text it has not seen. Use it to rank files for reading, not to judge one sentence.

Cleaner: 85.3% of 5,231 lines right on the five labels

As keep or drop: of the lines it keeps, 88.3% are real sentences; it keeps 92.1% of real sentences and drops 89.8% of junk. The test labels were themselves made by rule, so this measures agreement with those rules. It keeps headings, titles and number lists, drops about a third of chat messages, and does not detect two Zo languages mixed in one line.

Text type: 84.9% of 4,919 lines right

Sources used in training 94.9%; sources never seen 74.8%. Weak on Hakha song lyrics and on long answers written by a chat model. There is no "none of these" label.

Files

Each model folder holds classifier.json (labels, settings, the test results in full) and classifier.npz (the table, float16, and one bias per label). zo_classify.py is the reader and command-line tool. The labelled training lines are not included.

Training data and rights

Trained on a private research collection of Zo-language text: Bible translations, news sites, blogs, school books and community word lists, plus English, Hindi, Burmese and Meitei text for the four non-Zo classes. News, blogs and scripture remain their publishers' copyright, and some of the school books are licensed for educational and research use only, non-commercial. Do not pass the files on.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support