
OpenThai2.0-LATTE-WordSeg-Lite
Thai word segmentation on a CPU, from the OpenThai 2.0 model family. A small model built on LATTE (Lattice ATTentive Encoding) gives three segmentation units at once.
- Three units in one pass:
longkeeps compounds whole,shortgives words (the default),minimalgives morphemes. - Several readings: for an ambiguous phrase, the n most probable segmentations, best first: ตากลม → ตา | กลม, ตากลม, ตาก | ลม.
- Small and fast: 1.17M parameters (a 3 MB model and a 0.8 MB dictionary), about 100,000 characters per second on one CPU thread with lines batched (about 60,000 one line at a time). It needs only ONNX Runtime and NumPy.
- Commercial use: trained only on data whose licences allow commercial use. Model and dictionary: CC BY-SA 4.0. Code: Apache-2.0.
How it is made and evaluated: see the paper. Test sets, scorer and the outputs of every system: OpenThai2.0-WordSeg-Benchmark.
Units
| Unit | เจ้าหน้าที่การศึกษาเปิดเผยว่า |
|---|---|
long |
เจ้าหน้าที่ | การศึกษา | เปิดเผย | ว่า |
short (default) |
เจ้า | หน้าที่ | การ | ศึกษา | เปิด | เผย | ว่า |
minimal |
เจ้า | หน้า | ที่ | การ | ศึกษา | เปิด | เผย | ว่า |
Quick start
pip install https://huggingface.co/iapp/OpenThai2.0-LATTE-WordSeg-Lite/resolve/main/openthai2_0_latte_wordseg_lite-0.1.2-py3-none-any.whl
from openthai_latte_wordseg_lite import tokenize
tokenize("เจ้าหน้าที่การศึกษาเปิดเผยว่า") # ['เจ้า', 'หน้าที่', 'การ', 'ศึกษา', 'เปิด', 'เผย', 'ว่า']
tokenize("เจ้าหน้าที่การศึกษาเปิดเผยว่า", unit="long") # ['เจ้าหน้าที่', 'การศึกษา', 'เปิดเผย', 'ว่า']
The wheel holds the model, runs offline and needs only NumPy and ONNX Runtime (Python 3.9 or later).
Options
| To | Use |
|---|---|
| choose the unit | unit="long", "short" (the default) or "minimal" |
| see other readings | tokenize_nbest(text, n=3): the n most probable segmentations, each with its probability |
| segment many lines faster | tokenize_batch(texts): many lines in one call |
| add your own words | Segmenter(user_words=[...]): names or terms the dictionary lacks (below) |
| get the same output alone or in a batch | Segmenter(batch_chars=0): one line at a time |
| use the command line | openthai-latte-wordseg-lite --unit long "เจ้าหน้าที่การศึกษา" |
Your own words are hints that the model reads, not cuts it must make. They work best with long:
from openthai_latte_wordseg_lite import Segmenter
seg = Segmenter(unit="long", user_words=["ไอโอที", "ดิจิทัลทวิน"])
seg.tokenize("โรงงานใช้ระบบไอโอทีกับดิจิทัลทวิน")
# ['โรงงาน', 'ใช้', 'ระบบ', 'ไอโอที', 'กับ', 'ดิจิทัลทวิน'] (without them: ไอโอ | ที, ดิจิทัล | ทวิน)
In iApp's package
The model is also in iapp-ai, with token offsets and Thai text utilities (guide):
pip install "iapp-ai[nlp]"
from iapp_ai.nlp import word_tokenize
word_tokenize("แม่น้ำเจ้าพระยาไหลผ่านกรุงเทพ") # ['แม่น้ำ', 'เจ้าพระยา', 'ไหล', 'ผ่าน', 'กรุงเทพ']
word_tokenize(text, unit="long", extra_words=[...]) # your own words, as user_words above
By default it rejoins numbers such as 4.1 and reads Thai digits as Arabic digits, so its output differs from the wheel on some lines (up to 3% of the lines of a test set, with an average word F1 of 85.7).
Results
Average word F1 and boundary F1 over seven Thai test sets, and speed on one CPU thread:
| System | Word F1 | Boundary F1 | Characters/s |
|---|---|---|---|
| LATTE-WordSeg-Lite | 85.6 | 93.3 | 60K |
| SEFR CUT | 84.5 | 92.6 | 1.9K |
| DeepCut | 84.5 | 92.6 | 2.6K |
| AttaCut-SC | 84.0 | 92.4 | 78K |
| khammoon | 81.8 | 91.3 | 43K |
| PyThaiNLP newmm | 70.9 | 85.7 | 937K |
- LATTE-WordSeg-Lite with its default unit,
short(long: 83.1 and 92.0,minimal: 82.1 and 91.5). - Speed with one line per call, as DeepCut, AttaCut-SC and newmm are called. With lines batched it runs at 105K. khammoon and SEFR CUT take all lines in one call.
- khammoon segments morphemes, not words, so compare it with
minimal. - Boundary F1 scores each cut between two characters. Every system and test set: the benchmark and the paper.
Training data
The model learnt from LATTE models trained on VISTEC-TP-TH-2021 (long), UD Thai-TUD and Wisesight-1000
(short), and the CC BY-SA 4.0 part of the khammoon data (minimal). No BEST, ORCHID or other non-commercial
data were used. Sources, licences and attributions: NOTICE.
Limitations
- Thai corpora disagree on what a word is. Scores are lower on BEST and ORCHID, whose conventions none of the
three units follows, and
longkeeps the compounds of its training corpus, not every dictionary compound. - An ambiguous string can be cut wrongly, above all a short one without context (แปลกว่า → แปล | ก | ว่า).
tokenize_nbestlists the other readings. - Thai only. A run of ASCII letters or of digits (Thai digits too) is never split, but text in other scripts, emoji made of several characters and some URLs or e-mail addresses can be split.
- Because of int8 quantisation, a line can be segmented slightly differently alone than in a batch: up to 2.9%
of the lines of a test set, with no score moving by more than 0.05.
Segmenter(batch_chars=0)processes one line at a time.
Files
Besides the wheel, this repository holds the model files on their own: model.int8.onnx (the network),
lexicon.txt.gz (the dictionary), charclass.npz and manifest.json, which lists the three units and the SHA-256
of every file. Licences: LICENSE (model and dictionary), LICENSES/ and NOTICE.
Citation
@misc{chayintr2026lattewordseglite,
title = {{LATTE-WordSeg-Lite}: Multi-Granularity {Thai} Word Segmentation on a {CPU}, with a Reproducible Benchmark},
author = {Chay-intr, Thodsaporn and Viriyayudhakorn, Kobkrit and Theeramunkong, Thanaruk},
year = {2026},
eprint = {2611.21300},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2611.21300}
}
@article{chayintr2023latte,
title = {{LATTE}: Lattice {ATT}entive Encoding for Character-based Word Segmentation},
author = {Chay-intr, Thodsaporn and Kamigaito, Hidetaka and Funakoshi, Kotaro and Okumura, Manabu},
journal = {Journal of Natural Language Processing},
volume = {30},
number = {2},
pages = {456--488},
year = {2023}
}