OpenThai · iApp Technology

OpenThai2.0-LATTE-WordSeg-Lite

Thai word segmentation on a CPU, from the OpenThai 2.0 model family. A small model built on LATTE (Lattice ATTentive Encoding) gives three segmentation units at once.

  • Three units in one pass: long keeps compounds whole, short gives words (the default), minimal gives morphemes.
  • Several readings: for an ambiguous phrase, the n most probable segmentations, best first: ตากลม → ตา | กลม, ตากลม, ตาก | ลม.
  • Small and fast: 1.17M parameters (a 3 MB model and a 0.8 MB dictionary), about 100,000 characters per second on one CPU thread with lines batched (about 60,000 one line at a time). It needs only ONNX Runtime and NumPy.
  • Commercial use: trained only on data whose licences allow commercial use. Model and dictionary: CC BY-SA 4.0. Code: Apache-2.0.

How it is made and evaluated: see the paper. Test sets, scorer and the outputs of every system: OpenThai2.0-WordSeg-Benchmark.

Units

Unit เจ้าหน้าที่การศึกษาเปิดเผยว่า
long เจ้าหน้าที่ | การศึกษา | เปิดเผย | ว่า
short (default) เจ้า | หน้าที่ | การ | ศึกษา | เปิด | เผย | ว่า
minimal เจ้า | หน้า | ที่ | การ | ศึกษา | เปิด | เผย | ว่า

Quick start

pip install https://huggingface.co/iapp/OpenThai2.0-LATTE-WordSeg-Lite/resolve/main/openthai2_0_latte_wordseg_lite-0.1.2-py3-none-any.whl
from openthai_latte_wordseg_lite import tokenize

tokenize("เจ้าหน้าที่การศึกษาเปิดเผยว่า")                # ['เจ้า', 'หน้าที่', 'การ', 'ศึกษา', 'เปิด', 'เผย', 'ว่า']
tokenize("เจ้าหน้าที่การศึกษาเปิดเผยว่า", unit="long")   # ['เจ้าหน้าที่', 'การศึกษา', 'เปิดเผย', 'ว่า']

The wheel holds the model, runs offline and needs only NumPy and ONNX Runtime (Python 3.9 or later).

Options

To Use
choose the unit unit="long", "short" (the default) or "minimal"
see other readings tokenize_nbest(text, n=3): the n most probable segmentations, each with its probability
segment many lines faster tokenize_batch(texts): many lines in one call
add your own words Segmenter(user_words=[...]): names or terms the dictionary lacks (below)
get the same output alone or in a batch Segmenter(batch_chars=0): one line at a time
use the command line openthai-latte-wordseg-lite --unit long "เจ้าหน้าที่การศึกษา"

Your own words are hints that the model reads, not cuts it must make. They work best with long:

from openthai_latte_wordseg_lite import Segmenter

seg = Segmenter(unit="long", user_words=["ไอโอที", "ดิจิทัลทวิน"])
seg.tokenize("โรงงานใช้ระบบไอโอทีกับดิจิทัลทวิน")
# ['โรงงาน', 'ใช้', 'ระบบ', 'ไอโอที', 'กับ', 'ดิจิทัลทวิน']   (without them: ไอโอ | ที, ดิจิทัล | ทวิน)

In iApp's package

The model is also in iapp-ai, with token offsets and Thai text utilities (guide):

pip install "iapp-ai[nlp]"
from iapp_ai.nlp import word_tokenize

word_tokenize("แม่น้ำเจ้าพระยาไหลผ่านกรุงเทพ")           # ['แม่น้ำ', 'เจ้าพระยา', 'ไหล', 'ผ่าน', 'กรุงเทพ']
word_tokenize(text, unit="long", extra_words=[...])   # your own words, as user_words above

By default it rejoins numbers such as 4.1 and reads Thai digits as Arabic digits, so its output differs from the wheel on some lines (up to 3% of the lines of a test set, with an average word F1 of 85.7).

Results

Average word F1 and boundary F1 over seven Thai test sets, and speed on one CPU thread:

System Word F1 Boundary F1 Characters/s
LATTE-WordSeg-Lite 85.6 93.3 60K
SEFR CUT 84.5 92.6 1.9K
DeepCut 84.5 92.6 2.6K
AttaCut-SC 84.0 92.4 78K
khammoon 81.8 91.3 43K
PyThaiNLP newmm 70.9 85.7 937K
  • LATTE-WordSeg-Lite with its default unit, short (long: 83.1 and 92.0, minimal: 82.1 and 91.5).
  • Speed with one line per call, as DeepCut, AttaCut-SC and newmm are called. With lines batched it runs at 105K. khammoon and SEFR CUT take all lines in one call.
  • khammoon segments morphemes, not words, so compare it with minimal.
  • Boundary F1 scores each cut between two characters. Every system and test set: the benchmark and the paper.

Training data

The model learnt from LATTE models trained on VISTEC-TP-TH-2021 (long), UD Thai-TUD and Wisesight-1000 (short), and the CC BY-SA 4.0 part of the khammoon data (minimal). No BEST, ORCHID or other non-commercial data were used. Sources, licences and attributions: NOTICE.

Limitations

  • Thai corpora disagree on what a word is. Scores are lower on BEST and ORCHID, whose conventions none of the three units follows, and long keeps the compounds of its training corpus, not every dictionary compound.
  • An ambiguous string can be cut wrongly, above all a short one without context (แปลกว่า → แปล | ก | ว่า). tokenize_nbest lists the other readings.
  • Thai only. A run of ASCII letters or of digits (Thai digits too) is never split, but text in other scripts, emoji made of several characters and some URLs or e-mail addresses can be split.
  • Because of int8 quantisation, a line can be segmented slightly differently alone than in a batch: up to 2.9% of the lines of a test set, with no score moving by more than 0.05. Segmenter(batch_chars=0) processes one line at a time.

Files

Besides the wheel, this repository holds the model files on their own: model.int8.onnx (the network), lexicon.txt.gz (the dictionary), charclass.npz and manifest.json, which lists the three units and the SHA-256 of every file. Licences: LICENSE (model and dictionary), LICENSES/ and NOTICE.

Citation

@misc{chayintr2026lattewordseglite,
  title         = {{LATTE-WordSeg-Lite}: Multi-Granularity {Thai} Word Segmentation on a {CPU}, with a Reproducible Benchmark},
  author        = {Chay-intr, Thodsaporn and Viriyayudhakorn, Kobkrit and Theeramunkong, Thanaruk},
  year          = {2026},
  eprint        = {2611.21300},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2611.21300}
}

@article{chayintr2023latte,
  title   = {{LATTE}: Lattice {ATT}entive Encoding for Character-based Word Segmentation},
  author  = {Chay-intr, Thodsaporn and Kamigaito, Hidetaka and Funakoshi, Kotaro and Okumura, Manabu},
  journal = {Journal of Natural Language Processing},
  volume  = {30},
  number  = {2},
  pages   = {456--488},
  year    = {2023}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train iapp/OpenThai2.0-LATTE-WordSeg-Lite

Collection including iapp/OpenThai2.0-LATTE-WordSeg-Lite

Paper for iapp/OpenThai2.0-LATTE-WordSeg-Lite