OpenThai-NER

Hugging Face Model Dataset License: CC BY 3.0 Python

OpenThai-NER is a production-grade Named Entity Recognition (NER) library and model for the Thai language. It builds upon Pavarissy/phayathaibert-thainer and fine-tunes on the multi-domain OpenThai-NER-Corpus with subword span reconstruction, numerical stability fixes, an optional linear-chain CRF decoding layer, and INT8 ONNX export for low-latency CPU deployment.


Key Features

  • Numerical Stability: Eliminates training instability (eval_loss: NaN) by replacing mixed-precision fp16 with bf16/fp32, applying gradient clipping (max_grad_norm=1.0), and strictly masking padding/special tokens (-100) in DataCollatorForTokenClassification.
  • Dataset Quality: Normalized 8,601 valid samples across 407 domains. Resolved syntax typos (DTAE โ†’ DATE) and consolidated synonym tags (LOC โ†’ LOCATION, ORG โ†’ ORGANIZATION, PER โ†’ PERSON).
  • Subword Span Reconstruction: Automatically aggregates SentencePiece subword fragments into complete Thai word spans with exact character start and end offsets.
  • Decoding Architecture: Implements a pure PyTorch linear-chain CRF layer with Viterbi decoding and BIO transition constraints to prevent structural tag violations.
  • Class Imbalance Mitigation: Implements Focal Loss with smoothed inverse class frequency weights to improve recall on rare entity categories.
  • Hardware Acceleration: Provides dynamic INT8 quantization via ONNX Runtime, reducing latency by 3.8ร— on CPU environments.

Benchmark & Evaluation

Evaluated on data/test.jsonl (1,092 test sequences across 407 domains) using seqeval strict span-level matching:

OpenThai-NER Benchmark Leaderboard

Metric Summary

Metric OpenThai-NER (Final) WangchanBERTa Base PyThaiNLP ThaiNER-v2 PhayaThaiBERT Baseline
Strict Span F1 79.28% 78.10% 76.40% 71.20%
Precision 78.87% 78.20% 75.90% 70.80%
Recall 79.70% 78.00% 76.90% 71.60%
Token Accuracy 90.76% 89.90% 88.50% 85.20%
Validation Loss 0.3697 0.3850 N/A NaN (Unstable)
CPU Latency (INT8) 11.2 ms/seq 18.5 ms/seq 15.1 ms/seq 42.6 ms/seq

Directory Structure

OpenThai/
โ”œโ”€โ”€ openthai_ner/                 # Core Python package
โ”‚   โ”œโ”€โ”€ __init__.py               # Package entrypoint
โ”‚   โ”œโ”€โ”€ pipeline.py               # Inference pipeline & span reconstruction
โ”‚   โ”œโ”€โ”€ crf.py                    # Linear-chain CRF with Viterbi decoding
โ”‚   โ”œโ”€โ”€ losses.py                 # Focal Loss & class weighting
โ”‚   โ”œโ”€โ”€ model_crf.py              # Combined Transformer + CRF model class
โ”‚   โ””โ”€โ”€ utils.py                  # Offset alignment & HTML rendering
โ”œโ”€โ”€ scripts/
โ”‚   โ”œโ”€โ”€ clean_dataset.py          # Dataset cleaner & stratified split
โ”‚   โ”œโ”€โ”€ export_onnx.py            # ONNX export & INT8 quantization
โ”‚   โ”œโ”€โ”€ evaluate_benchmark.py     # Evaluation script (seqeval)
โ”‚   โ”œโ”€โ”€ benchmark_sota.py         # Multi-model comparative benchmark
โ”‚   โ””โ”€โ”€ build_package.py          # Packaging script for PyPI release
โ”œโ”€โ”€ notebooks/
โ”‚   โ””โ”€โ”€ OpenThai_NER_FineTuning_Final.ipynb  # Google Colab GPU training notebook
โ”œโ”€โ”€ space_deploy/                 # Standalone Hugging Face Space application
โ”‚   โ”œโ”€โ”€ app.py
โ”‚   โ”œโ”€โ”€ README.md
โ”‚   โ””โ”€โ”€ requirements.txt
โ”œโ”€โ”€ data/                         # Processed data splits
โ”‚   โ”œโ”€โ”€ train.jsonl               # 6,792 training sequences
โ”‚   โ”œโ”€โ”€ val.jsonl                 # 717 validation sequences
โ”‚   โ”œโ”€โ”€ test.jsonl                # 1,092 test sequences
โ”‚   โ””โ”€โ”€ label_map.json            # Canonical 128-tag label mapping
โ”œโ”€โ”€ train_ner.py                  # Self-contained training script
โ”œโ”€โ”€ app.py                        # Local Gradio web demo
โ”œโ”€โ”€ pyproject.toml                # Build configuration
โ””โ”€โ”€ README.md

Installation

From Source

git clone https://github.com/JonusNattapong/OpenThai.git
cd OpenThai
pip install -r requirements.txt

Install as Package

pip install .
# Or directly via Git:
pip install git+https://github.com/JonusNattapong/OpenThai.git

Quickstart

Basic Inference

from openthai_ner import OpenThaiNER

ner = OpenThaiNER("JonusNattapong/OpenThai-NER")

text = "เธ™เธฒเธขเธชเธกเธŠเธฒเธข เน€เธ‚เน‡เธกเธเธฅเธฑเธ” เน€เธ”เธดเธ™เธ—เธฒเธ‡เน„เธ›เธ›เธฃเธฐเธŠเธธเธกเธ—เธตเนˆเธเธฃเธฐเธ—เธฃเธงเธ‡เธเธฒเธฃเธ„เธฅเธฑเธ‡ เธ–เธ™เธ™เธžเธฃเธฐเธฃเธฒเธก 6 เนƒเธ™เธงเธฑเธ™เธ—เธตเนˆ 15 เธกเธเธฃเธฒเธ„เธก"
entities = ner.predict(text, threshold=0.5)

for ent in entities:
    print(f"[{ent['entity']}] '{ent['word']}' (Span: {ent['start']}:{ent['end']}, Score: {ent['score']:.4f})")

Output:

[PERSON] 'เธ™เธฒเธขเธชเธกเธŠเธฒเธข เน€เธ‚เน‡เธกเธเธฅเธฑเธ”' (Span: 0:17, Score: 0.9812)
[ORGANIZATION] 'เธเธฃเธฐเธ—เธฃเธงเธ‡เธเธฒเธฃเธ„เธฅเธฑเธ‡' (Span: 36:50, Score: 0.9924)
[LOCATION] 'เธ–เธ™เธ™เธžเธฃเธฐเธฃเธฒเธก 6' (Span: 51:62, Score: 0.9540)
[DATE] 'เธงเธฑเธ™เธ—เธตเนˆ 15 เธกเธเธฃเธฒเธ„เธก' (Span: 66:82, Score: 0.9715)

HTML Rendering (Notebooks & Web)

html_output = ner.render_html(text)
# In Jupyter Notebook:
# from IPython.display import HTML; display(HTML(html_output))

Fast Inference with ONNX Runtime

from openthai_ner import OpenThaiNER

ner_onnx = OpenThaiNER(
    "JonusNattapong/OpenThai-NER",
    onnx_path="models/onnx/openthai_ner_quantized.onnx"
)
results = ner_onnx.predict("เธ˜เธ™เธฒเธ„เธฒเธฃเนเธซเนˆเธ‡เธ›เธฃเธฐเน€เธ—เธจเน„เธ—เธข เธ›เธฃเธฐเธเธฒเธจเธ›เธฃเธฑเธšเธฅเธ”เธญเธฑเธ•เธฃเธฒเธ”เธญเธเน€เธšเธตเน‰เธข")

Training

The training script is self-contained and handles dataset downloading, subword alignment, and metric logging automatically.

Run on Local GPU or Colab

python train_ner.py \
  --model_name Pavarissy/phayathaibert-thainer \
  --data_dir data \
  --output_dir models/openthai-ner-final \
  --epochs 3 \
  --batch_size 16 \
  --learning_rate 2e-5

Advanced Options (CRF & Focal Loss)

# Train with Linear-Chain CRF Layer
python train_ner.py --use_crf --output_dir models/openthai-ner-crf

# Train with Focal Loss for class imbalance
python train_ner.py --loss_type focal --focal_gamma 2.0

Google Colab Execution

Run directly via the Google Colab CLI:

colab run --gpu T4 train_ner.py

Or open notebooks/OpenThai_NER_FineTuning_Final.ipynb in Google Colab.


Model Export & Optimization

ONNX Export & INT8 Quantization

python scripts/export_onnx.py \
  --model JonusNattapong/OpenThai-NER \
  --output_dir models/onnx

Multi-Model Benchmark

python scripts/benchmark_sota.py --test_file data/test.jsonl --samples 100

Web Demo

Run the interactive Gradio demo locally:

python app.py

To deploy directly to Hugging Face Spaces, push the contents of space_deploy/ to your Space repository.


Supported Entities

The canonical schema covers 128 BIO labels across the following core categories:

  • Agents: PERSON, ORGANIZATION
  • Locations & Facilities: LOCATION, FACILITY
  • Temporal & Quantities: DATE, TIME, MONEY, PERCENT
  • Identifiers & Contact: ID, ACCOUNT, PHONE, EMAIL, URL
  • Specialized Domains: LAW, PRODUCT, DISEASE, TECHNOLOGY

Citation

@software{openthai_ner2026,
  author = {Nattapong Tapachoom},
  title = {OpenThai-NER: Production-Ready Thai Named Entity Recognition},
  url = {https://github.com/JonusNattapong/OpenThai},
  version = {0.1.0},
  year = {2026}
}

License

This project is released under the Creative Commons Attribution 3.0 Unported (CC BY 3.0) license.

Downloads last month
22
Safetensors
Model size
0.3B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for JonusNattapong/OpenThai-NER

Finetuned
(1)
this model

Dataset used to train JonusNattapong/OpenThai-NER

Evaluation results