docs: release model card, benchmark figures, and architecture metadata
Browse files- .gitattributes +2 -0
- README.md +141 -3
- accuracy_vs_latency_frontier.png +3 -0
- live_t4_benchmark_comparison.png +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
accuracy_vs_latency_frontier.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
live_t4_benchmark_comparison.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -1,3 +1,141 @@
|
|
| 1 |
-
---
|
| 2 |
-
|
| 3 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- th
|
| 4 |
+
license: apache-2.0
|
| 5 |
+
tags:
|
| 6 |
+
- token-classification
|
| 7 |
+
- ner
|
| 8 |
+
- temporal-extraction
|
| 9 |
+
- wangchanberta
|
| 10 |
+
- roberta
|
| 11 |
+
- thai-nlp
|
| 12 |
+
base_model: airesearch/wangchanberta-base-att-spm-uncased
|
| 13 |
+
datasets:
|
| 14 |
+
- DuckerMaster/Thai-Natural-DateTime-Dataset
|
| 15 |
+
pipeline_tag: token-classification
|
| 16 |
+
metrics:
|
| 17 |
+
- f1
|
| 18 |
+
- precision
|
| 19 |
+
- recall
|
| 20 |
+
- accuracy
|
| 21 |
+
model-index:
|
| 22 |
+
- name: DuckerTime-Normalize-Thai
|
| 23 |
+
results:
|
| 24 |
+
- task:
|
| 25 |
+
type: token-classification
|
| 26 |
+
name: Named Entity Recognition
|
| 27 |
+
dataset:
|
| 28 |
+
name: Thai Natural DateTime Dataset (v1.1)
|
| 29 |
+
type: DuckerMaster/Thai-Natural-DateTime-Dataset
|
| 30 |
+
metrics:
|
| 31 |
+
- name: In-Distribution F1
|
| 32 |
+
type: f1
|
| 33 |
+
value: 1.0000
|
| 34 |
+
- name: Out-of-Distribution F1
|
| 35 |
+
type: f1
|
| 36 |
+
value: 0.9692
|
| 37 |
+
- name: Out-of-Distribution Recall
|
| 38 |
+
type: recall
|
| 39 |
+
value: 0.9957
|
| 40 |
+
- name: Out-of-Distribution Precision
|
| 41 |
+
type: precision
|
| 42 |
+
value: 0.9441
|
| 43 |
+
---
|
| 44 |
+
|
| 45 |
+
# DuckerTime-Normalize-Thai (WangchanBERTa Temporal Entity Slicer)
|
| 46 |
+
|
| 47 |
+
Fine-tuned `wangchanberta-base-att-spm-uncased` for token-level temporal boundary extraction (`B-TIME`, `I-TIME`, `O`) across colloquial Thai sentences, conversational chat slang, and compound date-time structures.
|
| 48 |
+
|
| 49 |
+
Paired Dataset: [DuckerMaster/Thai-Natural-DateTime-Dataset](https://huggingface.co/datasets/DuckerMaster/Thai-Natural-DateTime-Dataset)
|
| 50 |
+
|
| 51 |
+
Designed as the neural frontend for the End-to-End Hybrid Architecture:
|
| 52 |
+
1. **WangchanBERTa Frontend**: Extracts precise character and token spans from raw unstructured text.
|
| 53 |
+
2. **Deterministic Rule Engine Backend**: Parses the extracted temporal span into valid ISO-8601 points and ranges in sub-millisecond execution.
|
| 54 |
+
|
| 55 |
+
## Model Performance & Empirical Benchmark
|
| 56 |
+
|
| 57 |
+

|
| 58 |
+
|
| 59 |
+
Evaluated against disjoint stratified benchmarks and tested side-by-side with rule baselines and instruction-tuned LLMs on NVIDIA T4 hardware.
|
| 60 |
+
|
| 61 |
+
### Benchmark Summary (Batch Size = 1, Streaming Latency)
|
| 62 |
+
|
| 63 |
+
| System / Model Architecture | Parameter Size | Hardware Execution | OOD Accuracy / F1 | Mean Latency (ms) | Production SLA (<50ms) |
|
| 64 |
+
| :--- | :---: | :---: | :---: | :---: | :---: |
|
| 65 |
+
| **dateparser** (Heuristic Multilingual) | 0 | CPU | 6.0% | 12.4 ms | Pass |
|
| 66 |
+
| **PyThaiNLP** (Rule Parser) | 0 | CPU | 18.7% | 1.8 ms | Pass |
|
| 67 |
+
| **Qwen2.5-3B-Instruct** (4-bit NF4) | 3B | GPU (T4) | 3.3% | 383.3 ms | Fail (Over 7x threshold) |
|
| 68 |
+
| **Typhoon 30B (Cloud REST API)** | 30B | Cloud Server | 47.3% | 2,753.9 ms | Fail (Over 55x threshold) |
|
| 69 |
+
| **Our Hybrid System (WangchanBERTa + Rule)** | **106M** | **GPU (T4)** | **96.9%** | **10.2 ms** | **Pass (Optimal Frontier)** |
|
| 70 |
+
|
| 71 |
+
### Architectural Insights
|
| 72 |
+
|
| 73 |
+
- **Why Pure LLMs Fail on Strict Temporal Extraction**: Generative LLMs are prone to conversational framing (e.g. outputting conversational preambles or rephrasing targets) rather than returning strict token spans. This results in high parsing failure rates (3.3% exact match) and high latency (383ms+ per request).
|
| 74 |
+
- **Why Hybrid Architecture Wins**: Token classification via WangchanBERTa isolates target temporal spans with 99.57% Recall on out-of-distribution text. Passing this span to a deterministic rule engine guarantees valid ISO-8601 timestamps without hallucination.
|
| 75 |
+
|
| 76 |
+
### Accuracy vs. Latency Pareto Frontier
|
| 77 |
+
|
| 78 |
+

|
| 79 |
+
|
| 80 |
+
## Model Specifications and Hyperparameters
|
| 81 |
+
|
| 82 |
+
- **Base Architecture**: RoBERTa / CamemBERT architecture pre-trained on Thai Wikipedia, news, and social text (`airesearch/wangchanberta-base-att-spm-uncased`).
|
| 83 |
+
- **Target Entities**:
|
| 84 |
+
- `O`: Outside temporal entity.
|
| 85 |
+
- `B-TIME`: Beginning of temporal expression.
|
| 86 |
+
- `I-TIME`: Continuation of temporal expression.
|
| 87 |
+
- **Training Epochs**: 4
|
| 88 |
+
- **Batch Size**: 16 (per device)
|
| 89 |
+
- **Optimizer**: AdamW (Learning Rate: 3e-5, Weight Decay: 0.01)
|
| 90 |
+
- **Warmup Steps**: 50
|
| 91 |
+
- **FP16 Mixed Precision**: Enabled
|
| 92 |
+
- **Inference Robustness**: Multi-token stitching with confidence thresholding (0.35) to handle spaced particles and chat modifiers.
|
| 93 |
+
|
| 94 |
+
## Intended Use and Pipeline Example
|
| 95 |
+
|
| 96 |
+
```python
|
| 97 |
+
import torch
|
| 98 |
+
from transformers import AutoTokenizer, AutoModelForTokenClassification
|
| 99 |
+
|
| 100 |
+
model_name = "DuckerMaster/DuckerTime-Normalize-Thai"
|
| 101 |
+
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
| 102 |
+
model = AutoModelForTokenClassification.from_pretrained(model_name)
|
| 103 |
+
|
| 104 |
+
text = "ขอเลื่อนเป็นวันพุธหน้าช่วงบ่ายแก่ๆ สักสี่โมงเย็นได้ไหมคะ"
|
| 105 |
+
inputs = tokenizer(text, return_tensors="pt")
|
| 106 |
+
|
| 107 |
+
with torch.no_grad():
|
| 108 |
+
logits = model(**inputs).logits
|
| 109 |
+
preds = torch.argmax(logits, dim=2)[0].tolist()
|
| 110 |
+
|
| 111 |
+
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
|
| 112 |
+
extracted_tokens = [tok for tok, pred in zip(tokens, preds) if pred != 0]
|
| 113 |
+
print("Extracted Temporal Tokens:", extracted_tokens)
|
| 114 |
+
```
|
| 115 |
+
|
| 116 |
+
## Limitations and Negative Sample Handling
|
| 117 |
+
|
| 118 |
+
- The model is fine-tuned to recognize temporal expressions in Thai. It actively suppresses non-temporal numeric entities (monetary sums, telephone sequences, tracking numbers) due to negative sampling in training data.
|
| 119 |
+
- For non-Thai text, fallback to multilingual parsers is recommended.
|
| 120 |
+
- Pair with `thai-temporal-normalizer` core package for full ISO-8601 normalization.
|
| 121 |
+
|
| 122 |
+
## Citation
|
| 123 |
+
|
| 124 |
+
If you use this model in your research or production systems, please cite:
|
| 125 |
+
|
| 126 |
+
```bibtex
|
| 127 |
+
@software{duckertime_thai_2026,
|
| 128 |
+
author = {DuckerMaster},
|
| 129 |
+
title = {DuckerTime-Normalize-Thai: Fine-Tuned WangchanBERTa for Colloquial Thai Temporal NLU},
|
| 130 |
+
year = {2026},
|
| 131 |
+
publisher = {HuggingFace},
|
| 132 |
+
howpublished = {\url{https://huggingface.co/DuckerMaster/DuckerTime-Normalize-Thai}}
|
| 133 |
+
}
|
| 134 |
+
```
|
| 135 |
+
|
| 136 |
+
## References
|
| 137 |
+
|
| 138 |
+
- **WangchanBERTa**: Lowphansirikul, L., et al. (2021). WangchanBERTa: Pretrained language models for Thai language. *arXiv preprint arXiv:2101.09635*.
|
| 139 |
+
- **ISO-TimeML**: Pustejovsky, J., et al. (2010). ISO-TimeML: An International Standard for Semantic Annotation. *ISO/TC 37/SC 4*.
|
| 140 |
+
- **SUTime**: Chang, A. X., & Manning, C. D. (2012). SUTime: A library for recognizing and normalizing time expressions. In *LREC 2012*.
|
| 141 |
+
- **HeidelTime**: Strötgen, J., & Gertz, M. (2010). HeidelTime: High quality rule-based extraction and normalization of temporal expressions. In *SemEval 2010*.
|
accuracy_vs_latency_frontier.png
ADDED
|
Git LFS Details
|
live_t4_benchmark_comparison.png
ADDED
|
Git LFS Details
|