DuckerMaster commited on
Commit
d640b95
·
verified ·
1 Parent(s): fde4e62

docs: release model card, benchmark figures, and architecture metadata

Browse files
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ accuracy_vs_latency_frontier.png filter=lfs diff=lfs merge=lfs -text
37
+ live_t4_benchmark_comparison.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,3 +1,141 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - th
4
+ license: apache-2.0
5
+ tags:
6
+ - token-classification
7
+ - ner
8
+ - temporal-extraction
9
+ - wangchanberta
10
+ - roberta
11
+ - thai-nlp
12
+ base_model: airesearch/wangchanberta-base-att-spm-uncased
13
+ datasets:
14
+ - DuckerMaster/Thai-Natural-DateTime-Dataset
15
+ pipeline_tag: token-classification
16
+ metrics:
17
+ - f1
18
+ - precision
19
+ - recall
20
+ - accuracy
21
+ model-index:
22
+ - name: DuckerTime-Normalize-Thai
23
+ results:
24
+ - task:
25
+ type: token-classification
26
+ name: Named Entity Recognition
27
+ dataset:
28
+ name: Thai Natural DateTime Dataset (v1.1)
29
+ type: DuckerMaster/Thai-Natural-DateTime-Dataset
30
+ metrics:
31
+ - name: In-Distribution F1
32
+ type: f1
33
+ value: 1.0000
34
+ - name: Out-of-Distribution F1
35
+ type: f1
36
+ value: 0.9692
37
+ - name: Out-of-Distribution Recall
38
+ type: recall
39
+ value: 0.9957
40
+ - name: Out-of-Distribution Precision
41
+ type: precision
42
+ value: 0.9441
43
+ ---
44
+
45
+ # DuckerTime-Normalize-Thai (WangchanBERTa Temporal Entity Slicer)
46
+
47
+ Fine-tuned `wangchanberta-base-att-spm-uncased` for token-level temporal boundary extraction (`B-TIME`, `I-TIME`, `O`) across colloquial Thai sentences, conversational chat slang, and compound date-time structures.
48
+
49
+ Paired Dataset: [DuckerMaster/Thai-Natural-DateTime-Dataset](https://huggingface.co/datasets/DuckerMaster/Thai-Natural-DateTime-Dataset)
50
+
51
+ Designed as the neural frontend for the End-to-End Hybrid Architecture:
52
+ 1. **WangchanBERTa Frontend**: Extracts precise character and token spans from raw unstructured text.
53
+ 2. **Deterministic Rule Engine Backend**: Parses the extracted temporal span into valid ISO-8601 points and ranges in sub-millisecond execution.
54
+
55
+ ## Model Performance & Empirical Benchmark
56
+
57
+ ![Empirical Benchmark Comparison](live_t4_benchmark_comparison.png)
58
+
59
+ Evaluated against disjoint stratified benchmarks and tested side-by-side with rule baselines and instruction-tuned LLMs on NVIDIA T4 hardware.
60
+
61
+ ### Benchmark Summary (Batch Size = 1, Streaming Latency)
62
+
63
+ | System / Model Architecture | Parameter Size | Hardware Execution | OOD Accuracy / F1 | Mean Latency (ms) | Production SLA (<50ms) |
64
+ | :--- | :---: | :---: | :---: | :---: | :---: |
65
+ | **dateparser** (Heuristic Multilingual) | 0 | CPU | 6.0% | 12.4 ms | Pass |
66
+ | **PyThaiNLP** (Rule Parser) | 0 | CPU | 18.7% | 1.8 ms | Pass |
67
+ | **Qwen2.5-3B-Instruct** (4-bit NF4) | 3B | GPU (T4) | 3.3% | 383.3 ms | Fail (Over 7x threshold) |
68
+ | **Typhoon 30B (Cloud REST API)** | 30B | Cloud Server | 47.3% | 2,753.9 ms | Fail (Over 55x threshold) |
69
+ | **Our Hybrid System (WangchanBERTa + Rule)** | **106M** | **GPU (T4)** | **96.9%** | **10.2 ms** | **Pass (Optimal Frontier)** |
70
+
71
+ ### Architectural Insights
72
+
73
+ - **Why Pure LLMs Fail on Strict Temporal Extraction**: Generative LLMs are prone to conversational framing (e.g. outputting conversational preambles or rephrasing targets) rather than returning strict token spans. This results in high parsing failure rates (3.3% exact match) and high latency (383ms+ per request).
74
+ - **Why Hybrid Architecture Wins**: Token classification via WangchanBERTa isolates target temporal spans with 99.57% Recall on out-of-distribution text. Passing this span to a deterministic rule engine guarantees valid ISO-8601 timestamps without hallucination.
75
+
76
+ ### Accuracy vs. Latency Pareto Frontier
77
+
78
+ ![Accuracy vs Latency Frontier](accuracy_vs_latency_frontier.png)
79
+
80
+ ## Model Specifications and Hyperparameters
81
+
82
+ - **Base Architecture**: RoBERTa / CamemBERT architecture pre-trained on Thai Wikipedia, news, and social text (`airesearch/wangchanberta-base-att-spm-uncased`).
83
+ - **Target Entities**:
84
+ - `O`: Outside temporal entity.
85
+ - `B-TIME`: Beginning of temporal expression.
86
+ - `I-TIME`: Continuation of temporal expression.
87
+ - **Training Epochs**: 4
88
+ - **Batch Size**: 16 (per device)
89
+ - **Optimizer**: AdamW (Learning Rate: 3e-5, Weight Decay: 0.01)
90
+ - **Warmup Steps**: 50
91
+ - **FP16 Mixed Precision**: Enabled
92
+ - **Inference Robustness**: Multi-token stitching with confidence thresholding (0.35) to handle spaced particles and chat modifiers.
93
+
94
+ ## Intended Use and Pipeline Example
95
+
96
+ ```python
97
+ import torch
98
+ from transformers import AutoTokenizer, AutoModelForTokenClassification
99
+
100
+ model_name = "DuckerMaster/DuckerTime-Normalize-Thai"
101
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
102
+ model = AutoModelForTokenClassification.from_pretrained(model_name)
103
+
104
+ text = "ขอเลื่อนเป็นวันพุธหน้าช่วงบ่ายแก่ๆ สักสี่โมงเย็นได้ไหมคะ"
105
+ inputs = tokenizer(text, return_tensors="pt")
106
+
107
+ with torch.no_grad():
108
+ logits = model(**inputs).logits
109
+ preds = torch.argmax(logits, dim=2)[0].tolist()
110
+
111
+ tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
112
+ extracted_tokens = [tok for tok, pred in zip(tokens, preds) if pred != 0]
113
+ print("Extracted Temporal Tokens:", extracted_tokens)
114
+ ```
115
+
116
+ ## Limitations and Negative Sample Handling
117
+
118
+ - The model is fine-tuned to recognize temporal expressions in Thai. It actively suppresses non-temporal numeric entities (monetary sums, telephone sequences, tracking numbers) due to negative sampling in training data.
119
+ - For non-Thai text, fallback to multilingual parsers is recommended.
120
+ - Pair with `thai-temporal-normalizer` core package for full ISO-8601 normalization.
121
+
122
+ ## Citation
123
+
124
+ If you use this model in your research or production systems, please cite:
125
+
126
+ ```bibtex
127
+ @software{duckertime_thai_2026,
128
+ author = {DuckerMaster},
129
+ title = {DuckerTime-Normalize-Thai: Fine-Tuned WangchanBERTa for Colloquial Thai Temporal NLU},
130
+ year = {2026},
131
+ publisher = {HuggingFace},
132
+ howpublished = {\url{https://huggingface.co/DuckerMaster/DuckerTime-Normalize-Thai}}
133
+ }
134
+ ```
135
+
136
+ ## References
137
+
138
+ - **WangchanBERTa**: Lowphansirikul, L., et al. (2021). WangchanBERTa: Pretrained language models for Thai language. *arXiv preprint arXiv:2101.09635*.
139
+ - **ISO-TimeML**: Pustejovsky, J., et al. (2010). ISO-TimeML: An International Standard for Semantic Annotation. *ISO/TC 37/SC 4*.
140
+ - **SUTime**: Chang, A. X., & Manning, C. D. (2012). SUTime: A library for recognizing and normalizing time expressions. In *LREC 2012*.
141
+ - **HeidelTime**: Strötgen, J., & Gertz, M. (2010). HeidelTime: High quality rule-based extraction and normalization of temporal expressions. In *SemEval 2010*.
accuracy_vs_latency_frontier.png ADDED

Git LFS Details

  • SHA256: cf967d237dfabbf01a3d7af541f882431f7b10f8c39cab346fa8b8f31c7e433e
  • Pointer size: 131 Bytes
  • Size of remote file: 273 kB
live_t4_benchmark_comparison.png ADDED

Git LFS Details

  • SHA256: e3bb072f9a20cc3857433d6a53f0891b6d0d787c489a01a5d7f6678998e9a8ee
  • Pointer size: 131 Bytes
  • Size of remote file: 246 kB