kobkrit commited on
Commit
eaaae4f
Β·
verified Β·
1 Parent(s): eae721f

model card: v0.2 changelog + known order-sensitivity issue

Browse files
Files changed (1) hide show
  1. README.md +18 -0
README.md CHANGED
@@ -182,6 +182,24 @@ Thai sets and on NLI/topic tasks, and route low-confidence English yes/no decisi
182
  | v0.2 | 2026-09-21 | +3,000 SFT steps from v0.1 with a 22k-record synthetic Thai social-sentiment set (4/3/5-class, yes/no, score schemes), re-calibrated | **63.2** | 51.5 |
183
  | v0.1 | 2026-09-20 | initial release: Thai CPT 4.47B tokens, 12k-step SFT, calibration | 61.9 | 38.7 |
184
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
185
  ## License & credits
186
 
187
  Apache-2.0. Built by iApp Technology / OpenThai on Qwen3.5-0.8B-Base (Apache-2.0). Inspired by TypeSafe AI's
 
182
  | v0.2 | 2026-09-21 | +3,000 SFT steps from v0.1 with a 22k-record synthetic Thai social-sentiment set (4/3/5-class, yes/no, score schemes), re-calibrated | **63.2** | 51.5 |
183
  | v0.1 | 2026-09-20 | initial release: Thai CPT 4.47B tokens, 12k-step SFT, calibration | 61.9 | 38.7 |
184
 
185
+ ## Changelog
186
+
187
+ **v0.2 β€” 2026-09-21**
188
+ - Continued fine-tuning for 3,000 steps from v0.1 with a new 22k-record synthetic Thai social-media sentiment set
189
+ (4-class with the "question" class, 3/5-class variants, yes/no flags, 5-level score), then re-calibrated.
190
+ - Public 13-subset macro 61.9 β†’ **63.2**; Wisesight 38.7 β†’ **51.5**; MASSIVE-en 75.7 β†’ 79.1; MASSIVE-th 86.4 β†’ 88.6;
191
+ MultiNLI 85.6 β†’ 87.3 (ECE 0.035 β†’ 0.016); Aegis2 58.0 β†’ 61.6; PubMedQA 53.6 β†’ 56.4.
192
+ - Regression: SIB-200 Thai 77.5 β†’ 74.0 (sentiment set was weighted 4Γ—; will be lowered next round). Unchanged: SQuAD2, SummEval-relevance.
193
+ - Known issue (measured on v0.2): answers can depend on the *order* of the options, mildly for ≀ 10 options
194
+ (5–12% of arg-max choices flip under a random permutation, mean probability shift 0.05–0.14) and strongly for very
195
+ large option sets (77-way banking77: 72% flip). A permutation-averaging inference mode is in progress; until then, for
196
+ large option sets query with several option orders and average the probabilities.
197
+ - Live API switched to v0.2 (same checkpoint as these weights).
198
+
199
+ **v0.1 β€” 2026-09-20**
200
+ - Initial release: Qwen3.5-0.8B text tower, 4.47B-token Thai CPT, 256-slot decision head, 12k-step SFT on 1.8M
201
+ public + 98k synthetic decision records, calibration stage. Public macro 61.9.
202
+
203
  ## License & credits
204
 
205
  Apache-2.0. Built by iApp Technology / OpenThai on Qwen3.5-0.8B-Base (Apache-2.0). Inspired by TypeSafe AI's