kobkrit commited on
Commit
4f5f205
·
verified ·
1 Parent(s): 8367d31

OpenThai-SystemOne release

Browse files
Files changed (4) hide show
  1. README.md +110 -32
  2. config.json +6 -6
  3. model.safetensors +2 -2
  4. modeling.py +79 -31
README.md CHANGED
@@ -2,23 +2,26 @@
2
  license: apache-2.0
3
  language: [th, en]
4
  pipeline_tag: image-text-to-text
5
- tags: [system-one, decision-model, gui-grounding, computer-use, thai, qwen3.5, vision]
6
  base_model: Qwen/Qwen3.5-0.8B-Base
7
  ---
8
 
9
- # OpenThai-SystemOne-Vision (v1.0)
10
 
11
  The vision variant of [OpenThai-SystemOne](https://huggingface.co/iapp/OpenThai-SystemOne): the same open Thai + English
12
  System One decision model, now with **images in the state** and a fourth question type, **`point`**, that returns where
13
  on a screenshot something is. Screenshot in, click out, in one forward pass, no text generation.
14
 
 
 
 
15
  | | OpenThai-SystemOne (text) | OpenThai-SystemOne-Vision |
16
  |---|---|---|
17
  | parameters | 0.75B | 0.86B (ViT 100M + point head 9M) |
18
  | question types | choice / score / noul | choice / score / noul / **point** |
19
  | state | text or JSON | text or JSON + up to 4 images |
20
- | typical use | routing, moderation, RAG relevance, tool choice, agents with element tables | GUI agents from pixels, Thai screen reading, game frames, documents |
21
- | text-only behaviour | v0.3 | identical to v0.3 (same tower and head; the vision path is skipped when no image is sent) |
22
 
23
  ## API (superset of the TypeSafe `/v1/systemone` contract)
24
 
@@ -51,40 +54,111 @@ r = c.system_one("goal: open the login form", {"where": {"type": "point", "instr
51
  print(r.answers["where"].x, r.answers["where"].y, r.answers["where"].confidence)
52
  ```
53
 
 
 
 
 
 
 
 
 
 
 
 
 
54
  ## Evaluation
55
 
56
- Numbers are for the released checkpoint (calibrated round 3); single option order (no permutation averaging).
 
57
 
58
  ### GUI grounding (`point`: predicted click inside the target box)
59
- | benchmark | acc | notes |
60
- |---|---|---|
61
- | ScreenSpot-v2 (1,272) | **82.7** | desktop 90.2 text / 71.4 icon · mobile 95.9 / 71.1 · web 87.6 / 70.9 · GUI-Actor-2B 88.6 |
62
- | Thai GUI held-out domains, `point` (1,611) | **80.1** | teacher-written Thai/English instructions on unseen Thai sites |
63
- | Thai GUI set-of-marks `choice` (476) | 92.9 | pick the element index from the element table |
64
- | Thai GUI `operation` choice (476) | 99.8 | CLICK / TYPE_TEXT / SELECT |
65
- | Thai screen QA `noul` / `choice` (623 / 330) | 94.9 / 85.0 | is a dialog open, which page type, is there a form… |
66
- | "not on screen" abstention (179) | 45.8 | known weak spot: check `abstain` before clicking |
67
-
68
- ### Doom from pixels (`demo/doom_demo.py --mode pixels`, defend_the_center, skill 3, 20 episodes)
69
- | input | kills / episode | ms / decision (H100, idle) |
70
- |---|---|---|
71
- | two 320×240 frames only | 5.5 | 57 |
72
- | frames + symbolic state | 6.0 | 51 |
73
- | symbolic state only (text model behaviour) | 7.3 | 41 |
74
- | the privileged scripted expert it imitates | 6.8 | – |
75
-
76
- ### Text-only tasks (must not regress vs the text model v0.3)
77
- | suite | v0.3 | this model |
78
- |---|---|---|
79
- | public 13-subset bench, 10 accuracy subsets, macro | 82.7 | 82.8 |
80
- | Thai held-out sets (8), macro | 77.5 | 79.1 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
81
 
82
  ### Latency (batch 1, H100, end-to-end `SystemOneClient`)
83
  | request | tokens | ms |
84
  |---|---|---|
85
- | two 320×240 game frames + short state, 2 questions | 298 (160 visual) | 52 |
86
- | 1280×800 screenshot + goal, `point` + `choice` | 1,062 (1,000 visual) | 55 |
87
  | text only (3-question Thai ticket) | 166 | ~40 |
 
 
 
 
 
 
 
 
88
 
89
  ## How it was built
90
 
@@ -92,9 +166,13 @@ Numbers are for the released checkpoint (calibrated round 3); single option orde
92
  2. Vision tower + merger from Qwen3.5-0.8B-Base, re-aligned with an LM loss on 200k+ image→text pairs (Thai rendered
93
  OCR pages in 55 fonts, Thai web screenshots → element tables, Doom frames → object tables, public screen/document QA).
94
  3. Decision SFT on the v0.3 text mix plus Thai GUI grounding (Playwright crawl of 300+ Thai sites, teacher-written and
95
- blind-verified Thai/English instructions), public English grounding (Wave-UI, GroundUI), Doom rollouts (privileged
96
- expert, 1.5M frames) with `move` / `attack` / `in_danger` / `nearest_enemy` questions, and screen QA.
97
- 4. Calibration of the four per-type temperatures.
 
 
 
 
98
 
99
  ## License & credits
100
 
 
2
  license: apache-2.0
3
  language: [th, en]
4
  pipeline_tag: image-text-to-text
5
+ tags: [system-one, decision-model, gui-grounding, computer-use, voice-control, thai, qwen3.5, vision]
6
  base_model: Qwen/Qwen3.5-0.8B-Base
7
  ---
8
 
9
+ # OpenThai-SystemOne-Vision (v1.1)
10
 
11
  The vision variant of [OpenThai-SystemOne](https://huggingface.co/iapp/OpenThai-SystemOne): the same open Thai + English
12
  System One decision model, now with **images in the state** and a fourth question type, **`point`**, that returns where
13
  on a screenshot something is. Screenshot in, click out, in one forward pass, no text generation.
14
 
15
+ **v1.1 (2026-09-24)** adds Thai **voice-command understanding** for hands-free computer control (a spoken Thai command +
16
+ the app and window title in, the operation and its target out) and stronger desktop and Doom play. See [Versions](#versions).
17
+
18
  | | OpenThai-SystemOne (text) | OpenThai-SystemOne-Vision |
19
  |---|---|---|
20
  | parameters | 0.75B | 0.86B (ViT 100M + point head 9M) |
21
  | question types | choice / score / noul | choice / score / noul / **point** |
22
  | state | text or JSON | text or JSON + up to 4 images |
23
+ | typical use | routing, moderation, RAG relevance, tool choice, agents with element tables | GUI agents from pixels, Thai voice control, Thai screen reading, game frames, documents |
24
+ | text-only behaviour | v0.3 | same API; text scores equal or better than v0.3 (tables below) |
25
 
26
  ## API (superset of the TypeSafe `/v1/systemone` contract)
27
 
 
54
  print(r.answers["where"].x, r.answers["where"].y, r.answers["where"].confidence)
55
  ```
56
 
57
+ ### Thai voice control
58
+
59
+ The voice-command schema (21 operations such as `SEARCH`, `CLICK_NTH`, `OPEN_WEBSITE`, `QUIT_APP`, `PLAY_PAUSE`, `LIKE`,
60
+ `COMMENT`, `NONE` = not a command, plus `search_site` / `app` / `site` / `key` / `nth` slots) and a macOS controller
61
+ (on-device Thai speech recognition → this model → native mouse / keyboard events) will be published in the GitHub
62
+ repo under `demo/voice_control/`. One text-only call decides the operation; clicks add one screenshot call.
63
+
64
+ ```python
65
+ state = 'คำสั่งเสียงภาษาไทยจากผู้ใช้ที่ควบคุมคอมพิวเตอร์ด้วยเสียง: "อยากดูเนเน่ร้องเพลง"\nโปรแกรมที่ใช้อยู่: Google Chrome\nหน้าต่าง: Facebook'
66
+ # questions = intent_schema.intent_questions(apps) -> operation SEARCH, search_site YouTube
67
+ ```
68
+
69
  ## Evaluation
70
 
71
+ All numbers are for this checkpoint (v1.1) next to v1.0 and the text model v0.3, measured with the same scripts, single
72
+ option order (no permutation averaging) unless noted. Accuracy in %.
73
 
74
  ### GUI grounding (`point`: predicted click inside the target box)
75
+ | benchmark | v1.0 | **v1.1** | notes |
76
+ |---|---|---|---|
77
+ | ScreenSpot-v2, all (1,272) | 82.7 | **80.7** | reference: GUI-Actor-2B 88.6, Qwen3-VL-2B 86.7, UI-TARS-1.5-7B 89.0 (published) |
78
+ | · desktop text / icon (194 / 140) | 90.2 / 71.4 | 89.7 / **74.3** | |
79
+ | · mobile text / icon (290 / 211) | 95.9 / 71.1 | 94.5 / 70.6 | |
80
+ | · web text / icon (234 / 203) | 87.6 / 70.9 | 85.0 / 62.1 | web icons: the main regression of v1.1 |
81
+ | Desktop (OS-Atlas macOS / Windows / Linux, held-out, 1,028) | 39.2 | **50.6** | |
82
+ | Desktop "not on screen" abstention (26) | 38.5 | **80.8** | small set |
83
+ | Thai GUI held-out domains, `point` (1,611) | 80.1 | 78.0 | teacher-written Thai/English instructions on unseen Thai sites |
84
+ | Thai GUI set-of-marks `choice` (476) | 92.9 | **93.5** | pick the element index from the element table |
85
+ | Thai GUI `operation` choice (476) | 99.8 | 99.6 | CLICK / TYPE_TEXT / SELECT |
86
+ | Thai screen QA `noul` / `choice` (623 / 341) | 94.9 / 85.0 | 93.9 / 84.8 | is a dialog open, which page type, is there a form… |
87
+ | Thai GUI "not on screen" abstention (179) | 45.8 | 46.4 | known weak spot: check `abstain` before clicking |
88
+
89
+ ### Thai voice commands (operation + slots from one spoken sentence)
90
+ | set | v1.0 | **v1.1** | notes |
91
+ |---|---|---|---|
92
+ | real commands spoken in use, operation (183) | 65.0 | **88.5** | hand-labelled from a live macOS voice-control session |
93
+ | · slots (site / app / key / Nth) when the operation is right | 90.7 | **100.0** | |
94
+ | · chit-chat between commands acted on (41) | 14.6 | **7.3** | lower is better |
95
+ | real off-command conversation picked up by the microphone, acted on (227) | 3.5 | **2.6** | lower is better; before the controller's safety checks |
96
+ | synthetic voice commands, held out (1,406 questions) | 61.2 | **90.5** | |
97
+ | synthetic hard cases (fragments, site vs app, ordinals, 446 questions) | 55.0 | **90.8** | |
98
+
99
+ ### Doom from pixels (`demo/doom_demo.py`, defend_the_center, skill 3, 20 episodes, seed 1234)
100
+ | input | v1.0 kills / episode | **v1.1 kills / episode** | ms / decision (H100) |
101
+ |---|---|---|---|
102
+ | two 320×240 frames only | 5.5 | **11.9** (1–22) | 50 |
103
+ | frames + symbolic state | 6.0 | 6.4 | 49 |
104
+ | symbolic state only (text) | 7.3 | **8.7** | 40 |
105
+ | the privileged scripted expert it imitates | 6.8 | 6.8 | – |
106
+
107
+ Offline Doom decisions (600 frames): move 87.2, attack 96.8, in-danger 96.5, nearest-enemy point 52.9, enemy-absent 96.6.
108
+
109
+ ### Text-only tasks
110
+ Public 13-subset benchmark (the Bespoke / Jev public suite; `noul`/`choice` = accuracy, `score` = exact level):
111
+
112
+ | subset | v0.3 | v1.0 | **v1.1** | Jev 1.13.0* | Nimble-9B* |
113
+ |---|---|---|---|---|---|
114
+ | aegis2 | 83.2 | 84.4 | 82.0 | 80.4 | 81.2 |
115
+ | boolq | 79.7 | 80.0 | 79.7 | 89.7 | 86.0 |
116
+ | civil_comments | 79.0 | 74.0 | **86.0** | 81.0 | 70.3 |
117
+ | helpsteer2 (score) | 41.6 | 41.6 | 42.0 | 34.1 | 39.0 |
118
+ | massive-de-DE | 88.3 | 89.4 | 89.4 | 86.9 | 83.4 |
119
+ | massive-en-US | 88.3 | 88.9 | 89.4 | 87.4 | 86.9 |
120
+ | multinli | 89.0 | 88.3 | 89.0 | 82.9 | 85.3 |
121
+ | paws | 94.0 | 93.6 | 94.0 | 89.2 | 82.8 |
122
+ | pubmedqa | 64.0 | 69.2 | 67.6 | 77.2 | 75.6 |
123
+ | squad2 | 89.3 | 89.6 | 89.0 | 82.9 | 80.6 |
124
+ | summeval-consistency (score) | 75.0 | 71.5 | 75.0 | 81.2 | 75.7 |
125
+ | summeval-relevance (score) | 21.7 | 19.2 | 21.2 | 35.0 | 49.2 |
126
+ | vitaminc-dev | 72.5 | 70.3 | 71.3 | 80.1 | 76.6 |
127
+ | **macro, 13 subsets** | 74.3 | 73.8 | **75.0** | 76.0 | 74.8 |
128
+ | macro, 10 accuracy subsets | 82.7 | 82.8 | **83.7** | 83.8 | 80.9 |
129
+
130
+ \* published by Bespoke Labs for their own harness on the same subsets (Jev via the TypeSafe API).
131
+ With 8 option orders averaged: v1.1 macro 75.2 (10 accuracy subsets 84.0).
132
+
133
+ Thai held-out and eval-split sets (`th` unless noted; banking77 and xLAM are English):
134
+
135
+ | set | v0.3 | v1.0 | **v1.1** | v1.1, 8 orders |
136
+ |---|---|---|---|---|
137
+ | banking77 (77-way, whole dataset held out) | 45.4 | 57.5 | **62.0** | 64.6 |
138
+ | contrastive pairs, one-fact flips | 80.7 | 81.4 | **82.1** | 81.1 |
139
+ | MASSIVE-th intent | 90.0 | 90.8 | **91.1** | 92.0 |
140
+ | Prachathai topics | 98.1 | 98.7 | 98.7 | 98.7 |
141
+ | SIB-200 Thai topic (whole dataset held out) | 77.9 | 76.5 | 75.5 | 76.5 |
142
+ | Wisesight sentiment (whole dataset held out) | 51.6 | 52.6 | 52.8 | 52.0 |
143
+ | Wongnai stars (score, exact) | 63.5 | 63.1 | 63.1 | 63.1 |
144
+ | xLAM tool selection | 99.4 | 99.4 | 99.3 | 99.4 |
145
+ | XNLI-th | 77.1 | 76.3 | 75.6 | 75.8 |
146
+ | **macro, 8 accuracy sets** | 77.5 | 79.1 | **79.6** | 80.0 |
147
 
148
  ### Latency (batch 1, H100, end-to-end `SystemOneClient`)
149
  | request | tokens | ms |
150
  |---|---|---|
151
+ | two 320×240 game frames + short state, 2 questions | 298 (160 visual) | 50 |
152
+ | 1280×800 screenshot + goal, `point` + `choice` | 1,062 (1,000 visual) | 52 |
153
  | text only (3-question Thai ticket) | 166 | ~40 |
154
+ | voice command intent (6 questions, up to 250 app names) | ~2,000 | 100–250 round trip from a Mac to a remote GPU, incl. HTTP |
155
+
156
+ ## Versions
157
+
158
+ | version | date | change |
159
+ |---|---|---|
160
+ | **v1.1** | 2026-09-24 | + Thai voice-command intents (28k synthetic training records from 21k distinct commands + 8.9k hard-case records: fragments / chit-chat = NONE, site vs app, ordinal + topic, ASR spelling variants), desktop grounding round (OS-Atlas macOS / Windows / Linux), text-anchor round (civil_comments, VitaminC); temperatures recalibrated. Voice 65 → 88.5, desktop 39 → 51, Doom pixels 5.5 → 11.9 kills, ScreenSpot-v2 82.7 → 80.7. |
161
+ | v1.0 | 2026-09-23 | first vision release (Thai GUI crawl, public grounding, Doom rollouts, screen QA on the v0.3 text tower) |
162
 
163
  ## How it was built
164
 
 
166
  2. Vision tower + merger from Qwen3.5-0.8B-Base, re-aligned with an LM loss on 200k+ image→text pairs (Thai rendered
167
  OCR pages in 55 fonts, Thai web screenshots → element tables, Doom frames → object tables, public screen/document QA).
168
  3. Decision SFT on the v0.3 text mix plus Thai GUI grounding (Playwright crawl of 300+ Thai sites, teacher-written and
169
+ blind-verified Thai/English instructions), public English grounding (Wave-UI, GroundUI, ShowUI), desktop grounding
170
+ (OS-Atlas), Doom rollouts (privileged expert, 1.5M frames) with `move` / `attack` / `in_danger` / `nearest_enemy`
171
+ questions, and screen QA.
172
+ 4. v1.1: a text-anchor round and a desktop round averaged into one checkpoint, then a voice-command round
173
+ (label-conditioned generation by a teacher LLM, every sentence blind re-labelled and kept only on agreement) and a
174
+ hard-case round aimed at the mistakes seen in real use; the rest of the mix is rehearsed at the same time.
175
+ 5. Calibration of the per-type temperatures.
176
 
177
  ## License & credits
178
 
config.json CHANGED
@@ -4,12 +4,16 @@
4
  "architectures": [
5
  "OpenThaiSystemOneForDecision"
6
  ],
 
 
 
 
7
  "dtype": "bfloat16",
8
  "head_bias": true,
9
  "image_token_id": 248056,
10
  "model_type": "openthai_systemone",
11
  "n_slots": 256,
12
- "n_temperatures": 4,
13
  "pad_token_id": 248044,
14
  "point_head": {
15
  "dim": 256,
@@ -145,9 +149,5 @@
145
  "temporal_patch_size": 2
146
  },
147
  "vision_end_token_id": 248054,
148
- "vision_start_token_id": 248053,
149
- "auto_map": {
150
- "AutoConfig": "configuration.OpenThaiSystemOneConfig",
151
- "AutoModel": "modeling.OpenThaiSystemOneForDecision"
152
- }
153
  }
 
4
  "architectures": [
5
  "OpenThaiSystemOneForDecision"
6
  ],
7
+ "auto_map": {
8
+ "AutoConfig": "configuration.OpenThaiSystemOneConfig",
9
+ "AutoModel": "modeling.OpenThaiSystemOneForDecision"
10
+ },
11
  "dtype": "bfloat16",
12
  "head_bias": true,
13
  "image_token_id": 248056,
14
  "model_type": "openthai_systemone",
15
  "n_slots": 256,
16
+ "n_temperatures": 3,
17
  "pad_token_id": 248044,
18
  "point_head": {
19
  "dim": 256,
 
149
  "temporal_patch_size": 2
150
  },
151
  "vision_end_token_id": 248054,
152
+ "vision_start_token_id": 248053
 
 
 
 
153
  }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:9a69dba19b55dfe7ed3c80c24e16e708ce2571961ae844e4de5dd0da71fa7bbf
3
- size 1724713898
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b2b257815523a460751cf0fded102ac103ea9c46df8de645eda9bef0bd429bc4
3
+ size 1724708176
modeling.py CHANGED
@@ -78,24 +78,17 @@ class OpenThaiSystemOneForDecision(PreTrainedModel):
78
 
79
  def __init__(self, config: OpenThaiSystemOneConfig):
80
  super().__init__(config)
 
 
 
 
81
  if config.is_vision:
82
- from transformers.models.qwen3_5.configuration_qwen3_5 import Qwen3_5Config
83
- from transformers.models.qwen3_5.modeling_qwen3_5 import Qwen3_5Model
84
-
85
- mm = Qwen3_5Config(
86
- text_config=config.text_config,
87
- vision_config=config.vision_config,
88
- image_token_id=config.image_token_id,
89
- vision_start_token_id=config.vision_start_token_id,
90
- vision_end_token_id=config.vision_end_token_id,
91
- tie_word_embeddings=False,
92
- )
93
- self.model = Qwen3_5Model(mm)
94
  enable_vision_helper_caches()
95
  ph = dict(config.point_head or {})
96
  self.point_head = PointHead(config.hidden_size, ph.get("dim", 256), ph.get("n_sa_layers", 1), ph.get("n_heads", 8))
97
  else:
98
- self.model = AutoModel.from_config(config.text_config)
99
  self.point_head = None
100
  self.slot_head = nn.Linear(config.hidden_size, config.n_slots, bias=config.head_bias)
101
  # log-temperatures per question type (choice/score/noul[/point]); learned in the calibration stage
@@ -104,7 +97,26 @@ class OpenThaiSystemOneForDecision(PreTrainedModel):
104
 
105
  @property
106
  def language_model(self):
107
- return self.model.language_model if self.config.is_vision else self.model
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
108
 
109
  def get_input_embeddings(self):
110
  return self.language_model.get_input_embeddings()
@@ -180,7 +192,7 @@ class OpenThaiSystemOneForDecision(PreTrainedModel):
180
  abstain_slot=text_model.config.abstain_slot,
181
  answer_token_id=text_model.config.answer_token_id,
182
  head_bias=text_model.config.head_bias,
183
- n_temperatures=4,
184
  vision_config=vcfg,
185
  image_token_id=tok.convert_tokens_to_ids(QWEN_IMAGE_PAD),
186
  vision_start_token_id=tok.convert_tokens_to_ids(QWEN_VISION_START),
@@ -196,7 +208,7 @@ class OpenThaiSystemOneForDecision(PreTrainedModel):
196
  if len(tok) > old_vocab:
197
  text_model.model.resize_token_embeddings(len(tok), mean_resizing=False)
198
  _init_new_token_embeddings(text_model.get_input_embeddings().weight, tok, len(tok) - old_vocab)
199
- missing, unexpected = model.model.language_model.load_state_dict(text_model.model.state_dict(), strict=False)
200
  assert not unexpected and not missing, (missing, unexpected)
201
  model.slot_head.load_state_dict(text_model.slot_head.state_dict())
202
  with torch.no_grad():
@@ -212,9 +224,9 @@ class OpenThaiSystemOneForDecision(PreTrainedModel):
212
  if k.startswith(prefix):
213
  sd[k[len(prefix):]] = fh.get_tensor(k)
214
  break
215
- missing, unexpected = model.model.visual.load_state_dict({k: v.to(torch_dtype) for k, v in sd.items()}, strict=False)
216
  assert not unexpected and not missing, (missing, unexpected)
217
- model.model.config.text_config = cfg.text_config
218
  return model, tok
219
 
220
  # ------------------------------------------------------------------ forward
@@ -300,19 +312,9 @@ class OpenThaiSystemOneForDecision(PreTrainedModel):
300
  position_ids: Optional[torch.Tensor] = None,
301
  **kwargs,
302
  ) -> DecisionOutput:
303
- if pixel_values is not None:
304
- if not self.config.is_vision:
305
- raise ValueError("this checkpoint is text-only; images need OpenThai-SystemOne-Vision")
306
- # position_ids (3, B, T) from formatting.mrope_position_ids skips the slow per-token get_rope_index;
307
- # image_grid_thw may stay on the CPU (no GPU syncs in the python bookkeeping) — helper outputs go to the pixel device
308
- _VISION_TARGET_DEVICE[0] = pixel_values.device
309
- out = self.model(
310
- input_ids=input_ids, attention_mask=attention_mask, pixel_values=pixel_values,
311
- image_grid_thw=image_grid_thw, mm_token_type_ids=mm_token_type_ids, position_ids=position_ids, **kwargs,
312
- )
313
- else:
314
- out = self.model(input_ids=input_ids, attention_mask=attention_mask, **kwargs)
315
- hidden = out.last_hidden_state
316
  if answer_positions is None:
317
  answer_positions = (input_ids == self.config.answer_token_id).nonzero()[:, 1].unsqueeze(0)
318
  if option_counts is None:
@@ -416,6 +418,52 @@ def enable_vision_helper_caches(maxsize: int = 512):
416
  _VISION_CACHE_ON = True
417
 
418
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
419
  def use_reference_kernels():
420
  """Force the pure-PyTorch Gated-DeltaNet / causal-conv paths.
421
 
 
78
 
79
  def __init__(self, config: OpenThaiSystemOneConfig):
80
  super().__init__(config)
81
+ # Parameter layout is the SAME as the text-only line for everything they share: text tower `model.*`,
82
+ # `slot_head.*`, 3 `log_temperature`s. The vision parts sit beside it (`visual.*`, `point_head.*`), so a
83
+ # text-only v0.x client loading a -Vision checkpoint finds every weight it knows and ignores the rest.
84
+ self.model = AutoModel.from_config(config.text_config)
85
  if config.is_vision:
86
+ self.visual = AutoModel.from_config(config.vision_config)
 
 
 
 
 
 
 
 
 
 
 
87
  enable_vision_helper_caches()
88
  ph = dict(config.point_head or {})
89
  self.point_head = PointHead(config.hidden_size, ph.get("dim", 256), ph.get("n_sa_layers", 1), ph.get("n_heads", 8))
90
  else:
91
+ self.visual = None
92
  self.point_head = None
93
  self.slot_head = nn.Linear(config.hidden_size, config.n_slots, bias=config.head_bias)
94
  # log-temperatures per question type (choice/score/noul[/point]); learned in the calibration stage
 
97
 
98
  @property
99
  def language_model(self):
100
+ return self.model
101
+
102
+ def encode(self, input_ids, attention_mask=None, pixel_values=None, image_grid_thw=None, position_ids=None, **kwargs):
103
+ """Final hidden states. With images: ViT -> merger -> splice into the <|image_pad|> positions, 3-D (t,h,w) position
104
+ ids (computed here when not given), then the text tower. Without images: the plain text-tower forward."""
105
+ if pixel_values is None:
106
+ return self.model(input_ids=input_ids, attention_mask=attention_mask, position_ids=position_ids, **kwargs).last_hidden_state
107
+ if self.visual is None:
108
+ raise ValueError("this checkpoint is text-only; images need OpenThai-SystemOne-Vision")
109
+ _VISION_TARGET_DEVICE[0] = pixel_values.device
110
+ embeds = self.model.get_input_embeddings()(input_ids)
111
+ img = self.visual(pixel_values.to(self.visual.dtype), grid_thw=image_grid_thw).pooler_output
112
+ mask = (input_ids == self.config.image_token_id).unsqueeze(-1)
113
+ if int(mask.sum()) != img.shape[0]:
114
+ raise ValueError(f"image tokens ({int(mask.sum())}) != image features ({img.shape[0]})")
115
+ embeds = embeds.masked_scatter(mask.to(embeds.device), img.to(embeds.device, embeds.dtype))
116
+ if position_ids is None:
117
+ position_ids = mrope_positions_from_ids(input_ids, attention_mask, image_grid_thw, self.config.image_token_id,
118
+ int(getattr(self.config.vision_config, "spatial_merge_size", 2)))
119
+ return self.model(inputs_embeds=embeds, attention_mask=attention_mask, position_ids=position_ids.to(embeds.device), **kwargs).last_hidden_state
120
 
121
  def get_input_embeddings(self):
122
  return self.language_model.get_input_embeddings()
 
192
  abstain_slot=text_model.config.abstain_slot,
193
  answer_token_id=text_model.config.answer_token_id,
194
  head_bias=text_model.config.head_bias,
195
+ n_temperatures=text_model.config.n_temperatures,
196
  vision_config=vcfg,
197
  image_token_id=tok.convert_tokens_to_ids(QWEN_IMAGE_PAD),
198
  vision_start_token_id=tok.convert_tokens_to_ids(QWEN_VISION_START),
 
208
  if len(tok) > old_vocab:
209
  text_model.model.resize_token_embeddings(len(tok), mean_resizing=False)
210
  _init_new_token_embeddings(text_model.get_input_embeddings().weight, tok, len(tok) - old_vocab)
211
+ missing, unexpected = model.model.load_state_dict(text_model.model.state_dict(), strict=False)
212
  assert not unexpected and not missing, (missing, unexpected)
213
  model.slot_head.load_state_dict(text_model.slot_head.state_dict())
214
  with torch.no_grad():
 
224
  if k.startswith(prefix):
225
  sd[k[len(prefix):]] = fh.get_tensor(k)
226
  break
227
+ missing, unexpected = model.visual.load_state_dict({k: v.to(torch_dtype) for k, v in sd.items()}, strict=False)
228
  assert not unexpected and not missing, (missing, unexpected)
229
+ model.model.config = cfg.text_config
230
  return model, tok
231
 
232
  # ------------------------------------------------------------------ forward
 
312
  position_ids: Optional[torch.Tensor] = None,
313
  **kwargs,
314
  ) -> DecisionOutput:
315
+ # position_ids (3, B, T) from formatting.mrope_position_ids avoid recomputing them; image_grid_thw may stay on the CPU
316
+ hidden = self.encode(input_ids, attention_mask=attention_mask, pixel_values=pixel_values, image_grid_thw=image_grid_thw,
317
+ position_ids=position_ids if pixel_values is not None else None, **kwargs)
 
 
 
 
 
 
 
 
 
 
318
  if answer_positions is None:
319
  answer_positions = (input_ids == self.config.answer_token_id).nonzero()[:, 1].unsqueeze(0)
320
  if option_counts is None:
 
418
  _VISION_CACHE_ON = True
419
 
420
 
421
+ def mrope_positions_from_ids(input_ids, attention_mask, image_grid_thw, image_token_id: int, merge: int):
422
+ """3-D position ids from token ids alone (same result as formatting.mrope_position_ids / Qwen's get_rope_index)."""
423
+ B, T = input_ids.shape
424
+ grids = [[int(v) for v in g] for g in image_grid_thw.tolist()]
425
+ ids = input_ids.tolist()
426
+ am = attention_mask.tolist() if attention_mask is not None else [[1] * T for _ in range(B)]
427
+ pos = torch.zeros((3, B, T), dtype=torch.long)
428
+ gi = 0
429
+ for b in range(B):
430
+ cur, i = 0, 0
431
+ n = sum(am[b])
432
+ while i < n:
433
+ if ids[b][i] == image_token_id:
434
+ t, h, w = grids[gi]; gi += 1
435
+ hm, wm = h // merge, w // merge
436
+ L = t * hm * wm
437
+ tt = torch.arange(t).view(t, 1, 1).expand(t, hm, wm).reshape(-1)
438
+ hh = torch.arange(hm).view(1, hm, 1).expand(t, hm, wm).reshape(-1)
439
+ ww = torch.arange(wm).view(1, 1, wm).expand(t, hm, wm).reshape(-1)
440
+ pos[0, b, i:i + L] = tt + cur; pos[1, b, i:i + L] = hh + cur; pos[2, b, i:i + L] = ww + cur
441
+ cur += max(hm, wm); i += L
442
+ else:
443
+ j = i
444
+ while j < n and ids[b][j] != image_token_id:
445
+ j += 1
446
+ pos[:, b, i:j] = torch.arange(j - i) + cur
447
+ cur += j - i; i = j
448
+ return pos
449
+
450
+
451
+ def convert_legacy_vision_state_dict(sd: dict) -> dict:
452
+ """Checkpoints trained before 2026-09-23 stored the vision model as a Qwen3_5Model (`model.language_model.*`,
453
+ `model.visual.*`, 4 temperatures). Map them onto the shared layout (`model.*`, `visual.*`, 3 temperatures)."""
454
+ out = {}
455
+ for k, v in sd.items():
456
+ if k.startswith("model.language_model."):
457
+ out["model." + k[len("model.language_model."):]] = v
458
+ elif k.startswith("model.visual."):
459
+ out["visual." + k[len("model.visual."):]] = v
460
+ elif k == "log_temperature" and v.numel() == 4:
461
+ out[k] = v[:3].clone()
462
+ else:
463
+ out[k] = v
464
+ return out
465
+
466
+
467
  def use_reference_kernels():
468
  """Force the pure-PyTorch Gated-DeltaNet / causal-conv paths.
469