OpenThai-SystemOne release
Browse files- README.md +110 -32
- config.json +6 -6
- model.safetensors +2 -2
- modeling.py +79 -31
README.md
CHANGED
|
@@ -2,23 +2,26 @@
|
|
| 2 |
license: apache-2.0
|
| 3 |
language: [th, en]
|
| 4 |
pipeline_tag: image-text-to-text
|
| 5 |
-
tags: [system-one, decision-model, gui-grounding, computer-use, thai, qwen3.5, vision]
|
| 6 |
base_model: Qwen/Qwen3.5-0.8B-Base
|
| 7 |
---
|
| 8 |
|
| 9 |
-
# OpenThai-SystemOne-Vision (v1.
|
| 10 |
|
| 11 |
The vision variant of [OpenThai-SystemOne](https://huggingface.co/iapp/OpenThai-SystemOne): the same open Thai + English
|
| 12 |
System One decision model, now with **images in the state** and a fourth question type, **`point`**, that returns where
|
| 13 |
on a screenshot something is. Screenshot in, click out, in one forward pass, no text generation.
|
| 14 |
|
|
|
|
|
|
|
|
|
|
| 15 |
| | OpenThai-SystemOne (text) | OpenThai-SystemOne-Vision |
|
| 16 |
|---|---|---|
|
| 17 |
| parameters | 0.75B | 0.86B (ViT 100M + point head 9M) |
|
| 18 |
| question types | choice / score / noul | choice / score / noul / **point** |
|
| 19 |
| state | text or JSON | text or JSON + up to 4 images |
|
| 20 |
-
| typical use | routing, moderation, RAG relevance, tool choice, agents with element tables | GUI agents from pixels, Thai screen reading, game frames, documents |
|
| 21 |
-
| text-only behaviour | v0.3 |
|
| 22 |
|
| 23 |
## API (superset of the TypeSafe `/v1/systemone` contract)
|
| 24 |
|
|
@@ -51,40 +54,111 @@ r = c.system_one("goal: open the login form", {"where": {"type": "point", "instr
|
|
| 51 |
print(r.answers["where"].x, r.answers["where"].y, r.answers["where"].confidence)
|
| 52 |
```
|
| 53 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
## Evaluation
|
| 55 |
|
| 56 |
-
|
|
|
|
| 57 |
|
| 58 |
### GUI grounding (`point`: predicted click inside the target box)
|
| 59 |
-
| benchmark |
|
| 60 |
-
|---|---|---|
|
| 61 |
-
| ScreenSpot-v2 (1,272) | **
|
| 62 |
-
|
|
| 63 |
-
|
|
| 64 |
-
|
|
| 65 |
-
|
|
| 66 |
-
| "not on screen" abstention (
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
|
| 70 |
-
|
|
| 71 |
-
|
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
|
| 78 |
-
|-
|
| 79 |
-
|
|
| 80 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 81 |
|
| 82 |
### Latency (batch 1, H100, end-to-end `SystemOneClient`)
|
| 83 |
| request | tokens | ms |
|
| 84 |
|---|---|---|
|
| 85 |
-
| two 320×240 game frames + short state, 2 questions | 298 (160 visual) |
|
| 86 |
-
| 1280×800 screenshot + goal, `point` + `choice` | 1,062 (1,000 visual) |
|
| 87 |
| text only (3-question Thai ticket) | 166 | ~40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 88 |
|
| 89 |
## How it was built
|
| 90 |
|
|
@@ -92,9 +166,13 @@ Numbers are for the released checkpoint (calibrated round 3); single option orde
|
|
| 92 |
2. Vision tower + merger from Qwen3.5-0.8B-Base, re-aligned with an LM loss on 200k+ image→text pairs (Thai rendered
|
| 93 |
OCR pages in 55 fonts, Thai web screenshots → element tables, Doom frames → object tables, public screen/document QA).
|
| 94 |
3. Decision SFT on the v0.3 text mix plus Thai GUI grounding (Playwright crawl of 300+ Thai sites, teacher-written and
|
| 95 |
-
blind-verified Thai/English instructions), public English grounding (Wave-UI, GroundUI),
|
| 96 |
-
expert, 1.5M frames) with `move` / `attack` / `in_danger` / `nearest_enemy`
|
| 97 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
|
| 99 |
## License & credits
|
| 100 |
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
language: [th, en]
|
| 4 |
pipeline_tag: image-text-to-text
|
| 5 |
+
tags: [system-one, decision-model, gui-grounding, computer-use, voice-control, thai, qwen3.5, vision]
|
| 6 |
base_model: Qwen/Qwen3.5-0.8B-Base
|
| 7 |
---
|
| 8 |
|
| 9 |
+
# OpenThai-SystemOne-Vision (v1.1)
|
| 10 |
|
| 11 |
The vision variant of [OpenThai-SystemOne](https://huggingface.co/iapp/OpenThai-SystemOne): the same open Thai + English
|
| 12 |
System One decision model, now with **images in the state** and a fourth question type, **`point`**, that returns where
|
| 13 |
on a screenshot something is. Screenshot in, click out, in one forward pass, no text generation.
|
| 14 |
|
| 15 |
+
**v1.1 (2026-09-24)** adds Thai **voice-command understanding** for hands-free computer control (a spoken Thai command +
|
| 16 |
+
the app and window title in, the operation and its target out) and stronger desktop and Doom play. See [Versions](#versions).
|
| 17 |
+
|
| 18 |
| | OpenThai-SystemOne (text) | OpenThai-SystemOne-Vision |
|
| 19 |
|---|---|---|
|
| 20 |
| parameters | 0.75B | 0.86B (ViT 100M + point head 9M) |
|
| 21 |
| question types | choice / score / noul | choice / score / noul / **point** |
|
| 22 |
| state | text or JSON | text or JSON + up to 4 images |
|
| 23 |
+
| typical use | routing, moderation, RAG relevance, tool choice, agents with element tables | GUI agents from pixels, Thai voice control, Thai screen reading, game frames, documents |
|
| 24 |
+
| text-only behaviour | v0.3 | same API; text scores equal or better than v0.3 (tables below) |
|
| 25 |
|
| 26 |
## API (superset of the TypeSafe `/v1/systemone` contract)
|
| 27 |
|
|
|
|
| 54 |
print(r.answers["where"].x, r.answers["where"].y, r.answers["where"].confidence)
|
| 55 |
```
|
| 56 |
|
| 57 |
+
### Thai voice control
|
| 58 |
+
|
| 59 |
+
The voice-command schema (21 operations such as `SEARCH`, `CLICK_NTH`, `OPEN_WEBSITE`, `QUIT_APP`, `PLAY_PAUSE`, `LIKE`,
|
| 60 |
+
`COMMENT`, `NONE` = not a command, plus `search_site` / `app` / `site` / `key` / `nth` slots) and a macOS controller
|
| 61 |
+
(on-device Thai speech recognition → this model → native mouse / keyboard events) will be published in the GitHub
|
| 62 |
+
repo under `demo/voice_control/`. One text-only call decides the operation; clicks add one screenshot call.
|
| 63 |
+
|
| 64 |
+
```python
|
| 65 |
+
state = 'คำสั่งเสียงภาษาไทยจากผู้ใช้ที่ควบคุมคอมพิวเตอร์ด้วยเสียง: "อยากดูเนเน่ร้องเพลง"\nโปรแกรมที่ใช้อยู่: Google Chrome\nหน้าต่าง: Facebook'
|
| 66 |
+
# questions = intent_schema.intent_questions(apps) -> operation SEARCH, search_site YouTube
|
| 67 |
+
```
|
| 68 |
+
|
| 69 |
## Evaluation
|
| 70 |
|
| 71 |
+
All numbers are for this checkpoint (v1.1) next to v1.0 and the text model v0.3, measured with the same scripts, single
|
| 72 |
+
option order (no permutation averaging) unless noted. Accuracy in %.
|
| 73 |
|
| 74 |
### GUI grounding (`point`: predicted click inside the target box)
|
| 75 |
+
| benchmark | v1.0 | **v1.1** | notes |
|
| 76 |
+
|---|---|---|---|
|
| 77 |
+
| ScreenSpot-v2, all (1,272) | 82.7 | **80.7** | reference: GUI-Actor-2B 88.6, Qwen3-VL-2B 86.7, UI-TARS-1.5-7B 89.0 (published) |
|
| 78 |
+
| · desktop text / icon (194 / 140) | 90.2 / 71.4 | 89.7 / **74.3** | |
|
| 79 |
+
| · mobile text / icon (290 / 211) | 95.9 / 71.1 | 94.5 / 70.6 | |
|
| 80 |
+
| · web text / icon (234 / 203) | 87.6 / 70.9 | 85.0 / 62.1 | web icons: the main regression of v1.1 |
|
| 81 |
+
| Desktop (OS-Atlas macOS / Windows / Linux, held-out, 1,028) | 39.2 | **50.6** | |
|
| 82 |
+
| Desktop "not on screen" abstention (26) | 38.5 | **80.8** | small set |
|
| 83 |
+
| Thai GUI held-out domains, `point` (1,611) | 80.1 | 78.0 | teacher-written Thai/English instructions on unseen Thai sites |
|
| 84 |
+
| Thai GUI set-of-marks `choice` (476) | 92.9 | **93.5** | pick the element index from the element table |
|
| 85 |
+
| Thai GUI `operation` choice (476) | 99.8 | 99.6 | CLICK / TYPE_TEXT / SELECT |
|
| 86 |
+
| Thai screen QA `noul` / `choice` (623 / 341) | 94.9 / 85.0 | 93.9 / 84.8 | is a dialog open, which page type, is there a form… |
|
| 87 |
+
| Thai GUI "not on screen" abstention (179) | 45.8 | 46.4 | known weak spot: check `abstain` before clicking |
|
| 88 |
+
|
| 89 |
+
### Thai voice commands (operation + slots from one spoken sentence)
|
| 90 |
+
| set | v1.0 | **v1.1** | notes |
|
| 91 |
+
|---|---|---|---|
|
| 92 |
+
| real commands spoken in use, operation (183) | 65.0 | **88.5** | hand-labelled from a live macOS voice-control session |
|
| 93 |
+
| · slots (site / app / key / Nth) when the operation is right | 90.7 | **100.0** | |
|
| 94 |
+
| · chit-chat between commands acted on (41) | 14.6 | **7.3** | lower is better |
|
| 95 |
+
| real off-command conversation picked up by the microphone, acted on (227) | 3.5 | **2.6** | lower is better; before the controller's safety checks |
|
| 96 |
+
| synthetic voice commands, held out (1,406 questions) | 61.2 | **90.5** | |
|
| 97 |
+
| synthetic hard cases (fragments, site vs app, ordinals, 446 questions) | 55.0 | **90.8** | |
|
| 98 |
+
|
| 99 |
+
### Doom from pixels (`demo/doom_demo.py`, defend_the_center, skill 3, 20 episodes, seed 1234)
|
| 100 |
+
| input | v1.0 kills / episode | **v1.1 kills / episode** | ms / decision (H100) |
|
| 101 |
+
|---|---|---|---|
|
| 102 |
+
| two 320×240 frames only | 5.5 | **11.9** (1–22) | 50 |
|
| 103 |
+
| frames + symbolic state | 6.0 | 6.4 | 49 |
|
| 104 |
+
| symbolic state only (text) | 7.3 | **8.7** | 40 |
|
| 105 |
+
| the privileged scripted expert it imitates | 6.8 | 6.8 | – |
|
| 106 |
+
|
| 107 |
+
Offline Doom decisions (600 frames): move 87.2, attack 96.8, in-danger 96.5, nearest-enemy point 52.9, enemy-absent 96.6.
|
| 108 |
+
|
| 109 |
+
### Text-only tasks
|
| 110 |
+
Public 13-subset benchmark (the Bespoke / Jev public suite; `noul`/`choice` = accuracy, `score` = exact level):
|
| 111 |
+
|
| 112 |
+
| subset | v0.3 | v1.0 | **v1.1** | Jev 1.13.0* | Nimble-9B* |
|
| 113 |
+
|---|---|---|---|---|---|
|
| 114 |
+
| aegis2 | 83.2 | 84.4 | 82.0 | 80.4 | 81.2 |
|
| 115 |
+
| boolq | 79.7 | 80.0 | 79.7 | 89.7 | 86.0 |
|
| 116 |
+
| civil_comments | 79.0 | 74.0 | **86.0** | 81.0 | 70.3 |
|
| 117 |
+
| helpsteer2 (score) | 41.6 | 41.6 | 42.0 | 34.1 | 39.0 |
|
| 118 |
+
| massive-de-DE | 88.3 | 89.4 | 89.4 | 86.9 | 83.4 |
|
| 119 |
+
| massive-en-US | 88.3 | 88.9 | 89.4 | 87.4 | 86.9 |
|
| 120 |
+
| multinli | 89.0 | 88.3 | 89.0 | 82.9 | 85.3 |
|
| 121 |
+
| paws | 94.0 | 93.6 | 94.0 | 89.2 | 82.8 |
|
| 122 |
+
| pubmedqa | 64.0 | 69.2 | 67.6 | 77.2 | 75.6 |
|
| 123 |
+
| squad2 | 89.3 | 89.6 | 89.0 | 82.9 | 80.6 |
|
| 124 |
+
| summeval-consistency (score) | 75.0 | 71.5 | 75.0 | 81.2 | 75.7 |
|
| 125 |
+
| summeval-relevance (score) | 21.7 | 19.2 | 21.2 | 35.0 | 49.2 |
|
| 126 |
+
| vitaminc-dev | 72.5 | 70.3 | 71.3 | 80.1 | 76.6 |
|
| 127 |
+
| **macro, 13 subsets** | 74.3 | 73.8 | **75.0** | 76.0 | 74.8 |
|
| 128 |
+
| macro, 10 accuracy subsets | 82.7 | 82.8 | **83.7** | 83.8 | 80.9 |
|
| 129 |
+
|
| 130 |
+
\* published by Bespoke Labs for their own harness on the same subsets (Jev via the TypeSafe API).
|
| 131 |
+
With 8 option orders averaged: v1.1 macro 75.2 (10 accuracy subsets 84.0).
|
| 132 |
+
|
| 133 |
+
Thai held-out and eval-split sets (`th` unless noted; banking77 and xLAM are English):
|
| 134 |
+
|
| 135 |
+
| set | v0.3 | v1.0 | **v1.1** | v1.1, 8 orders |
|
| 136 |
+
|---|---|---|---|---|
|
| 137 |
+
| banking77 (77-way, whole dataset held out) | 45.4 | 57.5 | **62.0** | 64.6 |
|
| 138 |
+
| contrastive pairs, one-fact flips | 80.7 | 81.4 | **82.1** | 81.1 |
|
| 139 |
+
| MASSIVE-th intent | 90.0 | 90.8 | **91.1** | 92.0 |
|
| 140 |
+
| Prachathai topics | 98.1 | 98.7 | 98.7 | 98.7 |
|
| 141 |
+
| SIB-200 Thai topic (whole dataset held out) | 77.9 | 76.5 | 75.5 | 76.5 |
|
| 142 |
+
| Wisesight sentiment (whole dataset held out) | 51.6 | 52.6 | 52.8 | 52.0 |
|
| 143 |
+
| Wongnai stars (score, exact) | 63.5 | 63.1 | 63.1 | 63.1 |
|
| 144 |
+
| xLAM tool selection | 99.4 | 99.4 | 99.3 | 99.4 |
|
| 145 |
+
| XNLI-th | 77.1 | 76.3 | 75.6 | 75.8 |
|
| 146 |
+
| **macro, 8 accuracy sets** | 77.5 | 79.1 | **79.6** | 80.0 |
|
| 147 |
|
| 148 |
### Latency (batch 1, H100, end-to-end `SystemOneClient`)
|
| 149 |
| request | tokens | ms |
|
| 150 |
|---|---|---|
|
| 151 |
+
| two 320×240 game frames + short state, 2 questions | 298 (160 visual) | 50 |
|
| 152 |
+
| 1280×800 screenshot + goal, `point` + `choice` | 1,062 (1,000 visual) | 52 |
|
| 153 |
| text only (3-question Thai ticket) | 166 | ~40 |
|
| 154 |
+
| voice command intent (6 questions, up to 250 app names) | ~2,000 | 100–250 round trip from a Mac to a remote GPU, incl. HTTP |
|
| 155 |
+
|
| 156 |
+
## Versions
|
| 157 |
+
|
| 158 |
+
| version | date | change |
|
| 159 |
+
|---|---|---|
|
| 160 |
+
| **v1.1** | 2026-09-24 | + Thai voice-command intents (28k synthetic training records from 21k distinct commands + 8.9k hard-case records: fragments / chit-chat = NONE, site vs app, ordinal + topic, ASR spelling variants), desktop grounding round (OS-Atlas macOS / Windows / Linux), text-anchor round (civil_comments, VitaminC); temperatures recalibrated. Voice 65 → 88.5, desktop 39 → 51, Doom pixels 5.5 → 11.9 kills, ScreenSpot-v2 82.7 → 80.7. |
|
| 161 |
+
| v1.0 | 2026-09-23 | first vision release (Thai GUI crawl, public grounding, Doom rollouts, screen QA on the v0.3 text tower) |
|
| 162 |
|
| 163 |
## How it was built
|
| 164 |
|
|
|
|
| 166 |
2. Vision tower + merger from Qwen3.5-0.8B-Base, re-aligned with an LM loss on 200k+ image→text pairs (Thai rendered
|
| 167 |
OCR pages in 55 fonts, Thai web screenshots → element tables, Doom frames → object tables, public screen/document QA).
|
| 168 |
3. Decision SFT on the v0.3 text mix plus Thai GUI grounding (Playwright crawl of 300+ Thai sites, teacher-written and
|
| 169 |
+
blind-verified Thai/English instructions), public English grounding (Wave-UI, GroundUI, ShowUI), desktop grounding
|
| 170 |
+
(OS-Atlas), Doom rollouts (privileged expert, 1.5M frames) with `move` / `attack` / `in_danger` / `nearest_enemy`
|
| 171 |
+
questions, and screen QA.
|
| 172 |
+
4. v1.1: a text-anchor round and a desktop round averaged into one checkpoint, then a voice-command round
|
| 173 |
+
(label-conditioned generation by a teacher LLM, every sentence blind re-labelled and kept only on agreement) and a
|
| 174 |
+
hard-case round aimed at the mistakes seen in real use; the rest of the mix is rehearsed at the same time.
|
| 175 |
+
5. Calibration of the per-type temperatures.
|
| 176 |
|
| 177 |
## License & credits
|
| 178 |
|
config.json
CHANGED
|
@@ -4,12 +4,16 @@
|
|
| 4 |
"architectures": [
|
| 5 |
"OpenThaiSystemOneForDecision"
|
| 6 |
],
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
"dtype": "bfloat16",
|
| 8 |
"head_bias": true,
|
| 9 |
"image_token_id": 248056,
|
| 10 |
"model_type": "openthai_systemone",
|
| 11 |
"n_slots": 256,
|
| 12 |
-
"n_temperatures":
|
| 13 |
"pad_token_id": 248044,
|
| 14 |
"point_head": {
|
| 15 |
"dim": 256,
|
|
@@ -145,9 +149,5 @@
|
|
| 145 |
"temporal_patch_size": 2
|
| 146 |
},
|
| 147 |
"vision_end_token_id": 248054,
|
| 148 |
-
"vision_start_token_id": 248053
|
| 149 |
-
"auto_map": {
|
| 150 |
-
"AutoConfig": "configuration.OpenThaiSystemOneConfig",
|
| 151 |
-
"AutoModel": "modeling.OpenThaiSystemOneForDecision"
|
| 152 |
-
}
|
| 153 |
}
|
|
|
|
| 4 |
"architectures": [
|
| 5 |
"OpenThaiSystemOneForDecision"
|
| 6 |
],
|
| 7 |
+
"auto_map": {
|
| 8 |
+
"AutoConfig": "configuration.OpenThaiSystemOneConfig",
|
| 9 |
+
"AutoModel": "modeling.OpenThaiSystemOneForDecision"
|
| 10 |
+
},
|
| 11 |
"dtype": "bfloat16",
|
| 12 |
"head_bias": true,
|
| 13 |
"image_token_id": 248056,
|
| 14 |
"model_type": "openthai_systemone",
|
| 15 |
"n_slots": 256,
|
| 16 |
+
"n_temperatures": 3,
|
| 17 |
"pad_token_id": 248044,
|
| 18 |
"point_head": {
|
| 19 |
"dim": 256,
|
|
|
|
| 149 |
"temporal_patch_size": 2
|
| 150 |
},
|
| 151 |
"vision_end_token_id": 248054,
|
| 152 |
+
"vision_start_token_id": 248053
|
|
|
|
|
|
|
|
|
|
|
|
|
| 153 |
}
|
model.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b2b257815523a460751cf0fded102ac103ea9c46df8de645eda9bef0bd429bc4
|
| 3 |
+
size 1724708176
|
modeling.py
CHANGED
|
@@ -78,24 +78,17 @@ class OpenThaiSystemOneForDecision(PreTrainedModel):
|
|
| 78 |
|
| 79 |
def __init__(self, config: OpenThaiSystemOneConfig):
|
| 80 |
super().__init__(config)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 81 |
if config.is_vision:
|
| 82 |
-
|
| 83 |
-
from transformers.models.qwen3_5.modeling_qwen3_5 import Qwen3_5Model
|
| 84 |
-
|
| 85 |
-
mm = Qwen3_5Config(
|
| 86 |
-
text_config=config.text_config,
|
| 87 |
-
vision_config=config.vision_config,
|
| 88 |
-
image_token_id=config.image_token_id,
|
| 89 |
-
vision_start_token_id=config.vision_start_token_id,
|
| 90 |
-
vision_end_token_id=config.vision_end_token_id,
|
| 91 |
-
tie_word_embeddings=False,
|
| 92 |
-
)
|
| 93 |
-
self.model = Qwen3_5Model(mm)
|
| 94 |
enable_vision_helper_caches()
|
| 95 |
ph = dict(config.point_head or {})
|
| 96 |
self.point_head = PointHead(config.hidden_size, ph.get("dim", 256), ph.get("n_sa_layers", 1), ph.get("n_heads", 8))
|
| 97 |
else:
|
| 98 |
-
self.
|
| 99 |
self.point_head = None
|
| 100 |
self.slot_head = nn.Linear(config.hidden_size, config.n_slots, bias=config.head_bias)
|
| 101 |
# log-temperatures per question type (choice/score/noul[/point]); learned in the calibration stage
|
|
@@ -104,7 +97,26 @@ class OpenThaiSystemOneForDecision(PreTrainedModel):
|
|
| 104 |
|
| 105 |
@property
|
| 106 |
def language_model(self):
|
| 107 |
-
return self.model
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 108 |
|
| 109 |
def get_input_embeddings(self):
|
| 110 |
return self.language_model.get_input_embeddings()
|
|
@@ -180,7 +192,7 @@ class OpenThaiSystemOneForDecision(PreTrainedModel):
|
|
| 180 |
abstain_slot=text_model.config.abstain_slot,
|
| 181 |
answer_token_id=text_model.config.answer_token_id,
|
| 182 |
head_bias=text_model.config.head_bias,
|
| 183 |
-
n_temperatures=
|
| 184 |
vision_config=vcfg,
|
| 185 |
image_token_id=tok.convert_tokens_to_ids(QWEN_IMAGE_PAD),
|
| 186 |
vision_start_token_id=tok.convert_tokens_to_ids(QWEN_VISION_START),
|
|
@@ -196,7 +208,7 @@ class OpenThaiSystemOneForDecision(PreTrainedModel):
|
|
| 196 |
if len(tok) > old_vocab:
|
| 197 |
text_model.model.resize_token_embeddings(len(tok), mean_resizing=False)
|
| 198 |
_init_new_token_embeddings(text_model.get_input_embeddings().weight, tok, len(tok) - old_vocab)
|
| 199 |
-
missing, unexpected = model.model.
|
| 200 |
assert not unexpected and not missing, (missing, unexpected)
|
| 201 |
model.slot_head.load_state_dict(text_model.slot_head.state_dict())
|
| 202 |
with torch.no_grad():
|
|
@@ -212,9 +224,9 @@ class OpenThaiSystemOneForDecision(PreTrainedModel):
|
|
| 212 |
if k.startswith(prefix):
|
| 213 |
sd[k[len(prefix):]] = fh.get_tensor(k)
|
| 214 |
break
|
| 215 |
-
missing, unexpected = model.
|
| 216 |
assert not unexpected and not missing, (missing, unexpected)
|
| 217 |
-
model.model.config
|
| 218 |
return model, tok
|
| 219 |
|
| 220 |
# ------------------------------------------------------------------ forward
|
|
@@ -300,19 +312,9 @@ class OpenThaiSystemOneForDecision(PreTrainedModel):
|
|
| 300 |
position_ids: Optional[torch.Tensor] = None,
|
| 301 |
**kwargs,
|
| 302 |
) -> DecisionOutput:
|
| 303 |
-
|
| 304 |
-
|
| 305 |
-
|
| 306 |
-
# position_ids (3, B, T) from formatting.mrope_position_ids skips the slow per-token get_rope_index;
|
| 307 |
-
# image_grid_thw may stay on the CPU (no GPU syncs in the python bookkeeping) — helper outputs go to the pixel device
|
| 308 |
-
_VISION_TARGET_DEVICE[0] = pixel_values.device
|
| 309 |
-
out = self.model(
|
| 310 |
-
input_ids=input_ids, attention_mask=attention_mask, pixel_values=pixel_values,
|
| 311 |
-
image_grid_thw=image_grid_thw, mm_token_type_ids=mm_token_type_ids, position_ids=position_ids, **kwargs,
|
| 312 |
-
)
|
| 313 |
-
else:
|
| 314 |
-
out = self.model(input_ids=input_ids, attention_mask=attention_mask, **kwargs)
|
| 315 |
-
hidden = out.last_hidden_state
|
| 316 |
if answer_positions is None:
|
| 317 |
answer_positions = (input_ids == self.config.answer_token_id).nonzero()[:, 1].unsqueeze(0)
|
| 318 |
if option_counts is None:
|
|
@@ -416,6 +418,52 @@ def enable_vision_helper_caches(maxsize: int = 512):
|
|
| 416 |
_VISION_CACHE_ON = True
|
| 417 |
|
| 418 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 419 |
def use_reference_kernels():
|
| 420 |
"""Force the pure-PyTorch Gated-DeltaNet / causal-conv paths.
|
| 421 |
|
|
|
|
| 78 |
|
| 79 |
def __init__(self, config: OpenThaiSystemOneConfig):
|
| 80 |
super().__init__(config)
|
| 81 |
+
# Parameter layout is the SAME as the text-only line for everything they share: text tower `model.*`,
|
| 82 |
+
# `slot_head.*`, 3 `log_temperature`s. The vision parts sit beside it (`visual.*`, `point_head.*`), so a
|
| 83 |
+
# text-only v0.x client loading a -Vision checkpoint finds every weight it knows and ignores the rest.
|
| 84 |
+
self.model = AutoModel.from_config(config.text_config)
|
| 85 |
if config.is_vision:
|
| 86 |
+
self.visual = AutoModel.from_config(config.vision_config)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
enable_vision_helper_caches()
|
| 88 |
ph = dict(config.point_head or {})
|
| 89 |
self.point_head = PointHead(config.hidden_size, ph.get("dim", 256), ph.get("n_sa_layers", 1), ph.get("n_heads", 8))
|
| 90 |
else:
|
| 91 |
+
self.visual = None
|
| 92 |
self.point_head = None
|
| 93 |
self.slot_head = nn.Linear(config.hidden_size, config.n_slots, bias=config.head_bias)
|
| 94 |
# log-temperatures per question type (choice/score/noul[/point]); learned in the calibration stage
|
|
|
|
| 97 |
|
| 98 |
@property
|
| 99 |
def language_model(self):
|
| 100 |
+
return self.model
|
| 101 |
+
|
| 102 |
+
def encode(self, input_ids, attention_mask=None, pixel_values=None, image_grid_thw=None, position_ids=None, **kwargs):
|
| 103 |
+
"""Final hidden states. With images: ViT -> merger -> splice into the <|image_pad|> positions, 3-D (t,h,w) position
|
| 104 |
+
ids (computed here when not given), then the text tower. Without images: the plain text-tower forward."""
|
| 105 |
+
if pixel_values is None:
|
| 106 |
+
return self.model(input_ids=input_ids, attention_mask=attention_mask, position_ids=position_ids, **kwargs).last_hidden_state
|
| 107 |
+
if self.visual is None:
|
| 108 |
+
raise ValueError("this checkpoint is text-only; images need OpenThai-SystemOne-Vision")
|
| 109 |
+
_VISION_TARGET_DEVICE[0] = pixel_values.device
|
| 110 |
+
embeds = self.model.get_input_embeddings()(input_ids)
|
| 111 |
+
img = self.visual(pixel_values.to(self.visual.dtype), grid_thw=image_grid_thw).pooler_output
|
| 112 |
+
mask = (input_ids == self.config.image_token_id).unsqueeze(-1)
|
| 113 |
+
if int(mask.sum()) != img.shape[0]:
|
| 114 |
+
raise ValueError(f"image tokens ({int(mask.sum())}) != image features ({img.shape[0]})")
|
| 115 |
+
embeds = embeds.masked_scatter(mask.to(embeds.device), img.to(embeds.device, embeds.dtype))
|
| 116 |
+
if position_ids is None:
|
| 117 |
+
position_ids = mrope_positions_from_ids(input_ids, attention_mask, image_grid_thw, self.config.image_token_id,
|
| 118 |
+
int(getattr(self.config.vision_config, "spatial_merge_size", 2)))
|
| 119 |
+
return self.model(inputs_embeds=embeds, attention_mask=attention_mask, position_ids=position_ids.to(embeds.device), **kwargs).last_hidden_state
|
| 120 |
|
| 121 |
def get_input_embeddings(self):
|
| 122 |
return self.language_model.get_input_embeddings()
|
|
|
|
| 192 |
abstain_slot=text_model.config.abstain_slot,
|
| 193 |
answer_token_id=text_model.config.answer_token_id,
|
| 194 |
head_bias=text_model.config.head_bias,
|
| 195 |
+
n_temperatures=text_model.config.n_temperatures,
|
| 196 |
vision_config=vcfg,
|
| 197 |
image_token_id=tok.convert_tokens_to_ids(QWEN_IMAGE_PAD),
|
| 198 |
vision_start_token_id=tok.convert_tokens_to_ids(QWEN_VISION_START),
|
|
|
|
| 208 |
if len(tok) > old_vocab:
|
| 209 |
text_model.model.resize_token_embeddings(len(tok), mean_resizing=False)
|
| 210 |
_init_new_token_embeddings(text_model.get_input_embeddings().weight, tok, len(tok) - old_vocab)
|
| 211 |
+
missing, unexpected = model.model.load_state_dict(text_model.model.state_dict(), strict=False)
|
| 212 |
assert not unexpected and not missing, (missing, unexpected)
|
| 213 |
model.slot_head.load_state_dict(text_model.slot_head.state_dict())
|
| 214 |
with torch.no_grad():
|
|
|
|
| 224 |
if k.startswith(prefix):
|
| 225 |
sd[k[len(prefix):]] = fh.get_tensor(k)
|
| 226 |
break
|
| 227 |
+
missing, unexpected = model.visual.load_state_dict({k: v.to(torch_dtype) for k, v in sd.items()}, strict=False)
|
| 228 |
assert not unexpected and not missing, (missing, unexpected)
|
| 229 |
+
model.model.config = cfg.text_config
|
| 230 |
return model, tok
|
| 231 |
|
| 232 |
# ------------------------------------------------------------------ forward
|
|
|
|
| 312 |
position_ids: Optional[torch.Tensor] = None,
|
| 313 |
**kwargs,
|
| 314 |
) -> DecisionOutput:
|
| 315 |
+
# position_ids (3, B, T) from formatting.mrope_position_ids avoid recomputing them; image_grid_thw may stay on the CPU
|
| 316 |
+
hidden = self.encode(input_ids, attention_mask=attention_mask, pixel_values=pixel_values, image_grid_thw=image_grid_thw,
|
| 317 |
+
position_ids=position_ids if pixel_values is not None else None, **kwargs)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 318 |
if answer_positions is None:
|
| 319 |
answer_positions = (input_ids == self.config.answer_token_id).nonzero()[:, 1].unsqueeze(0)
|
| 320 |
if option_counts is None:
|
|
|
|
| 418 |
_VISION_CACHE_ON = True
|
| 419 |
|
| 420 |
|
| 421 |
+
def mrope_positions_from_ids(input_ids, attention_mask, image_grid_thw, image_token_id: int, merge: int):
|
| 422 |
+
"""3-D position ids from token ids alone (same result as formatting.mrope_position_ids / Qwen's get_rope_index)."""
|
| 423 |
+
B, T = input_ids.shape
|
| 424 |
+
grids = [[int(v) for v in g] for g in image_grid_thw.tolist()]
|
| 425 |
+
ids = input_ids.tolist()
|
| 426 |
+
am = attention_mask.tolist() if attention_mask is not None else [[1] * T for _ in range(B)]
|
| 427 |
+
pos = torch.zeros((3, B, T), dtype=torch.long)
|
| 428 |
+
gi = 0
|
| 429 |
+
for b in range(B):
|
| 430 |
+
cur, i = 0, 0
|
| 431 |
+
n = sum(am[b])
|
| 432 |
+
while i < n:
|
| 433 |
+
if ids[b][i] == image_token_id:
|
| 434 |
+
t, h, w = grids[gi]; gi += 1
|
| 435 |
+
hm, wm = h // merge, w // merge
|
| 436 |
+
L = t * hm * wm
|
| 437 |
+
tt = torch.arange(t).view(t, 1, 1).expand(t, hm, wm).reshape(-1)
|
| 438 |
+
hh = torch.arange(hm).view(1, hm, 1).expand(t, hm, wm).reshape(-1)
|
| 439 |
+
ww = torch.arange(wm).view(1, 1, wm).expand(t, hm, wm).reshape(-1)
|
| 440 |
+
pos[0, b, i:i + L] = tt + cur; pos[1, b, i:i + L] = hh + cur; pos[2, b, i:i + L] = ww + cur
|
| 441 |
+
cur += max(hm, wm); i += L
|
| 442 |
+
else:
|
| 443 |
+
j = i
|
| 444 |
+
while j < n and ids[b][j] != image_token_id:
|
| 445 |
+
j += 1
|
| 446 |
+
pos[:, b, i:j] = torch.arange(j - i) + cur
|
| 447 |
+
cur += j - i; i = j
|
| 448 |
+
return pos
|
| 449 |
+
|
| 450 |
+
|
| 451 |
+
def convert_legacy_vision_state_dict(sd: dict) -> dict:
|
| 452 |
+
"""Checkpoints trained before 2026-09-23 stored the vision model as a Qwen3_5Model (`model.language_model.*`,
|
| 453 |
+
`model.visual.*`, 4 temperatures). Map them onto the shared layout (`model.*`, `visual.*`, 3 temperatures)."""
|
| 454 |
+
out = {}
|
| 455 |
+
for k, v in sd.items():
|
| 456 |
+
if k.startswith("model.language_model."):
|
| 457 |
+
out["model." + k[len("model.language_model."):]] = v
|
| 458 |
+
elif k.startswith("model.visual."):
|
| 459 |
+
out["visual." + k[len("model.visual."):]] = v
|
| 460 |
+
elif k == "log_temperature" and v.numel() == 4:
|
| 461 |
+
out[k] = v[:3].clone()
|
| 462 |
+
else:
|
| 463 |
+
out[k] = v
|
| 464 |
+
return out
|
| 465 |
+
|
| 466 |
+
|
| 467 |
def use_reference_kernels():
|
| 468 |
"""Force the pure-PyTorch Gated-DeltaNet / causal-conv paths.
|
| 469 |
|