kobkrit commited on
Commit
f980f3f
Β·
verified Β·
1 Parent(s): 9322954

v1.4: icon grounding, not-on-screen refusals, symbol names; Windows +2.5

Browse files
Files changed (2) hide show
  1. README.md +86 -74
  2. model.safetensors +1 -1
README.md CHANGED
@@ -6,13 +6,15 @@ tags: [system-one, decision-model, gui-grounding, computer-use, voice-control, t
6
  base_model: Qwen/Qwen3.5-0.8B-Base
7
  ---
8
 
9
- # OpenThai-SystemOne-Vision (v1.3)
10
 
11
  The vision variant of [OpenThai-SystemOne](https://huggingface.co/iapp/OpenThai-SystemOne): the same open Thai + English
12
  System One decision model, now with **images in the state** and a fourth question type, **`point`**, that returns where
13
  on a screenshot something is. Screenshot in, click out, in one forward pass, no text generation.
14
 
15
- **v1.3 (2026-09-26)** clicks better on real Windows and Mac apps (trained on live Windows 11 and macOS screens) and on
 
 
16
  large screens, and adds `point_refine`;
17
  **v1.1 (2026-09-24)** added Thai **voice-command understanding** for hands-free computer control (a spoken Thai command +
18
  the app and window title in, the operation and its target out) and stronger desktop and Doom play. See [Versions](#versions).
@@ -70,41 +72,43 @@ state = 'ΰΈ„ΰΈ³ΰΈͺΰΈ±ΰΉˆΰΈ‡ΰΉ€ΰΈͺΰΈ΅ΰΈ’ΰΈ‡ΰΈ ΰΈ²ΰΈ©ΰΈ²ΰΉ„ΰΈ—ΰΈ’ΰΈˆΰΈ²ΰΈΰΈœΰΈΉ
70
 
71
  ## Evaluation
72
 
73
- v1.3 next to v1.1 (the previous published version), same scripts; single option order unless noted; accuracy in %.
74
- "default" = the plain API call (screenshot at ≀ 1280Γ—800); "+ refine" = screenshot at 2 MP (`"detail": "high"`) with
75
- `point_refine` over the top-3 first-pass candidates, landscape screens only: zoom 50 % on ScreenSpot-v2 / Thai GUI / OS-Atlas,
76
- 35 % on Windows / Mac live and ScreenSpot-Pro. On desktops and large screens, send `"detail": "high"` and `point_refine`.
77
-
78
- ### Live desktops (new): real apps, click targets from the accessibility tree
79
- | benchmark | v1.1 default | **v1.3 default** | **v1.3 + refine** | Qwen3-VL-2B | Qwen3-VL-4B |
80
- |---|---|---|---|---|---|
81
- | **Windows live** (319 targets, 28 screens of Windows 11: Settings, File Explorer, Control Panel, Notepad, Paint, Calculator, Store, Edge, …) | 63.3 | **68.3** | **69.3** | 67.1 | 73.0 |
82
- | Β· Windows Settings pages (114) | 64.9 | 69.3 | – | 66.7 | 68.4 |
83
- | **Mac live** (358 targets, 29 screens: System Settings, Finder, Safari, Office, iWork, …) | 65.4 | 63.7 | 67.6 | 67.0 | 67.9 |
84
- | Β· label-checked (340: 18 targets found unanswerable removed) | 68.5 | 66.8 | **71.2** | 70.6 | 71.5 |
85
- | latency per click (H100, eval harness) | ~70 ms | ~70 ms | 0.1–0.25 s | ~0.9 s | ~0.75 s |
86
-
87
- The Windows and Mac live screens are never trained on. v1.3's training data does include *other* screens of the same apps
88
- (for example other Settings pages), captured the same way. Weak spots: Windows Calculator (60; its buttons are named with
89
- words such as "Two" and "Multiply by", Qwen3-VL-4B 95), icon-only tiles, and small text links.
 
90
 
91
  ### GUI grounding benchmarks (`point`: predicted click inside the target box)
92
- | benchmark | v1.1 default | **v1.3 default** | **v1.3 + refine** | Qwen3-VL-2B | Qwen3-VL-4B |
93
- |---|---|---|---|---|---|
94
- | ScreenSpot-v2, all (1,272) | 80.7 | **82.2** | **85.2** | 85.8 | 90.4 |
95
- | Β· desktop text / icon | 89.7 / 74.3 | 90.2 / 75.7 | 91.8 / 83.6 | 93.8 / 82.9 | 96.9 / 85.7 |
96
- | Β· mobile text / icon | 94.5 / 70.6 | 94.5 / 71.6 | 93.8 / 74.9 | 94.1 / 82.0 | 96.9 / 83.4 |
97
- | Β· web text / icon | 85.0 / 62.1 | 84.2 / 70.0 | 88.5 / 74.9 | 85.9 / 72.4 | 91.9 / 83.7 |
98
- | ScreenSpot-Pro (1,581, 26 pro apps at up to 4K: Windows / macOS / Linux) | 9.8* | 9.0 | **32.9** (29.3 / 38.7 / 28.0) | 43.4* | 54.8* |
99
- | Desktop (OS-Atlas macOS / Windows / Linux, held-out) | 50.6 | **53.7** | 59.8 (600) | 63.8 | 65.6 |
100
- | Desktop "not on screen" abstention (26) | 80.8 | **92.3** | – | – | – |
101
- | Thai GUI held-out domains, `point` | 78.0 | 77.8 | 76.7 (600) | 72.4 | 80.4 |
102
- | Thai GUI set-of-marks `choice` / `operation` | 93.5 / 99.6 | 93.1 / 99.6 | – | – | – |
103
- | Thai screen QA `noul` / `choice` | 93.9 / 84.8 | 93.1 / 85.9 | – | – | – |
104
- | Thai GUI "not on screen" abstention | 46.4 | 48.0 | – | – | – |
105
 
106
  Qwen3-VL numbers: our runs with the same harness (screenshot + instruction, click inside the box); * every 2nd ScreenSpot-Pro
107
- item (790).
 
108
 
109
  ### Better clicks on large screens: `point_refine`
110
  ```python
@@ -114,63 +118,68 @@ r = c.system_one("goal: …", {"where": {"type": "point", "instructions": "ΰΈ›
114
  ```
115
  The first pass points on the whole screen; the client then re-points on a crop around each of the top candidates at native
116
  detail and keeps the most confident. It only zooms landscape screens larger than the pixel budget (zooming phone screenshots
117
- hurts). ScreenSpot-Pro 9.0 β†’ 32.9, ScreenSpot-v2 82.2 β†’ 85.2, Mac live 66.8 β†’ 71.2 (checked). Cost: 1 + `cands` forward passes.
 
118
 
119
  ### Thai voice commands (operation + slots from one spoken sentence)
120
- | set | v1.0 | v1.1 | **v1.3** |
121
  |---|---|---|---|
122
- | real commands spoken in use, operation (183) | 65.0 | 88.5 | **89.6** |
123
- | Β· slots (site / app / key / Nth) when the operation is right | 90.7 | 100.0 | **100.0** |
124
- | Β· chit-chat between commands acted on (41), lower is better | 14.6 | 7.3 | 7.3 |
125
- | real off-command conversation picked up by the microphone, acted on (227), lower is better | 3.5 | 2.6 | 4.4 |
126
- | synthetic voice commands, held out (1,406 questions) | 61.2 | 90.5 | **92.0** |
127
- | synthetic hard cases (fragments, site vs app, ordinals, 446 questions) | 55.0 | 90.8 | **93.3** |
128
-
129
- ### Doom from pixels (`demo/doom_demo.py`, defend_the_center, skill 3, 40 episodes, seed 999)
130
- | input | v1.0 | v1.1 | **v1.3** | ms / decision |
131
  |---|---|---|---|---|
132
- | two 320Γ—240 frames only | 5.5 (20 eps) | 11.8 | **13.0** (1–23) | 49 |
 
 
133
  | the privileged scripted expert it imitates | 6.8 | 6.8 | 6.8 | – |
134
 
135
- Offline Doom decisions (600 frames): move 88.2, attack 97.0, in-danger 96.3, nearest-enemy point 51.1, enemy-absent 96.6.
 
 
136
 
137
  ### Text-only tasks
138
  Public 13-subset benchmark (the Bespoke / Jev public suite; `noul`/`choice` = accuracy, `score` = exact level):
139
 
140
- | subset | v0.3 | v1.1 | **v1.3** | Jev 1.13.0* | Nimble-9B* |
141
  |---|---|---|---|---|---|
142
- | aegis2 | 83.2 | 82.0 | 82.0 | 80.4 | 81.2 |
143
- | boolq | 79.7 | 79.7 | 79.7 | 89.7 | 86.0 |
144
- | civil_comments | 79.0 | 86.0 | **86.3** | 81.0 | 70.3 |
145
- | helpsteer2 (score) | 41.6 | 42.0 | 40.4 | 34.1 | 39.0 |
146
- | massive-de-DE | 88.3 | 89.4 | 89.7 | 86.9 | 83.4 |
147
- | massive-en-US | 88.3 | 89.4 | **90.3** | 87.4 | 86.9 |
148
- | multinli | 89.0 | 89.0 | 88.3 | 82.9 | 85.3 |
149
  | paws | 94.0 | 94.0 | 94.0 | 89.2 | 82.8 |
150
- | pubmedqa | 64.0 | 67.6 | 67.2 | 77.2 | 75.6 |
151
- | squad2 | 89.3 | 89.0 | 88.6 | 82.9 | 80.6 |
152
- | summeval-consistency (score) | 75.0 | 75.0 | 75.0 | 81.2 | 75.7 |
153
- | summeval-relevance (score) | 21.7 | 21.2 | 20.4 | 35.0 | 49.2 |
154
- | vitaminc-dev | 72.5 | 71.3 | 71.0 | 80.1 | 76.6 |
155
- | **macro, 13 subsets** | 74.3 | 75.0 | 74.8 | 76.0 | 74.8 |
156
- | macro, 10 accuracy subsets | 82.7 | 83.7 | 83.7 | 83.8 | 80.9 |
157
 
158
  \* published by Bespoke Labs for their own harness on the same subsets (Jev via the TypeSafe API).
159
 
160
  Thai held-out and eval-split sets (`th` unless noted; banking77 and xLAM are English):
161
 
162
- | set | v0.3 | v1.1 | **v1.3** | v1.3, 8 orders |
163
  |---|---|---|---|---|
164
- | banking77 (77-way, whole dataset held out) | 45.4 | 62.0 | 62.0 | 64.6 |
165
- | contrastive pairs, one-fact flips | 80.7 | 82.1 | 82.1 | 80.4 |
166
- | MASSIVE-th intent | 90.0 | 91.1 | 91.1 | 91.9 |
167
- | Prachathai topics | 98.1 | 98.7 | 98.6 | 98.7 |
168
- | SIB-200 Thai topic (whole dataset held out) | 77.9 | 75.5 | 75.5 | 77.0 |
169
- | Wisesight sentiment (whole dataset held out) | 51.6 | 52.8 | 52.5 | 51.9 |
170
- | Wongnai stars (score, exact) | 63.5 | 63.1 | 63.3 | 63.3 |
171
- | xLAM tool selection | 99.4 | 99.3 | 99.4 | 99.3 |
172
- | XNLI-th | 77.1 | 75.6 | 75.5 | 75.6 |
173
- | **macro, 8 accuracy sets** | 77.5 | 79.6 | 79.6 | 79.9 |
174
 
175
  ### Latency (batch 1, H100, end-to-end `SystemOneClient`)
176
  | request | tokens | ms |
@@ -184,7 +193,8 @@ Thai held-out and eval-split sets (`th` unless noted; banking77 and xLAM are Eng
184
 
185
  | version | date | change |
186
  |---|---|---|
187
- | **v1.3** | 2026-09-26 | + live desktop grounding: 665 records from 319 live Windows 11 screens (UI Automation targets; Settings, Control Panel, dialogs, shell folders, Calculator modes, Store, Edge) and 386 records from 116 live macOS screens (accessibility targets; System Settings, Finder, utilities, Safari, Word / Excel / PowerPoint / Pages ribbons), with account names blacked out; on top of an internal v1.2 round (ShowUI desktop + web, 40k zoomed crops, 75k xlangai/Jedi records, 2 MP training, averaged with v1.1). Client / API `point_refine`. Windows live 63.3 β†’ 68.3, ScreenSpot-v2 80.7 β†’ 82.2 (85.2 with refine), ScreenSpot-Pro ~10 β†’ 32.9 with refine, OS-Atlas desktop 50.6 β†’ 53.7, Doom 11.8 β†’ 13.0, text and Thai unchanged. |
 
188
  | v1.1 | 2026-09-24 | + Thai voice-command intents (28k synthetic training records from 21k distinct commands + 8.9k hard-case records: fragments / chit-chat = NONE, site vs app, ordinal + topic, ASR spelling variants), desktop grounding round (OS-Atlas macOS / Windows / Linux), text-anchor round (civil_comments, VitaminC); temperatures recalibrated. Voice 65 β†’ 88.5, desktop 39 β†’ 51, Doom pixels 5.5 β†’ 11.9 kills, ScreenSpot-v2 82.7 β†’ 80.7. |
189
  | v1.0 | 2026-09-23 | first vision release (Thai GUI crawl, public grounding, Doom rollouts, screen QA on the v0.3 text tower) |
190
 
@@ -206,7 +216,9 @@ Thai held-out and eval-split sets (`th` unless noted; banking77 and xLAM are Eng
206
  6. v1.3: one round from v1.2 with live Windows 11 and macOS screens captured through the accessibility APIs (a small
207
  agent inside a Windows VM; the macOS accessibility tree), other apps / pages than the live eval screens, personal names
208
  blacked out, rest of the mix rehearsed.
209
- 7. Calibration of the per-type temperatures.
 
 
210
 
211
  ## License & credits
212
 
 
6
  base_model: Qwen/Qwen3.5-0.8B-Base
7
  ---
8
 
9
+ # OpenThai-SystemOne-Vision (v1.4)
10
 
11
  The vision variant of [OpenThai-SystemOne](https://huggingface.co/iapp/OpenThai-SystemOne): the same open Thai + English
12
  System One decision model, now with **images in the state** and a fourth question type, **`point`**, that returns where
13
  on a screenshot something is. Screenshot in, click out, in one forward pass, no text generation.
14
 
15
+ **v1.4 (2026-09-28)** clicks better on Windows apps (Settings, Calculator) and on icons, and understands voice commands a little
16
+ better; it plays Doom from pixels worse than v1.3 (see Evaluation).
17
+ **v1.3 (2026-09-26)** clicked better on real Windows and Mac apps (trained on live Windows 11 and macOS screens) and on
18
  large screens, and adds `point_refine`;
19
  **v1.1 (2026-09-24)** added Thai **voice-command understanding** for hands-free computer control (a spoken Thai command +
20
  the app and window title in, the operation and its target out) and stronger desktop and Doom play. See [Versions](#versions).
 
72
 
73
  ## Evaluation
74
 
75
+ v1.4 next to v1.3, same scripts; single option order unless noted; accuracy in %. "default" = the plain API call (screenshot at
76
+ ≀ 1280Γ—800); "+ refine" = screenshot at 2 MP (`"detail": "high"`) with `point_refine` over the top-3 first-pass candidates,
77
+ landscape screens only (zoom 50 % on ScreenSpot-v2 / Thai GUI / OS-Atlas, 35 % on Windows / Mac live and ScreenSpot-Pro).
78
+ On desktops and large screens, send `"detail": "high"` and `point_refine`.
79
+
80
+ ### Live desktops: real apps, click targets from the accessibility tree
81
+ | benchmark | v1.3 default | v1.3 + refine | **v1.4 default** | **v1.4 + refine** | Qwen3-VL-2B | Qwen3-VL-4B |
82
+ |---|---|---|---|---|---|---|
83
+ | **Windows live** (319 targets, 28 screens of Windows 11: Settings, File Explorer, Control Panel, Notepad, Paint, Calculator, Store, Edge, …) | 68.3 | 69.3 | **70.8** | 70.5 | 67.1 | 73.0 |
84
+ | Β· Windows Settings pages (114) | 69.3 | – | **74.6** | – | 66.7 | 68.4 |
85
+ | Β· Windows Calculator (20)† | 60 | – | **80** | – | 80 | 95 |
86
+ | **Mac live** (358 targets, 29 screens: System Settings, Finder, Safari, Office, iWork, …) | 63.7 | 67.6 | 64.5 | 67.3 | 67.0 | 67.9 |
87
+ | Β· label-checked (340: 18 targets found unanswerable removed) | 66.8 | **71.2** | 67.6 | 70.9 | 70.6 | 71.5 |
88
+ | latency per click (H100, eval harness) | ~70 ms | 0.1–0.25 s | ~70 ms | 0.1–0.25 s | ~0.9 s | ~0.75 s |
89
+
90
+ The Windows and Mac live screens are never trained on; the training data includes *other* screens of the same apps.
91
+ † v1.4's synthetic calculators use Windows' standard accessible names ("Two", "Multiply by"), which is how the Windows live
92
+ Calculator targets are named, so part of that gain is learning the naming convention.
93
 
94
  ### GUI grounding benchmarks (`point`: predicted click inside the target box)
95
+ | benchmark | v1.3 default | v1.3 + refine | **v1.4 default** | **v1.4 + refine** | Qwen3-VL-2B | Qwen3-VL-4B |
96
+ |---|---|---|---|---|---|---|
97
+ | ScreenSpot-v2, all (1,272) | 82.2 | 85.2 | **82.8** | **85.9** | 85.8 | 90.4 |
98
+ | Β· desktop text / icon | 90.2 / 75.7 | 91.8 / 83.6 | 89.2 / 77.1 | 91.2 / 85.0 | 93.8 / 82.9 | 96.9 / 85.7 |
99
+ | Β· mobile text / icon | 94.5 / 71.6 | 93.8 / 74.9 | 94.8 / 71.6 | 94.5 / 75.4 | 94.1 / 82.0 | 96.9 / 83.4 |
100
+ | Β· web text / icon | 84.2 / 70.0 | 88.5 / 74.9 | 85.9 / 71.4 | 89.3 / 76.4 | 85.9 / 72.4 | 91.9 / 83.7 |
101
+ | ScreenSpot-Pro (1,581, 26 pro apps up to 4K; Windows / macOS / Linux) | 9.0 | 32.9 (29.3 / 38.7 / 28.0) | 8.7 | 32.8 (29.2 / 38.9 / 26.0) | 43.4* | 54.8* |
102
+ | Desktop (OS-Atlas macOS / Windows / Linux, held-out) | 53.7 | 59.8 (600) | 53.8 | 59.3 (600) | 63.8 | 65.6 |
103
+ | Desktop "not on screen" abstention (26) | 92.3 | – | 92.3 | – | – | – |
104
+ | Thai GUI held-out domains, `point` (1,611) | 77.8 | 76.7 (600) | **78.6** | 74.9 (600) | 72.4 | 80.4 |
105
+ | Thai GUI set-of-marks `choice` / `operation` | 93.1 / 99.6 | – | 92.6 / 99.8 | – | – | – |
106
+ | Thai screen QA `noul` / `choice` | 93.1 / 85.9 | – | 94.7 / 85.0 | – | – | – |
107
+ | Thai GUI "not on screen" abstention (179) | 48.0 | – | **49.7** | – | – | – |
108
 
109
  Qwen3-VL numbers: our runs with the same harness (screenshot + instruction, click inside the box); * every 2nd ScreenSpot-Pro
110
+ item (790). Mac live and the 600-item Thai GUI / OS-Atlas subsets move by Β±1.5 points between settings; smaller differences
111
+ are noise.
112
 
113
  ### Better clicks on large screens: `point_refine`
114
  ```python
 
118
  ```
119
  The first pass points on the whole screen; the client then re-points on a crop around each of the top candidates at native
120
  detail and keeps the most confident. It only zooms landscape screens larger than the pixel budget (zooming phone screenshots
121
+ hurts). v1.4: ScreenSpot-Pro 8.7 β†’ 32.8, ScreenSpot-v2 82.8 β†’ 85.9, Mac live 67.6 β†’ 70.9 (checked). Windows windows usually fit
122
+ the 2 MP budget, so there it mostly adds the higher resolution. Cost: 1 + `cands` forward passes.
123
 
124
  ### Thai voice commands (operation + slots from one spoken sentence)
125
+ | set | v1.1 | v1.3 | **v1.4** |
126
  |---|---|---|---|
127
+ | real commands spoken in use, operation (183) | 88.5 | 89.6 | **90.7** |
128
+ | Β· slots (site / app / key / Nth) when the operation is right | 100.0 | 100.0 | 100.0 |
129
+ | Β· chit-chat between commands acted on (41), lower is better | 7.3 | 7.3 | 7.3 |
130
+ | real off-command conversation picked up by the microphone, acted on (227), lower is better | 2.6 | 4.4 | 4.0 |
131
+ | synthetic voice commands, held out (1,406 questions) | 90.5 | 92.0 | 91.8 |
132
+ | synthetic hard cases (fragments, site vs app, ordinals, 446 questions) | 90.8 | 93.3 | 92.6 |
133
+
134
+ ### Doom from pixels (`demo/doom_demo.py`, defend_the_center, skill 3, two 320Γ—240 frames, 40 episodes per seed)
135
+ | seed | v1.1 | v1.3 | **v1.4** | ms / decision |
136
  |---|---|---|---|---|
137
+ | 999 | 11.8 | 13.0 | 11.4 | ~50 |
138
+ | 777 | – | 11.9 | 9.7 | ~50 |
139
+ | mean of both (80 episodes) | – | **12.5** | 10.6 | |
140
  | the privileged scripted expert it imitates | 6.8 | 6.8 | 6.8 | – |
141
 
142
+ **Known regression:** v1.4 plays Doom from pixels worse than v1.3 (βˆ’1.9 kills per episode). Use v1.3 (`revision="v1.3"`) for the
143
+ Doom demo. Offline Doom decisions (600 frames), v1.4: move 88.5, attack 97.2, in-danger 96.7, nearest-enemy point 48.9,
144
+ enemy-absent 98.3.
145
 
146
  ### Text-only tasks
147
  Public 13-subset benchmark (the Bespoke / Jev public suite; `noul`/`choice` = accuracy, `score` = exact level):
148
 
149
+ | subset | v0.3 | v1.3 | **v1.4** | Jev 1.13.0* | Nimble-9B* |
150
  |---|---|---|---|---|---|
151
+ | aegis2 | 83.2 | 82.0 | 81.2 | 80.4 | 81.2 |
152
+ | boolq | 79.7 | 79.7 | 79.0 | 89.7 | 86.0 |
153
+ | civil_comments | 79.0 | 86.3 | 86.3 | 81.0 | 70.3 |
154
+ | helpsteer2 (score) | 41.6 | 40.4 | 40.4 | 34.1 | 39.0 |
155
+ | massive-de-DE | 88.3 | 89.7 | 90.0 | 86.9 | 83.4 |
156
+ | massive-en-US | 88.3 | 90.3 | 90.3 | 87.4 | 86.9 |
157
+ | multinli | 89.0 | 88.3 | 89.0 | 82.9 | 85.3 |
158
  | paws | 94.0 | 94.0 | 94.0 | 89.2 | 82.8 |
159
+ | pubmedqa | 64.0 | 67.2 | 67.2 | 77.2 | 75.6 |
160
+ | squad2 | 89.3 | 88.6 | 88.3 | 82.9 | 80.6 |
161
+ | summeval-consistency (score) | 75.0 | 75.0 | 75.7 | 81.2 | 75.7 |
162
+ | summeval-relevance (score) | 21.7 | 20.4 | 20.4 | 35.0 | 49.2 |
163
+ | vitaminc-dev | 72.5 | 71.0 | 70.8 | 80.1 | 76.6 |
164
+ | **macro, 13 subsets** | 74.3 | 74.8 | 74.8 | 76.0 | 74.8 |
165
+ | macro, 10 accuracy subsets | 82.7 | 83.7 | 83.6 | 83.8 | 80.9 |
166
 
167
  \* published by Bespoke Labs for their own harness on the same subsets (Jev via the TypeSafe API).
168
 
169
  Thai held-out and eval-split sets (`th` unless noted; banking77 and xLAM are English):
170
 
171
+ | set | v0.3 | v1.3 | **v1.4** | v1.4, 8 orders |
172
  |---|---|---|---|---|
173
+ | banking77 (77-way, whole dataset held out) | 45.4 | 62.0 | 62.1 | 64.4 |
174
+ | contrastive pairs, one-fact flips | 80.7 | 82.1 | 81.8 | 80.1 |
175
+ | MASSIVE-th intent | 90.0 | 91.1 | 91.2 | 92.0 |
176
+ | Prachathai topics | 98.1 | 98.6 | 98.6 | 98.7 |
177
+ | SIB-200 Thai topic (whole dataset held out) | 77.9 | 75.5 | 75.0 | 77.0 |
178
+ | Wisesight sentiment (whole dataset held out) | 51.6 | 52.5 | 52.2 | 51.7 |
179
+ | Wongnai stars (score, exact) | 63.5 | 63.3 | 62.9 | 62.9 |
180
+ | xLAM tool selection | 99.4 | 99.4 | 99.4 | 99.3 |
181
+ | XNLI-th | 77.1 | 75.5 | 75.6 | 75.7 |
182
+ | **macro, 8 accuracy sets** | 77.5 | 79.6 | 79.5 | 79.9 |
183
 
184
  ### Latency (batch 1, H100, end-to-end `SystemOneClient`)
185
  | request | tokens | ms |
 
193
 
194
  | version | date | change |
195
  |---|---|---|
196
+ | **v1.4** | 2026-09-28 | + icon grounding and "not on screen" refusals from xlangai/Jedi (20k icon snaps, 20k refusal rows), 12k synthetic calculators / title bars / toolbars / media controls labelled with their standard accessible names (English + Thai), Mac and Thai GUI rehearsal up; from v1.3. Windows live 68.3 β†’ 70.8 (Settings 69.3 β†’ 74.6, Calculator 60 β†’ 80), ScreenSpot-v2 82.2 β†’ 82.8 (85.9 with refine), Thai not-on-screen 48.0 β†’ 49.7, voice 89.6 β†’ 90.7; **Doom from pixels 12.5 β†’ 10.6 kills**; text, Thai, Mac live, ScreenSpot-Pro, OS-Atlas unchanged within noise. |
197
+ | v1.3 | 2026-09-26 | + live desktop grounding: 665 records from 319 live Windows 11 screens (UI Automation targets; Settings, Control Panel, dialogs, shell folders, Calculator modes, Store, Edge) and 386 records from 116 live macOS screens (accessibility targets; System Settings, Finder, utilities, Safari, Word / Excel / PowerPoint / Pages ribbons), with account names blacked out; on top of an internal v1.2 round (ShowUI desktop + web, 40k zoomed crops, 75k xlangai/Jedi records, 2 MP training, averaged with v1.1). Client / API `point_refine`. Windows live 63.3 β†’ 68.3, ScreenSpot-v2 80.7 β†’ 82.2 (85.2 with refine), ScreenSpot-Pro ~10 β†’ 32.9 with refine, OS-Atlas desktop 50.6 β†’ 53.7, Doom 11.8 β†’ 13.0, text and Thai unchanged. |
198
  | v1.1 | 2026-09-24 | + Thai voice-command intents (28k synthetic training records from 21k distinct commands + 8.9k hard-case records: fragments / chit-chat = NONE, site vs app, ordinal + topic, ASR spelling variants), desktop grounding round (OS-Atlas macOS / Windows / Linux), text-anchor round (civil_comments, VitaminC); temperatures recalibrated. Voice 65 β†’ 88.5, desktop 39 β†’ 51, Doom pixels 5.5 β†’ 11.9 kills, ScreenSpot-v2 82.7 β†’ 80.7. |
199
  | v1.0 | 2026-09-23 | first vision release (Thai GUI crawl, public grounding, Doom rollouts, screen QA on the v0.3 text tower) |
200
 
 
216
  6. v1.3: one round from v1.2 with live Windows 11 and macOS screens captured through the accessibility APIs (a small
217
  agent inside a Windows VM; the macOS accessibility tree), other apps / pages than the live eval screens, personal names
218
  blacked out, rest of the mix rehearsed.
219
+ 7. v1.4: one round from v1.3 with icon grounding / refusal data from xlangai/Jedi and synthetic symbol screens, with the Mac and
220
+ Thai GUI data weighted up so they do not slip.
221
+ 8. Calibration of the per-type temperatures.
222
 
223
  ## License & credits
224
 
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c302e4660e4a6af3135309740a59ad1819b9483b1a436487e4bd8fdafe6a5217
3
  size 1724708176
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c46884e5006cc0133ff1959377984f5f4a335900d20d2b318eeedac0864418bb
3
  size 1724708176