v1.3: live Windows 11 + macOS grounding, point_refine
Browse files- README.md +101 -66
- model.safetensors +1 -1
- types.py +3 -0
README.md
CHANGED
|
@@ -6,13 +6,15 @@ tags: [system-one, decision-model, gui-grounding, computer-use, voice-control, t
|
|
| 6 |
base_model: Qwen/Qwen3.5-0.8B-Base
|
| 7 |
---
|
| 8 |
|
| 9 |
-
# OpenThai-SystemOne-Vision (v1.
|
| 10 |
|
| 11 |
The vision variant of [OpenThai-SystemOne](https://huggingface.co/iapp/OpenThai-SystemOne): the same open Thai + English
|
| 12 |
System One decision model, now with **images in the state** and a fourth question type, **`point`**, that returns where
|
| 13 |
on a screenshot something is. Screenshot in, click out, in one forward pass, no text generation.
|
| 14 |
|
| 15 |
-
**v1.
|
|
|
|
|
|
|
| 16 |
the app and window title in, the operation and its target out) and stronger desktop and Doom play. See [Versions](#versions).
|
| 17 |
|
| 18 |
| | OpenThai-SystemOne (text) | OpenThai-SystemOne-Vision |
|
|
@@ -68,96 +70,122 @@ state = 'คำสั่งเสียงภาษาไทยจากผู
|
|
| 68 |
|
| 69 |
## Evaluation
|
| 70 |
|
| 71 |
-
|
| 72 |
-
|
|
|
|
|
|
|
| 73 |
|
| 74 |
-
###
|
| 75 |
-
| benchmark | v1.
|
| 76 |
-
|---|---|---|---|
|
| 77 |
-
|
|
| 78 |
-
| ·
|
| 79 |
-
|
|
| 80 |
-
| ·
|
| 81 |
-
|
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 88 |
|
| 89 |
### Thai voice commands (operation + slots from one spoken sentence)
|
| 90 |
-
| set | v1.0 | **v1.
|
| 91 |
-
|---|---|---|---|
|
| 92 |
-
| real commands spoken in use, operation (183) | 65.0 | **88.5** | hand-labelled from a live macOS voice-control session |
|
| 93 |
-
| · slots (site / app / key / Nth) when the operation is right | 90.7 | **100.0** | |
|
| 94 |
-
| · chit-chat between commands acted on (41) | 14.6 | **7.3** | lower is better |
|
| 95 |
-
| real off-command conversation picked up by the microphone, acted on (227) | 3.5 | **2.6** | lower is better; before the controller's safety checks |
|
| 96 |
-
| synthetic voice commands, held out (1,406 questions) | 61.2 | **90.5** | |
|
| 97 |
-
| synthetic hard cases (fragments, site vs app, ordinals, 446 questions) | 55.0 | **90.8** | |
|
| 98 |
-
|
| 99 |
-
### Doom from pixels (`demo/doom_demo.py`, defend_the_center, skill 3, 20 episodes, seed 1234)
|
| 100 |
-
| input | v1.0 kills / episode | **v1.1 kills / episode** | ms / decision (H100) |
|
| 101 |
|---|---|---|---|
|
| 102 |
-
|
|
| 103 |
-
|
|
| 104 |
-
|
|
| 105 |
-
| the
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
|
| 107 |
-
Offline Doom decisions (600 frames): move
|
| 108 |
|
| 109 |
### Text-only tasks
|
| 110 |
Public 13-subset benchmark (the Bespoke / Jev public suite; `noul`/`choice` = accuracy, `score` = exact level):
|
| 111 |
|
| 112 |
-
| subset | v0.3 | v1.
|
| 113 |
|---|---|---|---|---|---|
|
| 114 |
-
| aegis2 | 83.2 |
|
| 115 |
-
| boolq | 79.7 |
|
| 116 |
-
| civil_comments | 79.0 |
|
| 117 |
-
| helpsteer2 (score) | 41.6 |
|
| 118 |
-
| massive-de-DE | 88.3 | 89.4 | 89.
|
| 119 |
-
| massive-en-US | 88.3 |
|
| 120 |
-
| multinli | 89.0 |
|
| 121 |
-
| paws | 94.0 |
|
| 122 |
-
| pubmedqa | 64.0 |
|
| 123 |
-
| squad2 | 89.3 | 89.
|
| 124 |
-
| summeval-consistency (score) | 75.0 |
|
| 125 |
-
| summeval-relevance (score) | 21.7 |
|
| 126 |
-
| vitaminc-dev | 72.5 |
|
| 127 |
-
| **macro, 13 subsets** | 74.3 |
|
| 128 |
-
| macro, 10 accuracy subsets | 82.7 |
|
| 129 |
|
| 130 |
\* published by Bespoke Labs for their own harness on the same subsets (Jev via the TypeSafe API).
|
| 131 |
-
With 8 option orders averaged: v1.1 macro 75.2 (10 accuracy subsets 84.0).
|
| 132 |
|
| 133 |
Thai held-out and eval-split sets (`th` unless noted; banking77 and xLAM are English):
|
| 134 |
|
| 135 |
-
| set | v0.3 | v1.
|
| 136 |
|---|---|---|---|---|
|
| 137 |
-
| banking77 (77-way, whole dataset held out) | 45.4 |
|
| 138 |
-
| contrastive pairs, one-fact flips | 80.7 |
|
| 139 |
-
| MASSIVE-th intent | 90.0 |
|
| 140 |
-
| Prachathai topics | 98.1 | 98.7 | 98.
|
| 141 |
-
| SIB-200 Thai topic (whole dataset held out) | 77.9 |
|
| 142 |
-
| Wisesight sentiment (whole dataset held out) | 51.6 | 52.
|
| 143 |
-
| Wongnai stars (score, exact) | 63.5 | 63.1 | 63.
|
| 144 |
-
| xLAM tool selection | 99.4 | 99.
|
| 145 |
-
| XNLI-th | 77.1 |
|
| 146 |
-
| **macro, 8 accuracy sets** | 77.5 | 79.
|
| 147 |
|
| 148 |
### Latency (batch 1, H100, end-to-end `SystemOneClient`)
|
| 149 |
| request | tokens | ms |
|
| 150 |
|---|---|---|
|
| 151 |
| two 320×240 game frames + short state, 2 questions | 298 (160 visual) | 50 |
|
| 152 |
| 1280×800 screenshot + goal, `point` + `choice` | 1,062 (1,000 visual) | 52 |
|
|
|
|
| 153 |
| text only (3-question Thai ticket) | 166 | ~40 |
|
| 154 |
-
| voice command intent (6 questions, up to 250 app names) | ~2,000 | 100–250 round trip from a Mac to a remote GPU, incl. HTTP |
|
| 155 |
|
| 156 |
## Versions
|
| 157 |
|
| 158 |
| version | date | change |
|
| 159 |
|---|---|---|
|
| 160 |
-
| **v1.
|
|
|
|
| 161 |
| v1.0 | 2026-09-23 | first vision release (Thai GUI crawl, public grounding, Doom rollouts, screen QA on the v0.3 text tower) |
|
| 162 |
|
| 163 |
## How it was built
|
|
@@ -172,9 +200,16 @@ Thai held-out and eval-split sets (`th` unless noted; banking77 and xLAM are Eng
|
|
| 172 |
4. v1.1: a text-anchor round and a desktop round averaged into one checkpoint, then a voice-command round
|
| 173 |
(label-conditioned generation by a teacher LLM, every sentence blind re-labelled and kept only on agreement) and a
|
| 174 |
hard-case round aimed at the mistakes seen in real use; the rest of the mix is rehearsed at the same time.
|
| 175 |
-
5.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 176 |
|
| 177 |
## License & credits
|
| 178 |
|
| 179 |
Apache-2.0. Built by iApp Technology / OpenThaiGPT on Qwen3.5-0.8B-Base. GUI grounding recipe after GUI-Actor; System One
|
| 180 |
-
contract after TypeSafe AI's Jev.
|
|
|
|
|
|
| 6 |
base_model: Qwen/Qwen3.5-0.8B-Base
|
| 7 |
---
|
| 8 |
|
| 9 |
+
# OpenThai-SystemOne-Vision (v1.3)
|
| 10 |
|
| 11 |
The vision variant of [OpenThai-SystemOne](https://huggingface.co/iapp/OpenThai-SystemOne): the same open Thai + English
|
| 12 |
System One decision model, now with **images in the state** and a fourth question type, **`point`**, that returns where
|
| 13 |
on a screenshot something is. Screenshot in, click out, in one forward pass, no text generation.
|
| 14 |
|
| 15 |
+
**v1.3 (2026-09-26)** clicks better on real Windows and Mac apps (trained on live Windows 11 and macOS screens) and on
|
| 16 |
+
large screens, and adds `point_refine`;
|
| 17 |
+
**v1.1 (2026-09-24)** added Thai **voice-command understanding** for hands-free computer control (a spoken Thai command +
|
| 18 |
the app and window title in, the operation and its target out) and stronger desktop and Doom play. See [Versions](#versions).
|
| 19 |
|
| 20 |
| | OpenThai-SystemOne (text) | OpenThai-SystemOne-Vision |
|
|
|
|
| 70 |
|
| 71 |
## Evaluation
|
| 72 |
|
| 73 |
+
v1.3 next to v1.1 (the previous published version), same scripts; single option order unless noted; accuracy in %.
|
| 74 |
+
"default" = the plain API call (screenshot at ≤ 1280×800); "+ refine" = screenshot at 2 MP (`"detail": "high"`) with
|
| 75 |
+
`point_refine` over the top-3 first-pass candidates, landscape screens only: zoom 50 % on ScreenSpot-v2 / Thai GUI / OS-Atlas,
|
| 76 |
+
35 % on Windows / Mac live and ScreenSpot-Pro. On desktops and large screens, send `"detail": "high"` and `point_refine`.
|
| 77 |
|
| 78 |
+
### Live desktops (new): real apps, click targets from the accessibility tree
|
| 79 |
+
| benchmark | v1.1 default | **v1.3 default** | **v1.3 + refine** | Qwen3-VL-2B | Qwen3-VL-4B |
|
| 80 |
+
|---|---|---|---|---|---|
|
| 81 |
+
| **Windows live** (319 targets, 28 screens of Windows 11: Settings, File Explorer, Control Panel, Notepad, Paint, Calculator, Store, Edge, …) | 63.3 | **68.3** | **69.3** | 67.1 | 73.0 |
|
| 82 |
+
| · Windows Settings pages (114) | 64.9 | 69.3 | – | 66.7 | 68.4 |
|
| 83 |
+
| **Mac live** (358 targets, 29 screens: System Settings, Finder, Safari, Office, iWork, …) | 65.4 | 63.7 | 67.6 | 67.0 | 67.9 |
|
| 84 |
+
| · label-checked (340: 18 targets found unanswerable removed) | 68.5 | 66.8 | **71.2** | 70.6 | 71.5 |
|
| 85 |
+
| latency per click (H100, eval harness) | ~70 ms | ~70 ms | 0.1–0.25 s | ~0.9 s | ~0.75 s |
|
| 86 |
+
|
| 87 |
+
The Windows and Mac live screens are never trained on. v1.3's training data does include *other* screens of the same apps
|
| 88 |
+
(for example other Settings pages), captured the same way. Weak spots: Windows Calculator (60; its buttons are named with
|
| 89 |
+
words such as "Two" and "Multiply by", Qwen3-VL-4B 95), icon-only tiles, and small text links.
|
| 90 |
+
|
| 91 |
+
### GUI grounding benchmarks (`point`: predicted click inside the target box)
|
| 92 |
+
| benchmark | v1.1 default | **v1.3 default** | **v1.3 + refine** | Qwen3-VL-2B | Qwen3-VL-4B |
|
| 93 |
+
|---|---|---|---|---|---|
|
| 94 |
+
| ScreenSpot-v2, all (1,272) | 80.7 | **82.2** | **85.2** | 85.8 | 90.4 |
|
| 95 |
+
| · desktop text / icon | 89.7 / 74.3 | 90.2 / 75.7 | 91.8 / 83.6 | 93.8 / 82.9 | 96.9 / 85.7 |
|
| 96 |
+
| · mobile text / icon | 94.5 / 70.6 | 94.5 / 71.6 | 93.8 / 74.9 | 94.1 / 82.0 | 96.9 / 83.4 |
|
| 97 |
+
| · web text / icon | 85.0 / 62.1 | 84.2 / 70.0 | 88.5 / 74.9 | 85.9 / 72.4 | 91.9 / 83.7 |
|
| 98 |
+
| ScreenSpot-Pro (1,581, 26 pro apps at up to 4K: Windows / macOS / Linux) | 9.8* | 9.0 | **32.9** (29.3 / 38.7 / 28.0) | 43.4* | 54.8* |
|
| 99 |
+
| Desktop (OS-Atlas macOS / Windows / Linux, held-out) | 50.6 | **53.7** | 59.8 (600) | 63.8 | 65.6 |
|
| 100 |
+
| Desktop "not on screen" abstention (26) | 80.8 | **92.3** | – | – | – |
|
| 101 |
+
| Thai GUI held-out domains, `point` | 78.0 | 77.8 | 76.7 (600) | 72.4 | 80.4 |
|
| 102 |
+
| Thai GUI set-of-marks `choice` / `operation` | 93.5 / 99.6 | 93.1 / 99.6 | – | – | – |
|
| 103 |
+
| Thai screen QA `noul` / `choice` | 93.9 / 84.8 | 93.1 / 85.9 | – | – | – |
|
| 104 |
+
| Thai GUI "not on screen" abstention | 46.4 | 48.0 | – | – | – |
|
| 105 |
+
|
| 106 |
+
Qwen3-VL numbers: our runs with the same harness (screenshot + instruction, click inside the box); * every 2nd ScreenSpot-Pro
|
| 107 |
+
item (790).
|
| 108 |
+
|
| 109 |
+
### Better clicks on large screens: `point_refine`
|
| 110 |
+
```python
|
| 111 |
+
r = c.system_one("goal: …", {"where": {"type": "point", "instructions": "ปุ่มบันทึก", "image": "s"}},
|
| 112 |
+
images=[{"id": "s", "path": "screen.png", "role": "screenshot", "detail": "high"}],
|
| 113 |
+
point_refine={"zoom": 0.35, "cands": 3}) # HTTP API: "point_refine": {...} or true
|
| 114 |
+
```
|
| 115 |
+
The first pass points on the whole screen; the client then re-points on a crop around each of the top candidates at native
|
| 116 |
+
detail and keeps the most confident. It only zooms landscape screens larger than the pixel budget (zooming phone screenshots
|
| 117 |
+
hurts). ScreenSpot-Pro 9.0 → 32.9, ScreenSpot-v2 82.2 → 85.2, Mac live 66.8 → 71.2 (checked). Cost: 1 + `cands` forward passes.
|
| 118 |
|
| 119 |
### Thai voice commands (operation + slots from one spoken sentence)
|
| 120 |
+
| set | v1.0 | v1.1 | **v1.3** |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
|---|---|---|---|
|
| 122 |
+
| real commands spoken in use, operation (183) | 65.0 | 88.5 | **89.6** |
|
| 123 |
+
| · slots (site / app / key / Nth) when the operation is right | 90.7 | 100.0 | **100.0** |
|
| 124 |
+
| · chit-chat between commands acted on (41), lower is better | 14.6 | 7.3 | 7.3 |
|
| 125 |
+
| real off-command conversation picked up by the microphone, acted on (227), lower is better | 3.5 | 2.6 | 4.4 |
|
| 126 |
+
| synthetic voice commands, held out (1,406 questions) | 61.2 | 90.5 | **92.0** |
|
| 127 |
+
| synthetic hard cases (fragments, site vs app, ordinals, 446 questions) | 55.0 | 90.8 | **93.3** |
|
| 128 |
+
|
| 129 |
+
### Doom from pixels (`demo/doom_demo.py`, defend_the_center, skill 3, 40 episodes, seed 999)
|
| 130 |
+
| input | v1.0 | v1.1 | **v1.3** | ms / decision |
|
| 131 |
+
|---|---|---|---|---|
|
| 132 |
+
| two 320×240 frames only | 5.5 (20 eps) | 11.8 | **13.0** (1–23) | 49 |
|
| 133 |
+
| the privileged scripted expert it imitates | 6.8 | 6.8 | 6.8 | – |
|
| 134 |
|
| 135 |
+
Offline Doom decisions (600 frames): move 88.2, attack 97.0, in-danger 96.3, nearest-enemy point 51.1, enemy-absent 96.6.
|
| 136 |
|
| 137 |
### Text-only tasks
|
| 138 |
Public 13-subset benchmark (the Bespoke / Jev public suite; `noul`/`choice` = accuracy, `score` = exact level):
|
| 139 |
|
| 140 |
+
| subset | v0.3 | v1.1 | **v1.3** | Jev 1.13.0* | Nimble-9B* |
|
| 141 |
|---|---|---|---|---|---|
|
| 142 |
+
| aegis2 | 83.2 | 82.0 | 82.0 | 80.4 | 81.2 |
|
| 143 |
+
| boolq | 79.7 | 79.7 | 79.7 | 89.7 | 86.0 |
|
| 144 |
+
| civil_comments | 79.0 | 86.0 | **86.3** | 81.0 | 70.3 |
|
| 145 |
+
| helpsteer2 (score) | 41.6 | 42.0 | 40.4 | 34.1 | 39.0 |
|
| 146 |
+
| massive-de-DE | 88.3 | 89.4 | 89.7 | 86.9 | 83.4 |
|
| 147 |
+
| massive-en-US | 88.3 | 89.4 | **90.3** | 87.4 | 86.9 |
|
| 148 |
+
| multinli | 89.0 | 89.0 | 88.3 | 82.9 | 85.3 |
|
| 149 |
+
| paws | 94.0 | 94.0 | 94.0 | 89.2 | 82.8 |
|
| 150 |
+
| pubmedqa | 64.0 | 67.6 | 67.2 | 77.2 | 75.6 |
|
| 151 |
+
| squad2 | 89.3 | 89.0 | 88.6 | 82.9 | 80.6 |
|
| 152 |
+
| summeval-consistency (score) | 75.0 | 75.0 | 75.0 | 81.2 | 75.7 |
|
| 153 |
+
| summeval-relevance (score) | 21.7 | 21.2 | 20.4 | 35.0 | 49.2 |
|
| 154 |
+
| vitaminc-dev | 72.5 | 71.3 | 71.0 | 80.1 | 76.6 |
|
| 155 |
+
| **macro, 13 subsets** | 74.3 | 75.0 | 74.8 | 76.0 | 74.8 |
|
| 156 |
+
| macro, 10 accuracy subsets | 82.7 | 83.7 | 83.7 | 83.8 | 80.9 |
|
| 157 |
|
| 158 |
\* published by Bespoke Labs for their own harness on the same subsets (Jev via the TypeSafe API).
|
|
|
|
| 159 |
|
| 160 |
Thai held-out and eval-split sets (`th` unless noted; banking77 and xLAM are English):
|
| 161 |
|
| 162 |
+
| set | v0.3 | v1.1 | **v1.3** | v1.3, 8 orders |
|
| 163 |
|---|---|---|---|---|
|
| 164 |
+
| banking77 (77-way, whole dataset held out) | 45.4 | 62.0 | 62.0 | 64.6 |
|
| 165 |
+
| contrastive pairs, one-fact flips | 80.7 | 82.1 | 82.1 | 80.4 |
|
| 166 |
+
| MASSIVE-th intent | 90.0 | 91.1 | 91.1 | 91.9 |
|
| 167 |
+
| Prachathai topics | 98.1 | 98.7 | 98.6 | 98.7 |
|
| 168 |
+
| SIB-200 Thai topic (whole dataset held out) | 77.9 | 75.5 | 75.5 | 77.0 |
|
| 169 |
+
| Wisesight sentiment (whole dataset held out) | 51.6 | 52.8 | 52.5 | 51.9 |
|
| 170 |
+
| Wongnai stars (score, exact) | 63.5 | 63.1 | 63.3 | 63.3 |
|
| 171 |
+
| xLAM tool selection | 99.4 | 99.3 | 99.4 | 99.3 |
|
| 172 |
+
| XNLI-th | 77.1 | 75.6 | 75.5 | 75.6 |
|
| 173 |
+
| **macro, 8 accuracy sets** | 77.5 | 79.6 | 79.6 | 79.9 |
|
| 174 |
|
| 175 |
### Latency (batch 1, H100, end-to-end `SystemOneClient`)
|
| 176 |
| request | tokens | ms |
|
| 177 |
|---|---|---|
|
| 178 |
| two 320×240 game frames + short state, 2 questions | 298 (160 visual) | 50 |
|
| 179 |
| 1280×800 screenshot + goal, `point` + `choice` | 1,062 (1,000 visual) | 52 |
|
| 180 |
+
| large screenshot at `"detail": "high"` + `point_refine` (3 candidates), eval harness | ~2,000 visual per pass | ~135 (FHD) – 245 (4K) |
|
| 181 |
| text only (3-question Thai ticket) | 166 | ~40 |
|
|
|
|
| 182 |
|
| 183 |
## Versions
|
| 184 |
|
| 185 |
| version | date | change |
|
| 186 |
|---|---|---|
|
| 187 |
+
| **v1.3** | 2026-09-26 | + live desktop grounding: 665 records from 319 live Windows 11 screens (UI Automation targets; Settings, Control Panel, dialogs, shell folders, Calculator modes, Store, Edge) and 386 records from 116 live macOS screens (accessibility targets; System Settings, Finder, utilities, Safari, Word / Excel / PowerPoint / Pages ribbons), with account names blacked out; on top of an internal v1.2 round (ShowUI desktop + web, 40k zoomed crops, 75k xlangai/Jedi records, 2 MP training, averaged with v1.1). Client / API `point_refine`. Windows live 63.3 → 68.3, ScreenSpot-v2 80.7 → 82.2 (85.2 with refine), ScreenSpot-Pro ~10 → 32.9 with refine, OS-Atlas desktop 50.6 → 53.7, Doom 11.8 → 13.0, text and Thai unchanged. |
|
| 188 |
+
| v1.1 | 2026-09-24 | + Thai voice-command intents (28k synthetic training records from 21k distinct commands + 8.9k hard-case records: fragments / chit-chat = NONE, site vs app, ordinal + topic, ASR spelling variants), desktop grounding round (OS-Atlas macOS / Windows / Linux), text-anchor round (civil_comments, VitaminC); temperatures recalibrated. Voice 65 → 88.5, desktop 39 → 51, Doom pixels 5.5 → 11.9 kills, ScreenSpot-v2 82.7 → 80.7. |
|
| 189 |
| v1.0 | 2026-09-23 | first vision release (Thai GUI crawl, public grounding, Doom rollouts, screen QA on the v0.3 text tower) |
|
| 190 |
|
| 191 |
## How it was built
|
|
|
|
| 200 |
4. v1.1: a text-anchor round and a desktop round averaged into one checkpoint, then a voice-command round
|
| 201 |
(label-conditioned generation by a teacher LLM, every sentence blind re-labelled and kept only on agreement) and a
|
| 202 |
hard-case round aimed at the mistakes seen in real use; the rest of the mix is rehearsed at the same time.
|
| 203 |
+
5. v1.2 (internal): three grounding rounds from v1.1 (ShowUI desktop + web and 2 MP screenshots; 40k zoomed crops at native detail
|
| 204 |
+
that teach the `point_refine` second pass; 75k xlangai/Jedi records), with voice, text and Doom rehearsed; the release is
|
| 205 |
+
the equal-weight average of v1.1 and the last round, which kept v1.1's Thai GUI and Doom skills and the new desktop gains.
|
| 206 |
+
6. v1.3: one round from v1.2 with live Windows 11 and macOS screens captured through the accessibility APIs (a small
|
| 207 |
+
agent inside a Windows VM; the macOS accessibility tree), other apps / pages than the live eval screens, personal names
|
| 208 |
+
blacked out, rest of the mix rehearsed.
|
| 209 |
+
7. Calibration of the per-type temperatures.
|
| 210 |
|
| 211 |
## License & credits
|
| 212 |
|
| 213 |
Apache-2.0. Built by iApp Technology / OpenThaiGPT on Qwen3.5-0.8B-Base. GUI grounding recipe after GUI-Actor; System One
|
| 214 |
+
contract after TypeSafe AI's Jev. Grounding data from OS-Atlas, ShowUI, Wave-UI, GroundUI and Jedi (OSWorld-G);
|
| 215 |
+
benchmarks ScreenSpot-v2 and ScreenSpot-Pro.
|
model.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 1724708176
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c302e4660e4a6af3135309740a59ad1819b9483b1a436487e4bd8fdafe6a5217
|
| 3 |
size 1724708176
|
types.py
CHANGED
|
@@ -160,6 +160,9 @@ class SystemOneRequest(BaseModel):
|
|
| 160 |
permutations: Optional[int] = Field(default=None, ge=1, le=32)
|
| 161 |
# vision variant: up to 4 images; text-only models reject requests that carry images or point questions.
|
| 162 |
images: Optional[List[ImageInput]] = None
|
|
|
|
|
|
|
|
|
|
| 163 |
|
| 164 |
@model_validator(mode="after")
|
| 165 |
def _non_empty(self):
|
|
|
|
| 160 |
permutations: Optional[int] = Field(default=None, ge=1, le=32)
|
| 161 |
# vision variant: up to 4 images; text-only models reject requests that carry images or point questions.
|
| 162 |
images: Optional[List[ImageInput]] = None
|
| 163 |
+
# vision variant: re-point `point` answers on zoomed crops (True = {"steps": 1, "zoom": 0.5, "adaptive": true}); helps small
|
| 164 |
+
# targets on large desktop screens, ~1 extra forward pass per step and point question.
|
| 165 |
+
point_refine: Optional[Union[bool, Dict[str, Any]]] = None
|
| 166 |
|
| 167 |
@model_validator(mode="after")
|
| 168 |
def _non_empty(self):
|