|
Download README.md from iapp/OpenThai-SystemOne-Vision: direct link, hf CLI and curl.
- Browser
- Download file 17.1 kB
-
https://huggingface.co/iapp/OpenThai-SystemOne-Vision/resolve/main/README.md
- Command line
-
hf download hf://iapp/OpenThai-SystemOne-Vision/README.md
-
curl -L -o README.md https://huggingface.co/iapp/OpenThai-SystemOne-Vision/resolve/main/README.md
17.1 kB
| license: apache-2.0 | |
| language: [th, en] | |
| pipeline_tag: image-text-to-text | |
| tags: [system-one, decision-model, gui-grounding, computer-use, voice-control, thai, qwen3.5, vision] | |
| base_model: Qwen/Qwen3.5-0.8B-Base | |
| # OpenThai-SystemOne-Vision (v1.4) | |
| Developed by iApp, launched with AIEAT. Part of OpenThai (formerly OpenThaiGPT). | |
| The vision variant of [OpenThai-SystemOne](https://huggingface.co/iapp/OpenThai-SystemOne): the same open Thai + English | |
| System One decision model, now with **images in the state** and a fourth question type, **`point`**, that returns where | |
| on a screenshot something is. Screenshot in, click out, in one forward pass, no text generation. | |
| **v1.4 (2026-09-28)** clicks better on Windows apps (Settings, Calculator) and on icons, and understands voice commands a little | |
| better; it plays Doom from pixels worse than v1.3 (see Evaluation). | |
| **v1.3 (2026-09-26)** clicked better on real Windows and Mac apps (trained on live Windows 11 and macOS screens) and on | |
| large screens, and adds `point_refine`; | |
| **v1.1 (2026-09-24)** added Thai **voice-command understanding** for hands-free computer control (a spoken Thai command + | |
| the app and window title in, the operation and its target out) and stronger desktop and Doom play. See [Versions](#versions). | |
| | | OpenThai-SystemOne (text) | OpenThai-SystemOne-Vision | | |
| |---|---|---| | |
| | parameters | 0.75B | 0.86B (ViT 100M + point head 9M) | | |
| | question types | choice / score / noul | choice / score / noul / **point** | | |
| | state | text or JSON | text or JSON + up to 4 images | | |
| | typical use | routing, moderation, RAG relevance, tool choice, agents with element tables | GUI agents from pixels, Thai voice control, Thai screen reading, game frames, documents | | |
| | text-only behaviour | v0.3 | same API; text scores equal or better than v0.3 (tables below) | | |
| ## API (superset of the TypeSafe `/v1/systemone` contract) | |
| ```json | |
| { | |
| "state": "เป้าหมาย: ค้นหาเที่ยวบิน BKK→CNX", | |
| "images": [{"id": "screen", "data": "data:image/png;base64,...", "role": "screenshot"}], | |
| "questions": { | |
| "where": {"type": "point", "instructions": "ปุ่มค้นหาเที่ยวบิน", "image": "screen"}, | |
| "op": {"type": "choice", "instructions": "next operation", "criteria": {"CLICK": null, "TYPE_TEXT": null, "SCROLL_DOWN": null, "DONE": null}}, | |
| "loaded": {"type": "noul", "instructions": "is the page fully loaded?"} | |
| } | |
| } | |
| ``` | |
| `point` answers: `{"x": 0.63, "y": 0.41, "confidence": 0.86, "abstain": 0.02, "candidates": [{"x", "y", "p", "bbox"}, ...]}` | |
| (normalised image coordinates; `abstain` = probability that the target is not on the image; candidates = the | |
| above-threshold token clusters of the attention heat-map, GUI-Actor style). | |
| Limits: ≤ 4 images, ≤ 8k visual tokens per request (a 1280×800 screenshot ≈ 1,000 tokens at 32 px/token; a 320×240 game | |
| frame ≈ 80); `"detail": "high"` allows up to 1920×1200. | |
| ## Usage | |
| > **Code status (2026-11-15).** The Python client shown on this card (image inputs, `point`, `point_refine`) is not in the | |
| > public GitHub repository yet; it arrives with the next code update there. What runs today with public code: the GGUF build | |
| > [iapp/OpenThai-SystemOne-Vision-GGUF](https://huggingface.co/iapp/OpenThai-SystemOne-Vision-GGUF) on stock llama.cpp | |
| > (`llama-server`, `/v1/systemone`): choice / score / noul questions, with or without a screenshot. It has no `point` head, so | |
| > a click there is a choice among the controls the app reads from the screen (see that card). | |
| ```python | |
| from openthai_systemone import SystemOneClient | |
| c = SystemOneClient("iapp/OpenThai-SystemOne-Vision") | |
| r = c.system_one("goal: open the login form", {"where": {"type": "point", "instructions": "ปุ่มเข้าสู่ระบบ", "image": "s"}}, | |
| images=[{"id": "s", "path": "screen.png", "role": "screenshot"}]) | |
| print(r.answers["where"].x, r.answers["where"].y, r.answers["where"].confidence) | |
| ``` | |
| ### Thai voice control | |
| The voice-command schema (21 operations such as `SEARCH`, `CLICK_NTH`, `OPEN_WEBSITE`, `QUIT_APP`, `PLAY_PAUSE`, `LIKE`, | |
| `COMMENT`, `NONE` = not a command, plus `search_site` / `app` / `site` / `key` / `nth` slots) and a macOS controller | |
| (on-device Thai speech recognition → this model → native mouse / keyboard events) will be published in the GitHub | |
| repo under `demo/voice_control/`. One text-only call decides the operation; clicks add one screenshot call. | |
| ```python | |
| state = 'คำสั่งเสียงภาษาไทยจากผู้ใช้ที่ควบคุมคอมพิวเตอร์ด้วยเสียง: "อยากดูเนเน่ร้องเพลง"\nโปรแกรมที่ใช้อยู่: Google Chrome\nหน้าต่าง: Facebook' | |
| # questions = intent_schema.intent_questions(apps) -> operation SEARCH, search_site YouTube | |
| ``` | |
| ## Evaluation | |
| v1.4 next to v1.3, same scripts; single option order unless noted; accuracy in %. "default" = the plain API call (screenshot at | |
| ≤ 1280×800); "+ refine" = screenshot at 2 MP (`"detail": "high"`) with `point_refine` over the top-3 first-pass candidates, | |
| landscape screens only (zoom 50 % on ScreenSpot-v2 / Thai GUI / OS-Atlas, 35 % on Windows / Mac live and ScreenSpot-Pro). | |
| On desktops and large screens, send `"detail": "high"` and `point_refine`. | |
| ### Live desktops: real apps, click targets from the accessibility tree | |
| | benchmark | v1.3 default | v1.3 + refine | **v1.4 default** | **v1.4 + refine** | Qwen3-VL-2B | Qwen3-VL-4B | | |
| |---|---|---|---|---|---|---| | |
| | **Windows live** (319 targets, 28 screens of Windows 11: Settings, File Explorer, Control Panel, Notepad, Paint, Calculator, Store, Edge, …) | 68.3 | 69.3 | **70.8** | 70.5 | 67.1 | 73.0 | | |
| | · Windows Settings pages (114) | 69.3 | – | **74.6** | – | 66.7 | 68.4 | | |
| | · Windows Calculator (20)† | 60 | – | **80** | – | 80 | 95 | | |
| | **Mac live** (358 targets, 29 screens: System Settings, Finder, Safari, Office, iWork, …) | 63.7 | 67.6 | 64.5 | 67.3 | 67.0 | 67.9 | | |
| | · label-checked (340: 18 targets found unanswerable removed) | 66.8 | **71.2** | 67.6 | 70.9 | 70.6 | 71.5 | | |
| | latency per click (H100, eval harness) | ~70 ms | 0.1–0.25 s | ~70 ms | 0.1–0.25 s | ~0.9 s | ~0.75 s | | |
| The Windows and Mac live screens are never trained on; the training data includes *other* screens of the same apps. | |
| † v1.4's synthetic calculators use Windows' standard accessible names ("Two", "Multiply by"), which is how the Windows live | |
| Calculator targets are named, so part of that gain is learning the naming convention. | |
| ### GUI grounding benchmarks (`point`: predicted click inside the target box) | |
| | benchmark | v1.3 default | v1.3 + refine | **v1.4 default** | **v1.4 + refine** | Qwen3-VL-2B | Qwen3-VL-4B | | |
| |---|---|---|---|---|---|---| | |
| | ScreenSpot-v2, all (1,272) | 82.2 | 85.2 | **82.8** | **85.9** | 85.8 | 90.4 | | |
| | · desktop text / icon | 90.2 / 75.7 | 91.8 / 83.6 | 89.2 / 77.1 | 91.2 / 85.0 | 93.8 / 82.9 | 96.9 / 85.7 | | |
| | · mobile text / icon | 94.5 / 71.6 | 93.8 / 74.9 | 94.8 / 71.6 | 94.5 / 75.4 | 94.1 / 82.0 | 96.9 / 83.4 | | |
| | · web text / icon | 84.2 / 70.0 | 88.5 / 74.9 | 85.9 / 71.4 | 89.3 / 76.4 | 85.9 / 72.4 | 91.9 / 83.7 | | |
| | ScreenSpot-Pro (1,581, 26 pro apps up to 4K; Windows / macOS / Linux) | 9.0 | 32.9 (29.3 / 38.7 / 28.0) | 8.7 | 32.8 (29.2 / 38.9 / 26.0) | 43.4* | 54.8* | | |
| | Desktop (OS-Atlas macOS / Windows / Linux, held-out) | 53.7 | 59.8 (600) | 53.8 | 59.3 (600) | 63.8 | 65.6 | | |
| | Desktop "not on screen" abstention (26) | 92.3 | – | 92.3 | – | – | – | | |
| | Thai GUI held-out domains, `point` (1,611) | 77.8 | 76.7 (600) | **78.6** | 74.9 (600) | 72.4 | 80.4 | | |
| | Thai GUI set-of-marks `choice` / `operation` | 93.1 / 99.6 | – | 92.6 / 99.8 | – | – | – | | |
| | Thai screen QA `noul` / `choice` | 93.1 / 85.9 | – | 94.7 / 85.0 | – | – | – | | |
| | Thai GUI "not on screen" abstention (179) | 48.0 | – | **49.7** | – | – | – | | |
| Qwen3-VL numbers: our runs with the same harness (screenshot + instruction, click inside the box); * every 2nd ScreenSpot-Pro | |
| item (790). Mac live and the 600-item Thai GUI / OS-Atlas subsets move by ±1.5 points between settings; smaller differences | |
| are noise. | |
| ### Better clicks on large screens: `point_refine` | |
| ```python | |
| r = c.system_one("goal: …", {"where": {"type": "point", "instructions": "ปุ่มบันทึก", "image": "s"}}, | |
| images=[{"id": "s", "path": "screen.png", "role": "screenshot", "detail": "high"}], | |
| point_refine={"zoom": 0.35, "cands": 3}) # HTTP API: "point_refine": {...} or true | |
| ``` | |
| The first pass points on the whole screen; the client then re-points on a crop around each of the top candidates at native | |
| detail and keeps the most confident. It only zooms landscape screens larger than the pixel budget (zooming phone screenshots | |
| hurts). v1.4: ScreenSpot-Pro 8.7 → 32.8, ScreenSpot-v2 82.8 → 85.9, Mac live 67.6 → 70.9 (checked). Windows windows usually fit | |
| the 2 MP budget, so there it mostly adds the higher resolution. Cost: 1 + `cands` forward passes. | |
| ### Thai voice commands (operation + slots from one spoken sentence) | |
| | set | v1.1 | v1.3 | **v1.4** | | |
| |---|---|---|---| | |
| | real commands spoken in use, operation (183) | 88.5 | 89.6 | **90.7** | | |
| | · slots (site / app / key / Nth) when the operation is right | 100.0 | 100.0 | 100.0 | | |
| | · chit-chat between commands acted on (41), lower is better | 7.3 | 7.3 | 7.3 | | |
| | real off-command conversation picked up by the microphone, acted on (227), lower is better | 2.6 | 4.4 | 4.0 | | |
| | synthetic voice commands, held out (1,406 questions) | 90.5 | 92.0 | 91.8 | | |
| | synthetic hard cases (fragments, site vs app, ordinals, 446 questions) | 90.8 | 93.3 | 92.6 | | |
| ### Doom from pixels (`demo/doom_demo.py`, defend_the_center, skill 3, two 320×240 frames, 40 episodes per seed) | |
| | seed | v1.1 | v1.3 | **v1.4** | ms / decision | | |
| |---|---|---|---|---| | |
| | 999 | 11.8 | 13.0 | 11.4 | ~50 | | |
| | 777 | – | 11.9 | 9.7 | ~50 | | |
| | mean of both (80 episodes) | – | **12.5** | 10.6 | | | |
| | the privileged scripted expert it imitates | 6.8 | 6.8 | 6.8 | – | | |
| **Known regression:** v1.4 plays Doom from pixels worse than v1.3 (−1.9 kills per episode). Use v1.3 (`revision="v1.3"`) for the | |
| Doom demo. Offline Doom decisions (600 frames), v1.4: move 88.5, attack 97.2, in-danger 96.7, nearest-enemy point 48.9, | |
| enemy-absent 98.3. | |
| ### Text-only tasks | |
| Public 13-subset benchmark (the Bespoke / Jev public suite; `noul`/`choice` = accuracy, `score` = exact level): | |
| | subset | v0.3 | v1.3 | **v1.4** | Jev 1.13.0* | Nimble-9B* | | |
| |---|---|---|---|---|---| | |
| | aegis2 | 83.2 | 82.0 | 81.2 | 80.4 | 81.2 | | |
| | boolq | 79.7 | 79.7 | 79.0 | 89.7 | 86.0 | | |
| | civil_comments | 79.0 | 86.3 | 86.3 | 81.0 | 70.3 | | |
| | helpsteer2 (score) | 41.6 | 40.4 | 40.4 | 34.1 | 39.0 | | |
| | massive-de-DE | 88.3 | 89.7 | 90.0 | 86.9 | 83.4 | | |
| | massive-en-US | 88.3 | 90.3 | 90.3 | 87.4 | 86.9 | | |
| | multinli | 89.0 | 88.3 | 89.0 | 82.9 | 85.3 | | |
| | paws | 94.0 | 94.0 | 94.0 | 89.2 | 82.8 | | |
| | pubmedqa | 64.0 | 67.2 | 67.2 | 77.2 | 75.6 | | |
| | squad2 | 89.3 | 88.6 | 88.3 | 82.9 | 80.6 | | |
| | summeval-consistency (score) | 75.0 | 75.0 | 75.7 | 81.2 | 75.7 | | |
| | summeval-relevance (score) | 21.7 | 20.4 | 20.4 | 35.0 | 49.2 | | |
| | vitaminc-dev | 72.5 | 71.0 | 70.8 | 80.1 | 76.6 | | |
| | **macro, 13 subsets** | 74.3 | 74.8 | 74.8 | 76.0 | 74.8 | | |
| | macro, 10 accuracy subsets | 82.7 | 83.7 | 83.6 | 83.8 | 80.9 | | |
| \* published by Bespoke Labs for their own harness on the same subsets (Jev via the TypeSafe API). | |
| Thai held-out and eval-split sets (`th` unless noted; banking77 and xLAM are English): | |
| | set | v0.3 | v1.3 | **v1.4** | v1.4, 8 orders | | |
| |---|---|---|---|---| | |
| | banking77 (77-way, whole dataset held out) | 45.4 | 62.0 | 62.1 | 64.4 | | |
| | contrastive pairs, one-fact flips | 80.7 | 82.1 | 81.8 | 80.1 | | |
| | MASSIVE-th intent | 90.0 | 91.1 | 91.2 | 92.0 | | |
| | Prachathai topics | 98.1 | 98.6 | 98.6 | 98.7 | | |
| | SIB-200 Thai topic (whole dataset held out) | 77.9 | 75.5 | 75.0 | 77.0 | | |
| | Wisesight sentiment (whole dataset held out) | 51.6 | 52.5 | 52.2 | 51.7 | | |
| | Wongnai stars (score, exact) | 63.5 | 63.3 | 62.9 | 62.9 | | |
| | xLAM tool selection | 99.4 | 99.4 | 99.4 | 99.3 | | |
| | XNLI-th | 77.1 | 75.5 | 75.6 | 75.7 | | |
| | **macro, 8 accuracy sets** | 77.5 | 79.6 | 79.5 | 79.9 | | |
| ### Latency (batch 1, H100, end-to-end `SystemOneClient`) | |
| | request | tokens | ms | | |
| |---|---|---| | |
| | two 320×240 game frames + short state, 2 questions | 298 (160 visual) | 50 | | |
| | 1280×800 screenshot + goal, `point` + `choice` | 1,062 (1,000 visual) | 52 | | |
| | large screenshot at `"detail": "high"` + `point_refine` (3 candidates), eval harness | ~2,000 visual per pass | ~135 (FHD) – 245 (4K) | | |
| | text only (3-question Thai ticket) | 166 | ~40 | | |
| ## Versions | |
| | version | date | change | | |
| |---|---|---| | |
| | **v1.4** | 2026-09-28 | + icon grounding and "not on screen" refusals from xlangai/Jedi (20k icon snaps, 20k refusal rows), 12k synthetic calculators / title bars / toolbars / media controls labelled with their standard accessible names (English + Thai), Mac and Thai GUI rehearsal up; from v1.3. Windows live 68.3 → 70.8 (Settings 69.3 → 74.6, Calculator 60 → 80), ScreenSpot-v2 82.2 → 82.8 (85.9 with refine), Thai not-on-screen 48.0 → 49.7, voice 89.6 → 90.7; **Doom from pixels 12.5 → 10.6 kills**; text, Thai, Mac live, ScreenSpot-Pro, OS-Atlas unchanged within noise. | | |
| | v1.3 | 2026-09-26 | + live desktop grounding: 665 records from 319 live Windows 11 screens (UI Automation targets; Settings, Control Panel, dialogs, shell folders, Calculator modes, Store, Edge) and 386 records from 116 live macOS screens (accessibility targets; System Settings, Finder, utilities, Safari, Word / Excel / PowerPoint / Pages ribbons), with account names blacked out; on top of an internal v1.2 round (ShowUI desktop + web, 40k zoomed crops, 75k xlangai/Jedi records, 2 MP training, averaged with v1.1). Client / API `point_refine`. Windows live 63.3 → 68.3, ScreenSpot-v2 80.7 → 82.2 (85.2 with refine), ScreenSpot-Pro ~10 → 32.9 with refine, OS-Atlas desktop 50.6 → 53.7, Doom 11.8 → 13.0, text and Thai unchanged. | | |
| | v1.1 | 2026-09-24 | + Thai voice-command intents (28k synthetic training records from 21k distinct commands + 8.9k hard-case records: fragments / chit-chat = NONE, site vs app, ordinal + topic, ASR spelling variants), desktop grounding round (OS-Atlas macOS / Windows / Linux), text-anchor round (civil_comments, VitaminC); temperatures recalibrated. Voice 65 → 88.5, desktop 39 → 51, Doom pixels 5.5 → 11.9 kills, ScreenSpot-v2 82.7 → 80.7. | | |
| | v1.0 | 2026-09-23 | first vision release (Thai GUI crawl, public grounding, Doom rollouts, screen QA on the v0.3 text tower) | | |
| ## How it was built | |
| 1. Text tower + slot head from OpenThai-SystemOne v0.3 (Qwen3.5-0.8B, Thai CPT 4.5B tokens, decision SFT + calibration). | |
| 2. Vision tower + merger from Qwen3.5-0.8B-Base, re-aligned with an LM loss on 200k+ image→text pairs (Thai rendered | |
| OCR pages in 55 fonts, Thai web screenshots → element tables, Doom frames → object tables, public screen/document QA). | |
| 3. Decision SFT on the v0.3 text mix plus Thai GUI grounding (Playwright crawl of 300+ Thai sites, teacher-written and | |
| blind-verified Thai/English instructions), public English grounding (Wave-UI, GroundUI, ShowUI), desktop grounding | |
| (OS-Atlas), Doom rollouts (privileged expert, 1.5M frames) with `move` / `attack` / `in_danger` / `nearest_enemy` | |
| questions, and screen QA. | |
| 4. v1.1: a text-anchor round and a desktop round averaged into one checkpoint, then a voice-command round | |
| (label-conditioned generation by a teacher LLM, every sentence blind re-labelled and kept only on agreement) and a | |
| hard-case round aimed at the mistakes seen in real use; the rest of the mix is rehearsed at the same time. | |
| 5. v1.2 (internal): three grounding rounds from v1.1 (ShowUI desktop + web and 2 MP screenshots; 40k zoomed crops at native detail | |
| that teach the `point_refine` second pass; 75k xlangai/Jedi records), with voice, text and Doom rehearsed; the release is | |
| the equal-weight average of v1.1 and the last round, which kept v1.1's Thai GUI and Doom skills and the new desktop gains. | |
| 6. v1.3: one round from v1.2 with live Windows 11 and macOS screens captured through the accessibility APIs (a small | |
| agent inside a Windows VM; the macOS accessibility tree), other apps / pages than the live eval screens, personal names | |
| blacked out, rest of the mix rehearsed. | |
| 7. v1.4: one round from v1.3 with icon grounding / refusal data from xlangai/Jedi and synthetic symbol screens, with the Mac and | |
| Thai GUI data weighted up so they do not slip. | |
| 8. Calibration of the per-type temperatures. | |
| ## License & credits | |
| Apache-2.0, built on Qwen3.5-0.8B-Base. Developed by iApp, launched with AIEAT. GUI grounding recipe after GUI-Actor; System One | |
| contract after TypeSafe AI's Jev. Grounding data from OS-Atlas, ShowUI, Wave-UI, GroundUI and Jedi (OSWorld-G); | |
| benchmarks ScreenSpot-v2 and ScreenSpot-Pro. | |