| # KORMo-Web-9B Harness Specification |
|
|
| Running a GUI agent requires more than the chat template. This file covers the parts the template does not: what goes into each message, how the screenshot and history are managed, how actions are parsed and executed, and how feedback is written back to the model. |
|
|
| **Match this spec as closely as you can.** The model was trained on this exact layout, and deviating from it (for example by sending several past screenshots, dropping the tab list, or using pixel coordinates) measurably hurts it. |
|
|
| A runnable reference implementation is in [`examples/agent_playwright.py`](examples/agent_playwright.py). |
|
|
| --- |
|
|
| ## 1. What you need |
|
|
| | Piece | File | Role | |
| |---|---|---| |
| | Weights + tokenizer/processor | `model.safetensors`, `tokenizer*.json`, `processor_config.json` | Qwen3.5 VL architecture | |
| | Chat template | [`chat_template.jinja`](chat_template.jinja) | Renders tools, `<think>`, XML tool calls, `<tool_response>` | |
| | System prompt | [`system_prompt.md`](system_prompt.md) | Sent verbatim as the `system` message | |
| | Tool schema | [`tools.json`](tools.json) | The 22 tools, passed as `tools=` | |
| | Sampling | [`generation_config.json`](generation_config.json) | `temperature 0.6 · top_p 0.95 · top_k 20` | |
| | Harness | **this file** | Message layout, parsing, execution, feedback | |
|
|
| --- |
|
|
| ## 2. Conversation layout |
|
|
| ``` |
| system system_prompt.md (tools are injected by the chat template) |
| user <user_request>{task}</user_request> |
| <observation>…</observation> ← screenshot only if this is the current step |
| assistant <think>…</think>\n\n<tool_call>…</tool_call> |
| tool {feedback}\n<observation>…</observation> |
| assistant … |
| tool {feedback}\n<observation> … screenshot: <image> </observation> ← newest step only |
| ``` |
|
|
| - **Full history is kept.** Every previous assistant turn stays in the context with its full `<think>` and `<tool_call>`, and so does every tool feedback and observation. Nothing is truncated; the 50-step episodes in our evaluation stayed inside a 131k-token context. |
| - **Only one screenshot is sent:** the newest one. When a new observation is appended, the screenshot is removed from the previous observation, but its text is kept. |
| - **Tool results use `role: "tool"`.** The chat template wraps consecutive tool messages into a single `user` turn as `<tool_response>…</tool_response>`. |
| - **Thinking is on.** Pass `chat_template_kwargs={"enable_thinking": true}`; the generation prompt then opens with `<think>\n`. |
|
|
| ### 2.1 Observation block |
|
|
| ``` |
| <observation> |
| screen size: 1280 x 720 |
| Current tab: 0 |
| Available tabs: |
| - Tab 0 (active): https://www.google.com/ |
| - Tab 1: https://chatgpt.com/ |
| |
| url: https://www.google.com/ |
| screenshot: |
| <image> |
| </observation> |
| ``` |
|
|
| - `screen size` is the viewport size in pixels, which is also the screenshot size. |
| - List every open tab with its 0-based index, and mark the active tab with `(active)`. |
| - Cut URLs longer than 300 characters to their first 300 characters followed by `…`. Non-ASCII characters, such as Korean, are shown decoded rather than percent-encoded. |
| - For past observations, drop the `screenshot:` line and the image. The block then ends right after the `url:` line. |
|
|
| ### 2.2 First user message |
|
|
| ``` |
| <user_request> |
| {task text} |
| </user_request> |
| <observation> |
| … |
| </observation> |
| ``` |
|
|
| --- |
|
|
| ## 3. Model output and parsing |
|
|
| Each assistant turn contains an optional reasoning block followed by **exactly one** tool call in Qwen XML form: |
|
|
| ``` |
| <think> |
| I can see the search box at the top of the page. I have not searched yet … |
| My plan: 1. … 2. … |
| </think> |
| |
| <tool_call> |
| <function=click> |
| <parameter=x> |
| 498 |
| </parameter> |
| <parameter=y> |
| 101 |
| </parameter> |
| </function> |
| </tool_call> |
| ``` |
|
|
| Parsing rules: |
| 1. Remove everything up to and including `</think>`. |
| 2. The remaining text must contain exactly one `<tool_call>…</tool_call>`. |
| 3. Read `<function=NAME>` and each `<parameter=KEY>\nVALUE\n</parameter>`. |
| 4. If the schema types a parameter as `string` (for example `text`, `answer`, `url`, `path`), keep its value as raw text. Decode every other value with `json.loads` (numbers, the `keys` array, `index`). |
| 5. Check the parsed call against `tools.json` (required parameters, no unknown names). If parsing fails, do not act; reply with the failure as feedback (see §5). |
|
|
| Do **not** enable vLLM's server-side tool parser unless it supports this XML format. Parsing the raw text yourself is the simplest option. |
|
|
| --- |
|
|
| ## 4. Action space (22 tools) |
|
|
| **Coordinates `x, y, x2, y2` are normalized to 0–1000** relative to the screenshot. Convert them to pixels with `px = x / 1000 × width` and `py = y / 1000 × height`. **Scroll deltas are signed pixels** and are not normalized. |
|
|
| | Tool | Arguments | Execute as | |
| |---|---|---| |
| | `click` / `double_click` / `right_click` | `x, y` | mouse click at the point | |
| | `hover` | `x, y` | move the mouse without clicking | |
| | `focus` | `x, y` | focus the element at the point | |
| | `drag` | `x, y, x2, y2` | press, move, release | |
| | `type` | `text` | type into the **currently focused** input; never clears it first | |
| | `clear` | – | clear the **currently focused** input (Ctrl+A, Backspace) | |
| | `hotkey` | `keys: [..]` | press the keys together, modifiers first, e.g. `["Control","a"]` | |
| | `scroll` | `scroll_dy`, `scroll_dx?` | mouse wheel by those pixels (positive = down / right) | |
| | `wait` | `seconds` | sleep | |
| | `goto` | `url` | navigate the active tab | |
| | `go_back` / `go_forward` | – | browser history | |
| | `new_tab` / `switch_tab(index)` / `close_tab` | – / `index` / – | tab management | |
| | `upload` | `path` | set the file on the open file picker | |
| | `finished` | `answer?` | **terminal** – the task is done; `answer` is the final answer | |
| | `fail` | – | **terminal** – the task cannot be done | |
| | `EXIT` | – | **terminal** – explicit stop | |
| | `unknown` | – | no-op; reply `No action executed.` | |
|
|
| Two execution details from training: |
| - If an action opens a new tab, for example a `target=_blank` link, **switch to the new tab automatically**. |
| - `type` and `clear` work only on the focused element. The model learned to click or focus the input first. |
|
|
| --- |
|
|
| ## 5. Feedback strings |
|
|
| The text before `<observation>` in each tool message tells the model what happened. These are the patterns used in training; staying close to them helps. |
|
|
| ``` |
| Succeed: `click` on <input> type=text "Search" at (498, 101) executed. Note: no visible navigation or new tab detected. |
| Succeed: `click` on <h3> "Product title …" at (294, 314) executed. Page navigated to https://example.com/item/1. |
| Succeed: `click` on <span> "Menu" at (139, 342) executed. New tab opened (tab 1). |
| Succeed: `type` typed "KAIST graduate credit exchange". |
| Succeed: `hotkey` ['Enter'] executed. Page navigated to https://… |
| Succeed: `scroll` by (dx=0, dy=504) executed. |
| Succeed: waited 10.0 seconds. |
| Succeed: `goto` navigated to https://chatgpt.com/. |
| Succeed: `new_tab` opened blank tab (tab 1). |
| Succeed: `switch_tab` switched to tab 0 (https://…). |
| Failed: `goto` execution failed: Page.goto: Timeout 60000ms exceeded. |
| ``` |
|
|
| - Coordinates in feedback are the model's **normalized** values, not pixels. |
| - The element description (`<tag> type=… role=… "text"`) is optional, but it helps. |
| - On failure, use `Failed: \`{tool}\` execution failed: {reason}`. |
|
|
| --- |
|
|
| ## 6. Recommended settings |
|
|
| | Setting | Value | |
| |---|---| |
| | Sampling | `temperature 0.6`, `top_p 0.95`, `top_k 20`, `repetition_penalty 1.0`, `presence_penalty 0` | |
| | `max_tokens` per step | 16384 (thinking on) | |
| | Context (`--max-model-len`) | 131072 | |
| | Viewport | 1280 × 720 (77% of training trajectories; the rest were 1280×1000, 1366×768, 1440×900 and 1536×864) | |
| | Images per request | 1 (`--limit-mm-per-prompt '{"image":1,"video":0}'`) | |
| | Max steps | 50 | |
|
|
| All scores in the README were measured with these settings. |
|
|
