Blame for HARNESS.md

# KORMo-Web-9B Harness Specification

Running a GUI agent requires more than the chat template. This file covers the parts the template does not: what goes into each message, how the screenshot and history are managed, how actions are parsed and executed, and how feedback is written back to the model.

**Match this spec as closely as you can.** The model was trained on this exact layout, and deviating from it (for example by sending several past screenshots, dropping the tab list, or using pixel coordinates) measurably hurts it.

A runnable reference implementation is in [`examples/agent_playwright.py`](examples/agent_playwright.py).

---

## 1. What you need

| Piece | File | Role |
|---|---|---|
| Weights + tokenizer/processor | `model.safetensors`, `tokenizer*.json`, `processor_config.json` | Qwen3.5 VL architecture |
| Chat template | [`chat_template.jinja`](chat_template.jinja) | Renders tools, `<think>`, XML tool calls, `<tool_response>` |
| System prompt | [`system_prompt.md`](system_prompt.md) | Sent verbatim as the `system` message |
| Tool schema | [`tools.json`](tools.json) | The 22 tools, passed as `tools=` |
| Sampling | [`generation_config.json`](generation_config.json) | `temperature 0.6 · top_p 0.95 · top_k 20` |
| Harness | **this file** | Message layout, parsing, execution, feedback |

---

## 2. Conversation layout

```
system     system_prompt.md                     (tools are injected by the chat template)
user       <user_request>{task}</user_request>
           <observation>…</observation>         ← screenshot only if this is the current step
assistant  <think>…</think>\n\n<tool_call>…</tool_call>
tool       {feedback}\n<observation>…</observation>
assistant  …
tool       {feedback}\n<observation> … screenshot: <image> </observation>   ← newest step only
```

- **Full history is kept.** Every previous assistant turn stays in the context with its full `<think>` and `<tool_call>`, and so does every tool feedback and observation. Nothing is truncated; the 50-step episodes in our evaluation stayed inside a 131k-token context.
- **Only one screenshot is sent:** the newest one. When a new observation is appended, the screenshot is removed from the previous observation, but its text is kept.
- **Tool results use `role: "tool"`.** The chat template wraps consecutive tool messages into a single `user` turn as `<tool_response>…</tool_response>`.
- **Thinking is on.** Pass `chat_template_kwargs={"enable_thinking": true}`; the generation prompt then opens with `<think>\n`.

### 2.1 Observation block

```
<observation>
screen size: 1280 x 720
Current tab: 0
Available tabs:
- Tab 0 (active): https://www.google.com/
- Tab 1: https://chatgpt.com/

url: https://www.google.com/
screenshot:
<image>
</observation>
```

- `screen size` is the viewport size in pixels, which is also the screenshot size.
- List every open tab with its 0-based index, and mark the active tab with `(active)`.
- Cut URLs longer than 300 characters to their first 300 characters followed by `…`. Non-ASCII characters, such as Korean, are shown decoded rather than percent-encoded.
- For past observations, drop the `screenshot:` line and the image. The block then ends right after the `url:` line.

### 2.2 First user message

```
<user_request>
{task text}
</user_request>
<observation>
…
</observation>
```

---

## 3. Model output and parsing

Each assistant turn contains an optional reasoning block followed by **exactly one** tool call in Qwen XML form:

```
<think>
I can see the search box at the top of the page. I have not searched yet …
My plan: 1. … 2. …
</think>

<tool_call>
<function=click>
<parameter=x>
498
</parameter>
<parameter=y>
101
</parameter>
</function>
</tool_call>
```

Parsing rules:
1. Remove everything up to and including `</think>`.
2. The remaining text must contain exactly one `<tool_call>…</tool_call>`.
3. Read `<function=NAME>` and each `<parameter=KEY>\nVALUE\n</parameter>`.
4. If the schema types a parameter as `string` (for example `text`, `answer`, `url`, `path`), keep its value as raw text. Decode every other value with `json.loads` (numbers, the `keys` array, `index`).
5. Check the parsed call against `tools.json` (required parameters, no unknown names). If parsing fails, do not act; reply with the failure as feedback (see §5).

Do **not** enable vLLM's server-side tool parser unless it supports this XML format. Parsing the raw text yourself is the simplest option.

---

## 4. Action space (22 tools)

**Coordinates `x, y, x2, y2` are normalized to 0–1000** relative to the screenshot. Convert them to pixels with `px = x / 1000 × width` and `py = y / 1000 × height`. **Scroll deltas are signed pixels** and are not normalized.

| Tool | Arguments | Execute as |
|---|---|---|
| `click` / `double_click` / `right_click` | `x, y` | mouse click at the point |
| `hover` | `x, y` | move the mouse without clicking |
| `focus` | `x, y` | focus the element at the point |
| `drag` | `x, y, x2, y2` | press, move, release |
| `type` | `text` | type into the **currently focused** input; never clears it first |
| `clear` | – | clear the **currently focused** input (Ctrl+A, Backspace) |
| `hotkey` | `keys: [..]` | press the keys together, modifiers first, e.g. `["Control","a"]` |
| `scroll` | `scroll_dy`, `scroll_dx?` | mouse wheel by those pixels (positive = down / right) |
| `wait` | `seconds` | sleep |
| `goto` | `url` | navigate the active tab |
| `go_back` / `go_forward` | – | browser history |
| `new_tab` / `switch_tab(index)` / `close_tab` | – / `index` / – | tab management |
| `upload` | `path` | set the file on the open file picker |
| `finished` | `answer?` | **terminal** – the task is done; `answer` is the final answer |
| `fail` | – | **terminal** – the task cannot be done |
| `EXIT` | – | **terminal** – explicit stop |
| `unknown` | – | no-op; reply `No action executed.` |

Two execution details from training:
- If an action opens a new tab, for example a `target=_blank` link, **switch to the new tab automatically**.
- `type` and `clear` work only on the focused element. The model learned to click or focus the input first.

---

## 5. Feedback strings

The text before `<observation>` in each tool message tells the model what happened. These are the patterns used in training; staying close to them helps.

```
Succeed: `click` on <input> type=text "Search" at (498, 101) executed. Note: no visible navigation or new tab detected.
Succeed: `click` on <h3> "Product title …" at (294, 314) executed. Page navigated to https://example.com/item/1.
Succeed: `click` on <span> "Menu" at (139, 342) executed. New tab opened (tab 1).
Succeed: `type` typed "KAIST graduate credit exchange".
Succeed: `hotkey` ['Enter'] executed. Page navigated to https://…
Succeed: `scroll` by (dx=0, dy=504) executed.
Succeed: waited 10.0 seconds.
Succeed: `goto` navigated to https://chatgpt.com/.
Succeed: `new_tab` opened blank tab (tab 1).
Succeed: `switch_tab` switched to tab 0 (https://…).
Failed: `goto` execution failed: Page.goto: Timeout 60000ms exceeded.
```

- Coordinates in feedback are the model's **normalized** values, not pixels.
- The element description (`<tag> type=… role=… "text"`) is optional, but it helps.
- On failure, use `Failed: \`{tool}\` execution failed: {reason}`.

---

## 6. Recommended settings

| Setting | Value |
|---|---|
| Sampling | `temperature 0.6`, `top_p 0.95`, `top_k 20`, `repetition_penalty 1.0`, `presence_penalty 0` |
| `max_tokens` per step | 16384 (thinking on) |
| Context (`--max-model-len`) | 131072 |
| Viewport | 1280 × 720 (77% of training trajectories; the rest were 1280×1000, 1366×768, 1440×900 and 1536×864) |
| Images per request | 1 (`--limit-mm-per-prompt '{"image":1,"video":0}'`) |
| Max steps | 50 |

All scores in the README were measured with these settings.