jungsin3 commited on
Commit
f95fbc9
ยท
verified ยท
1 Parent(s): d9f0dce

README: lead with results and comparison, simplify quickstart

Browse files
Files changed (1) hide show
  1. README.md +110 -128
README.md CHANGED
@@ -17,24 +17,29 @@ tags:
17
 
18
  <style>
19
  #kormo-web-card { font-size: 16px; line-height: 1.6; overflow-wrap: break-word; }
20
- #kormo-web-card h1 { margin: 0 0 12px; font-size: 30px; line-height: 1.25; letter-spacing: -0.018em; }
21
- #kormo-web-card h2 { margin: 30px 0 10px; font-size: 22px; line-height: 1.3; letter-spacing: -0.012em; }
22
- #kormo-web-card h3 { margin: 22px 0 8px; font-size: 17px; font-weight: 700; line-height: 1.4; }
23
- #kormo-web-card h2 + h3 { margin-top: 14px; }
24
  #kormo-web-card p { margin: 0 0 10px; }
25
  #kormo-web-card ul, #kormo-web-card ol { margin: 6px 0 12px; padding-left: 22px; }
26
  #kormo-web-card li { margin: 4px 0; line-height: 1.55; }
27
- #kormo-web-card table { width: 100%; margin: 6px 0 14px; border-collapse: collapse; font-size: 14px; line-height: 1.45; font-variant-numeric: tabular-nums; }
28
  #kormo-web-card th, #kormo-web-card td { padding: 7px 10px; vertical-align: middle; }
 
29
  #kormo-web-card blockquote { margin: 12px 0 16px; padding: 0 0 0 14px; font-size: 15px; }
30
  #kormo-web-card blockquote p { margin: 0; }
31
  #kormo-web-card pre { margin: 10px 0 14px; padding: 14px 16px; overflow-x: auto; line-height: 1.5; }
32
  #kormo-web-card pre code { font-size: 13px; }
33
  #kormo-web-card > p:has(> em:only-child) { margin: 4px 0 14px; font-size: 13px; }
34
- #kormo-web-card .pills { margin: 4px 0 18px; }
35
- #kormo-web-card .pill { display: inline-block; margin: 0 6px 6px 0; padding: 3px 10px; border-radius: 999px; border: 1px solid rgba(127,127,127,.35); font-size: 13px; }
 
 
 
36
  @media (max-width: 640px) {
37
- #kormo-web-card h1 { font-size: 26px; }
 
38
  #kormo-web-card table { font-size: 13px; }
39
  #kormo-web-card th, #kormo-web-card td { padding: 7px 6px; }
40
  }
@@ -44,178 +49,155 @@ tags:
44
 
45
  # KORMo-Web-9B
46
 
47
- <div class="pills">
48
- <span class="pill">๐ŸŒ Web GUI Agent</span>
49
- <span class="pill">๐Ÿ‡ฐ๐Ÿ‡ท Korean + ๐Ÿ‡บ๐Ÿ‡ธ English</span>
50
- <span class="pill">๐Ÿ–ผ๏ธ Screenshot-only</span>
51
- <span class="pill">๐Ÿง  Thinking + XML tool calls</span>
52
- <span class="pill">9B ยท Qwen3.5</span>
 
 
53
  </div>
54
 
55
- **KORMo-Web-9B** is a **9B-parameter vision-language web agent** that completes real browser tasks from **screenshots alone**. It reads the user's request and the current screen, reasons in a `<think>` block, and emits **one browser action at a time** from a fixed set of **22 tools**. The model is built on **Qwen3.5-9B** and trained on **large-scale web understanding data and web trajectories**.
56
 
57
- > It is built for the long, multi-site tasks people actually do. In the example below, the model asks ChatGPT a question, checks the matching notice on a university portal, and writes the result into Notion, in one session and in Korean.
58
 
59
- ## Highlights
60
 
61
- - **Grounded, verifiable actions.** The model sees no DOM and no accessibility tree. Every pointer action is a coordinate normalized to 0โ€“1000 on the screenshot, and the system prompt tells the model to check each result in the next observation before it moves on.
62
- - **Robust to phrasing.** Handles both short commands and long, conversational requests.
63
- - **Cross-site and multi-tab.** The model has native tab tools (`new_tab`, `switch_tab`, `close_tab`), find-in-page, drag-to-copy, and `upload` for file pickers.
 
 
 
 
 
 
64
 
65
- ## Benchmarks
66
 
67
- ### Online-Mind2Web (live websites)
68
 
69
- | Benchmark | Tasks | Success | **Success rate** |
70
- |---|---:|---:|---:|
71
- | Online-Mind2Web (EN) | 300 | 153 | **51.0** |
72
- | Online-Mind2Web (KO) | 312 | 150 | **48.1** |
73
 
74
- *Evaluated on live websites. KO is a Korean version of Online-Mind2Web.*
 
 
 
 
 
 
75
 
76
  ## Quickstart
77
 
78
- ### 1. Serve with vLLM
 
 
79
 
80
  ```bash
81
- vllm serve KORMo-VL/KORMo-Web-9B \
82
- --max-model-len 131072 \
83
- --limit-mm-per-prompt '{"image":1,"video":0}' \
84
- --gpu-memory-utilization 0.90
85
  ```
86
 
87
- ### 2. One step: screenshot in, action out
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
88
 
89
  ```python
90
  import base64, json
91
  from openai import OpenAI
92
- from huggingface_hub import hf_hub_download
93
-
94
- repo = "KORMo-VL/KORMo-Web-9B"
95
- SYSTEM = open(hf_hub_download(repo, "system_prompt.md")).read().strip()
96
- TOOLS = json.load(open(hf_hub_download(repo, "tools.json"))) # 22 tools
97
 
98
- client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
 
99
  shot = base64.b64encode(open("screen.png", "rb").read()).decode()
100
 
101
- observation = (
102
- "<observation>\n"
103
- "screen size: 1280 x 720\n"
104
- "Current tab: 0\n"
105
- "Available tabs:\n"
106
- "- Tab 0 (active): https://www.google.com/\n"
107
- "\n"
108
- "url: https://www.google.com/\n"
109
- "screenshot:\n"
110
- )
111
  messages = [
112
  {"role": "system", "content": SYSTEM},
113
  {"role": "user", "content": [
114
- {"type": "text", "text": "<user_request>\n๊ตฌ๊ธ€์—์„œ KAIST ๋ฅผ ๊ฒ€์ƒ‰ํ•ด์„œ ๊ณต์‹ ํ™ˆํŽ˜์ด์ง€๋ฅผ ์—ด์–ด์ค˜.\n</user_request>\n" + observation},
115
  {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{shot}"}},
116
  {"type": "text", "text": "\n</observation>"},
117
  ]},
118
  ]
119
-
120
  out = client.chat.completions.create(
121
- model=repo, messages=messages, tools=TOOLS,
122
  temperature=0.6, top_p=0.95, max_tokens=16384,
123
- extra_body={"top_k": 20, "repetition_penalty": 1.0,
124
- "chat_template_kwargs": {"enable_thinking": True}},
125
  )
126
- print(out.choices[0].message.content)
127
  ```
128
 
129
- The output looks like this (illustrative):
130
-
131
- ```text
132
- <think>
133
- I can see the Google home page with an empty search box in the center. I have not searched yet.
134
- My plan: 1. Click the search box. 2. Type "KAIST". 3. Press Enter and open the official site.
135
- </think>
136
-
137
- <tool_call>
138
- <function=click>
139
- <parameter=x>
140
- 500
141
- </parameter>
142
- <parameter=y>
143
- 427
144
- </parameter>
145
- </function>
146
- </tool_call>
147
- ```
148
-
149
- Convert the coordinates to pixels with `x / 1000 ร— width` and `y / 1000 ร— height`.
150
 
151
- ### 3. Full agent loop
152
 
153
- [`examples/agent_playwright.py`](examples/agent_playwright.py) is a self-contained Playwright harness. It assembles messages, parses tool calls, executes actions, writes feedback and manages tabs, all exactly as in training:
154
-
155
- ```bash
156
- pip install openai playwright && playwright install chromium
157
- python examples/agent_playwright.py \
158
- --task "๋‹ค๋‚˜์™€์—์„œ 50๋งŒ ์› ์ดํ•˜ 10์ธ์น˜ ์ด์ƒ ํƒœ๋ธ”๋ฆฟ 3๊ฐœ๋ฅผ ์ฐพ์•„ ๊ฐ€๊ฒฉ์„ ์•Œ๋ ค์ค˜" \
159
- --start-url https://www.google.com/
160
  ```
161
 
162
- ## Recommended Settings
163
-
164
- | Setting | Value | Why |
165
- |---|---|---|
166
- | **Temperature** | **0.6** | All reported scores use it |
167
- | **Top-p / Top-k** | **0.95 / 20** | |
168
- | Repetition / presence penalty | 1.0 / 0.0 | Off |
169
- | Thinking | **on** (`enable_thinking: true`) | The model was trained to reason before every action |
170
- | `max_tokens` per step | 16384 | Leaves room for long `<think>` blocks |
171
- | Context | 131072 | The full history is kept, so long episodes need it |
172
- | Screenshots per request | **1 (current only)** | Matches training; past steps keep text only |
173
- | Viewport | 1280 ร— 720 | The most common size in training |
174
- | Max steps | 50 | |
175
-
176
- These sampling values are also written into `generation_config.json`, so vLLM and `transformers` pick them up by default.
177
-
178
- ## How to Talk to the Model
179
-
180
- Inference needs **two things to match training**:
181
 
182
- 1. **The chat template** ([`chat_template.jinja`](chat_template.jinja)): the Qwen3.5 template, which renders the tools, the `<think>` block, XML `<tool_call>`s and `<tool_response>` turns.
183
- 2. **The harness** ([`HARNESS.md`](HARNESS.md)): what goes inside the messages, which the template does not decide.
184
 
185
- ```
186
- system system_prompt.md (+ 22 tools rendered by the chat template)
187
- user <user_request>{task}</user_request> <observation> โ€ฆ screenshot </observation>
188
- assistant <think> โ€ฆ </think> <tool_call><function=click> โ€ฆ </function></tool_call>
189
- tool Succeed: `click` on <button> "๊ฒ€์ƒ‰" at (498, 101) executed. โ€ฆ
190
- <observation> screen size ยท tabs ยท url ยท screenshot (newest only) </observation>
191
- โ€ฆ
192
- ```
193
 
194
- - **Coordinates** (`x, y, x2, y2`) are normalized to **0โ€“1000**. Scroll deltas are signed **pixels**.
195
- - **History:** every past thought, action and observation stays in the context. Only the newest observation carries a screenshot.
196
- - **Tools (22):** `click` `double_click` `right_click` `hover` `focus` `drag` `type` `clear` `hotkey` `scroll` `wait` `goto` `go_back` `go_forward` `new_tab` `switch_tab` `close_tab` `upload` `finished` `fail` `unknown` `EXIT`
197
- - **Termination:** the episode ends on `finished(answer)`, `fail()` or `EXIT()`.
 
 
 
 
198
 
199
- See **[HARNESS.md](HARNESS.md)** for the exact observation format, the parsing rules, the execution semantics and the feedback strings.
200
 
201
  ## Files
202
 
203
  | File | Description |
204
  |---|---|
205
- | `chat_template.jinja` | Chat template used in training and inference |
206
- | `system_prompt.md` | System prompt, verbatim from the training data |
207
- | `tools.json` | The 22-tool schema (OpenAI function format) |
208
- | `HARNESS.md` | Harness spec: messages, observations, parsing, execution, feedback |
209
- | `examples/agent_playwright.py` | Reference end-to-end agent (Playwright + OpenAI-compatible API) |
210
- | `config.json`, `generation_config.json`, `processor_config.json`, `tokenizer*.json` | Model, sampling and processor configs |
211
 
212
  ## ํ•œ๊ตญ์–ด ์š”์•ฝ
213
 
214
- **KORMo-Web-9B**๋Š” ์Šคํฌ๋ฆฐ์ƒท๋งŒ ๋ณด๊ณ  ์›น ๋ธŒ๋ผ์šฐ์ € ์ž‘์—…์„ ์ˆ˜ํ–‰ํ•˜๋Š” 9B ๋น„์ „-์–ธ์–ด ์—์ด์ „ํŠธ์ž…๋‹ˆ๋‹ค. Qwen3.5-9B๋ฅผ ๋ฐ”ํƒ•์œผ๋กœ ๋Œ€๊ทœ๋ชจ ์›น ์ดํ•ด ๋ฐ์ดํ„ฐ์™€ ์›น ๊ถค์ (trajectory)์œผ๋กœ ํ•™์Šตํ–ˆ์Šต๋‹ˆ๋‹ค. ๋งค ์Šคํ… `<think>`๋กœ ์ƒ๊ฐํ•œ ๋’ค 22๊ฐœ ๋„๊ตฌ ์ค‘ ํ•˜๋‚˜๋ฅผ ํ˜ธ์ถœํ•ฉ๋‹ˆ๋‹ค. ์ขŒํ‘œ๋Š” 0โ€“1000์œผ๋กœ ์ •๊ทœํ™”ํ•ฉ๋‹ˆ๋‹ค.
215
 
216
- - **์ถ”๋ก  ์„ค์ •:** temperature 0.6 ยท top_p 0.95 ยท top_k 20 ยท thinking ์ผฌ ยท ์Šคํฌ๋ฆฐ์ƒท์€ ํ˜„์žฌ ํ™”๋ฉด 1์žฅ ยท ์ด๋ ฅ์€ ์ „๋ถ€ ์œ ์ง€
217
- - **ํ•„์š”ํ•œ ํŒŒ์ผ:** `chat_template.jinja`, `system_prompt.md`, `tools.json`. ๋ฉ”์‹œ์ง€ ์กฐ๋ฆฝ ๊ทœ์น™์€ [`HARNESS.md`](HARNESS.md)์— ์žˆ์Šต๋‹ˆ๋‹ค.
218
- - **์„ฑ๋Šฅ:** Online-Mind2Web EN 51.0 ยท KO 48.1
 
219
 
220
  ## Acknowledgements
221
 
 
17
 
18
  <style>
19
  #kormo-web-card { font-size: 16px; line-height: 1.6; overflow-wrap: break-word; }
20
+ #kormo-web-card h1 { margin: 0 0 6px; font-size: 32px; line-height: 1.2; letter-spacing: -0.02em; }
21
+ #kormo-web-card h2 { margin: 32px 0 10px; font-size: 22px; line-height: 1.3; letter-spacing: -0.012em; }
22
+ #kormo-web-card h3 { margin: 20px 0 8px; font-size: 17px; font-weight: 700; line-height: 1.4; }
23
+ #kormo-web-card h2 + h3 { margin-top: 12px; }
24
  #kormo-web-card p { margin: 0 0 10px; }
25
  #kormo-web-card ul, #kormo-web-card ol { margin: 6px 0 12px; padding-left: 22px; }
26
  #kormo-web-card li { margin: 4px 0; line-height: 1.55; }
27
+ #kormo-web-card table { width: 100%; margin: 6px 0 12px; border-collapse: collapse; font-size: 14px; line-height: 1.45; font-variant-numeric: tabular-nums; }
28
  #kormo-web-card th, #kormo-web-card td { padding: 7px 10px; vertical-align: middle; }
29
+ #kormo-web-card .perf th:not(:first-child), #kormo-web-card .perf td:not(:first-child) { text-align: right; }
30
  #kormo-web-card blockquote { margin: 12px 0 16px; padding: 0 0 0 14px; font-size: 15px; }
31
  #kormo-web-card blockquote p { margin: 0; }
32
  #kormo-web-card pre { margin: 10px 0 14px; padding: 14px 16px; overflow-x: auto; line-height: 1.5; }
33
  #kormo-web-card pre code { font-size: 13px; }
34
  #kormo-web-card > p:has(> em:only-child) { margin: 4px 0 14px; font-size: 13px; }
35
+ #kormo-web-card .tagline { margin: 0 0 14px; font-size: 19px; font-weight: 600; opacity: .85; }
36
+ #kormo-web-card .stats { display: flex; flex-wrap: wrap; gap: 12px; margin: 16px 0 8px; }
37
+ #kormo-web-card .stat { flex: 1 1 150px; padding: 14px 16px; border: 1px solid rgba(127,127,127,.3); border-radius: 12px; }
38
+ #kormo-web-card .stat b { display: block; font-size: 30px; line-height: 1.1; letter-spacing: -0.02em; }
39
+ #kormo-web-card .stat span { font-size: 13px; opacity: .75; }
40
  @media (max-width: 640px) {
41
+ #kormo-web-card h1 { font-size: 27px; }
42
+ #kormo-web-card .tagline { font-size: 17px; }
43
  #kormo-web-card table { font-size: 13px; }
44
  #kormo-web-card th, #kormo-web-card td { padding: 7px 6px; }
45
  }
 
49
 
50
  # KORMo-Web-9B
51
 
52
+ <p class="tagline">An open-weight web agent for Korean and English: the strongest open model in our live-web evaluation.</p>
53
+
54
+ **KORMo-Web-9B** is a 9B vision-language agent that completes real browser tasks **from screenshots alone**. Give it a request in Korean or English; it plans, clicks, types and moves between sites and tabs until the task is done. In the same live-web evaluation, it beats every open-weight web agent we compared, **by +18.3 points in English and +27.9 points in Korean**.
55
+
56
+ <div class="stats">
57
+ <div class="stat"><b>51.0%</b><span>Online-Mind2Web ยท English</span></div>
58
+ <div class="stat"><b>48.1%</b><span>Online-Mind2Web ยท Korean</span></div>
59
+ <div class="stat"><b>+26.3 / +31.4</b><span>points over Qwen3.5-9B (EN / KO)</span></div>
60
  </div>
61
 
62
+ ## Performance
63
 
64
+ All models below ran **the same 612 live-website tasks** (300 English, 312 Korean), each with its own prompt and action format, and were scored by **the same LLM judge**.
65
 
66
+ <div class="perf">
67
 
68
+ | Model | Size | Online-Mind2Web EN | Online-Mind2Web KO |
69
+ |---|---:|---:|---:|
70
+ | Qwen3.5-9B (base, no fine-tuning) | 9B | 24.7 | 16.7 |
71
+ | Fara-1.5-4B | 4B | 20.7 | 7.7 |
72
+ | UI-TARS-1.5-7B | 7B | 22.3 | 20.2 |
73
+ | OpenWebRL-4B | 4B | 32.7 | 15.4 |
74
+ | **KORMo-Web-9B (ours)** | **9B** | **51.0** | **48.1** |
75
+ | *Proprietary reference:* GPT-6-luna | โ€“ | *63.0* | *65.7* |
76
+ | *Proprietary reference:* Gemini-3.8-flash | โ€“ | *74.0* | *80.1* |
77
 
78
+ </div>
79
 
80
+ *Success rate (%). The denominator is always the full task count, so crashed or empty runs count as failures.*
81
 
82
+ - **Best open model in both languages.** It scores 51.0 in English and 48.1 in Korean. The strongest other open models score 32.7 in English (OpenWebRL-4B) and 20.2 in Korean (UI-TARS-1.5-7B).
83
+ - **Doubles the base model.** It scores 2.1ร— Qwen3.5-9B in English and 2.9ร— in Korean.
84
+ - **Korean is where it stands out.** The other open agents lose 2 to 17 points when moving from English to Korean. KORMo-Web-9B stays almost level (51.0 โ†’ 48.1) and closes much of the gap to proprietary models.
 
85
 
86
+ ## Why KORMo-Web
87
+
88
+ - **Korean and English, natural requests.** Handles short commands as well as long, conversational instructions in either language.
89
+ - **Screenshot-only.** It needs no DOM or accessibility tree; it acts on what is on screen, like a person, and checks each action's result on the next screenshot.
90
+ - **Thinks, then acts.** A short reasoning step precedes every action, so its behaviour is easy to inspect and debug.
91
+ - **Multi-site, multi-tab workflows.** It can search on one site, verify on another and write the result into a third, managing tabs on its own.
92
+ - **Open and ready to run.** Apache-2.0 weights, a reference browser harness, the tool schema and the exact inference settings are all included.
93
 
94
  ## Quickstart
95
 
96
+ Tested with **vLLM 0.23.0** and **transformers 5.12.1**. The model runs on a single GPU (TP=1).
97
+
98
+ **1. Install and download**
99
 
100
  ```bash
101
+ pip install -U "vllm>=0.23.0" openai playwright huggingface_hub
102
+ playwright install chromium
103
+ huggingface-cli download KORMo-VL/KORMo-Web-9B --local-dir KORMo-Web-9B
 
104
  ```
105
 
106
+ **2. Start the model server**
107
+
108
+ ```bash
109
+ vllm serve ./KORMo-Web-9B --served-model-name KORMo-Web-9B \
110
+ --max-model-len 131072 --limit-mm-per-prompt '{"image":1,"video":0}'
111
+ ```
112
+
113
+ **3. Run a browser task**
114
+
115
+ ```bash
116
+ python KORMo-Web-9B/examples/agent_playwright.py --model KORMo-Web-9B \
117
+ --task "๋‹ค๋‚˜์™€์—์„œ 50๋งŒ ์› ์ดํ•˜, 10์ธ์น˜ ์ด์ƒ ํƒœ๋ธ”๋ฆฟ 3๊ฐœ๋ฅผ ์ฐพ์•„์„œ ๊ฐ€๊ฒฉ์„ ์•Œ๋ ค์ค˜"
118
+ ```
119
+
120
+ The agent opens Chromium, prints each action as it goes, and finishes with the model's answer.
121
+
122
+ <details>
123
+ <summary><b>Use the model from your own code (one step)</b></summary>
124
 
125
  ```python
126
  import base64, json
127
  from openai import OpenAI
 
 
 
 
 
128
 
129
+ SYSTEM = open("KORMo-Web-9B/system_prompt.md").read().strip()
130
+ TOOLS = json.load(open("KORMo-Web-9B/tools.json"))
131
  shot = base64.b64encode(open("screen.png", "rb").read()).decode()
132
 
133
+ obs = ("<observation>\nscreen size: 1280 x 720\nCurrent tab: 0\nAvailable tabs:\n"
134
+ "- Tab 0 (active): https://www.google.com/\n\nurl: https://www.google.com/\nscreenshot:\n")
 
 
 
 
 
 
 
 
135
  messages = [
136
  {"role": "system", "content": SYSTEM},
137
  {"role": "user", "content": [
138
+ {"type": "text", "text": "<user_request>\n๊ตฌ๊ธ€์—์„œ KAIST ๋ฅผ ๊ฒ€์ƒ‰ํ•ด์„œ ๊ณต์‹ ํ™ˆํŽ˜์ด์ง€๋ฅผ ์—ด์–ด์ค˜.\n</user_request>\n" + obs},
139
  {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{shot}"}},
140
  {"type": "text", "text": "\n</observation>"},
141
  ]},
142
  ]
143
+ client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
144
  out = client.chat.completions.create(
145
+ model="KORMo-Web-9B", messages=messages, tools=TOOLS,
146
  temperature=0.6, top_p=0.95, max_tokens=16384,
147
+ extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": True}},
 
148
  )
149
+ print(out.choices[0].message.content) # <think> โ€ฆ </think> <tool_call><function=click> โ€ฆ </tool_call>
150
  ```
151
 
152
+ Coordinates in the tool call are normalized to 0โ€“1000. Convert them to pixels with `x / 1000 ร— width` and `y / 1000 ร— height`.
153
+ </details>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
154
 
155
+ ## How It Works
156
 
157
+ ```
158
+ user <user_request> task </user_request> + observation (tabs, url, screenshot)
159
+ assistant <think> what I see ยท what is done ยท plan </think> <tool_call> one action </tool_call>
160
+ tool action result + new observation (newest screenshot only)
161
+ โ€ฆ repeat until finished(answer)
 
 
162
  ```
163
 
164
+ - **Action space:** 22 browser tools: click, type, hotkey, scroll, drag, goto, tab control, `finished`, and others ([`tools.json`](tools.json)).
165
+ - **Memory:** the full text history of thoughts, actions and observations is kept. Only the current screenshot is sent.
166
+ - **Recommended settings:** temperature 0.6, top_p 0.95, top_k 20, thinking on, up to 16k tokens per step and 50 steps. These are already set in `generation_config.json`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
167
 
168
+ The exact message format, parsing rules and feedback strings are in **[HARNESS.md](HARNESS.md)**. Follow it if you build your own harness.
 
169
 
170
+ ## Evaluation Details
 
 
 
 
 
 
 
171
 
172
+ | | |
173
+ |---|---|
174
+ | Benchmark | Online-Mind2Web, 300 English tasks, plus a Korean version with 312 tasks on Korean websites |
175
+ | Environment | Live websites, Chromium, 1280ร—720 viewport for KORMo-Web-9B |
176
+ | Judge | One LLM judge for all models, using the WebVoyager judge prompt on the last 50 screenshots |
177
+ | Scoring | Success over the full task count; missing or failed runs count as failures |
178
+ | Budget | 50 steps (OpenWebRL-4B was run with its own 30-step setting) |
179
+ | Baselines | Each baseline used its own prompt, action format and viewport |
180
 
181
+ Live-web scores shift with site changes and bot protection, so all models in the table were run within the same evaluation campaign.
182
 
183
  ## Files
184
 
185
  | File | Description |
186
  |---|---|
187
+ | `chat_template.jinja` | Chat template (Qwen3.5 XML tool calls, thinking on) |
188
+ | `system_prompt.md` ยท `tools.json` | System prompt and 22-tool schema, verbatim from training |
189
+ | `HARNESS.md` | Full harness specification |
190
+ | `examples/agent_playwright.py` | Reference browser agent (Playwright + OpenAI-compatible API). It does not execute `upload`. |
191
+ | `config.json` ยท `generation_config.json` ยท `processor_config.json` ยท `tokenizer*.json` | Model, sampling and processor configs |
 
192
 
193
  ## ํ•œ๊ตญ์–ด ์š”์•ฝ
194
 
195
+ **KORMo-Web-9B**๋Š” ์Šคํฌ๋ฆฐ์ƒท๋งŒ ๋ณด๊ณ  ํ•œ๊ตญ์–ดยท์˜์–ด ์›น ์ž‘์—…์„ ์ˆ˜ํ–‰ํ•˜๋Š” 9B ๊ณต๊ฐœ ์›น ์—์ด์ „ํŠธ์ž…๋‹ˆ๋‹ค. ๊ฐ™์€ ์กฐ๊ฑด์˜ ์‹ค์ œ ์›น ํ‰๊ฐ€์—์„œ ๋น„๊ตํ•œ ๊ณต๊ฐœ ๋ชจ๋ธ ์ค‘ ๊ฐ€์žฅ ๋†’์€ ์ ์ˆ˜๋ฅผ ๋ฐ›์•˜์Šต๋‹ˆ๋‹ค.
196
 
197
+ - **Online-Mind2Web:** EN 51.0, KO 48.1
198
+ - **๋ฐ”ํƒ• ๋ชจ๋ธ(Qwen3.5-9B) ๋Œ€๋น„:** EN +26.3, KO +31.4
199
+ - **๊ฐ€์žฅ ๊ฐ•ํ•œ ๋‹ค๋ฅธ ๊ณต๊ฐœ ๋ชจ๋ธ ๋Œ€๋น„:** EN +18.3, KO +27.9
200
+ - ํ•œ๊ตญ์–ด์—์„œ ์„ฑ๋Šฅ์ด ๊ฑฐ์˜ ๋–จ์–ด์ง€์ง€ ์•Š์Šต๋‹ˆ๋‹ค(51.0 โ†’ 48.1).
201
 
202
  ## Acknowledgements
203