Image-Text-to-Text
Transformers
Safetensors
Korean
English
qwen3_5
gui-agent
web-agent
computer-use
browser
korean
conversational
Instructions to use KORMo-VL/KORMo-Web-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KORMo-VL/KORMo-Web-9B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="KORMo-VL/KORMo-Web-9B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("KORMo-VL/KORMo-Web-9B") model = AutoModelForMultimodalLM.from_pretrained("KORMo-VL/KORMo-Web-9B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use KORMo-VL/KORMo-Web-9B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "KORMo-VL/KORMo-Web-9B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-9B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/KORMo-VL/KORMo-Web-9B
- SGLang
How to use KORMo-VL/KORMo-Web-9B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-9B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-9B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-9B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-9B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use KORMo-VL/KORMo-Web-9B with Docker Model Runner:
docker model run hf.co/KORMo-VL/KORMo-Web-9B
README: lead with results and comparison, simplify quickstart
Browse files
README.md
CHANGED
|
@@ -17,24 +17,29 @@ tags:
|
|
| 17 |
|
| 18 |
<style>
|
| 19 |
#kormo-web-card { font-size: 16px; line-height: 1.6; overflow-wrap: break-word; }
|
| 20 |
-
#kormo-web-card h1 { margin: 0 0
|
| 21 |
-
#kormo-web-card h2 { margin:
|
| 22 |
-
#kormo-web-card h3 { margin:
|
| 23 |
-
#kormo-web-card h2 + h3 { margin-top:
|
| 24 |
#kormo-web-card p { margin: 0 0 10px; }
|
| 25 |
#kormo-web-card ul, #kormo-web-card ol { margin: 6px 0 12px; padding-left: 22px; }
|
| 26 |
#kormo-web-card li { margin: 4px 0; line-height: 1.55; }
|
| 27 |
-
#kormo-web-card table { width: 100%; margin: 6px 0
|
| 28 |
#kormo-web-card th, #kormo-web-card td { padding: 7px 10px; vertical-align: middle; }
|
|
|
|
| 29 |
#kormo-web-card blockquote { margin: 12px 0 16px; padding: 0 0 0 14px; font-size: 15px; }
|
| 30 |
#kormo-web-card blockquote p { margin: 0; }
|
| 31 |
#kormo-web-card pre { margin: 10px 0 14px; padding: 14px 16px; overflow-x: auto; line-height: 1.5; }
|
| 32 |
#kormo-web-card pre code { font-size: 13px; }
|
| 33 |
#kormo-web-card > p:has(> em:only-child) { margin: 4px 0 14px; font-size: 13px; }
|
| 34 |
-
#kormo-web-card .
|
| 35 |
-
#kormo-web-card .
|
|
|
|
|
|
|
|
|
|
| 36 |
@media (max-width: 640px) {
|
| 37 |
-
#kormo-web-card h1 { font-size:
|
|
|
|
| 38 |
#kormo-web-card table { font-size: 13px; }
|
| 39 |
#kormo-web-card th, #kormo-web-card td { padding: 7px 6px; }
|
| 40 |
}
|
|
@@ -44,178 +49,155 @@ tags:
|
|
| 44 |
|
| 45 |
# KORMo-Web-9B
|
| 46 |
|
| 47 |
-
<
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
<
|
| 52 |
-
<
|
|
|
|
|
|
|
| 53 |
</div>
|
| 54 |
|
| 55 |
-
|
| 56 |
|
| 57 |
-
|
| 58 |
|
| 59 |
-
|
| 60 |
|
| 61 |
-
|
| 62 |
-
-
|
| 63 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
|
| 65 |
-
|
| 66 |
|
| 67 |
-
|
| 68 |
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
| Online-Mind2Web (KO) | 312 | 150 | **48.1** |
|
| 73 |
|
| 74 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 75 |
|
| 76 |
## Quickstart
|
| 77 |
|
| 78 |
-
|
|
|
|
|
|
|
| 79 |
|
| 80 |
```bash
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
--gpu-memory-utilization 0.90
|
| 85 |
```
|
| 86 |
|
| 87 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 88 |
|
| 89 |
```python
|
| 90 |
import base64, json
|
| 91 |
from openai import OpenAI
|
| 92 |
-
from huggingface_hub import hf_hub_download
|
| 93 |
-
|
| 94 |
-
repo = "KORMo-VL/KORMo-Web-9B"
|
| 95 |
-
SYSTEM = open(hf_hub_download(repo, "system_prompt.md")).read().strip()
|
| 96 |
-
TOOLS = json.load(open(hf_hub_download(repo, "tools.json"))) # 22 tools
|
| 97 |
|
| 98 |
-
|
|
|
|
| 99 |
shot = base64.b64encode(open("screen.png", "rb").read()).decode()
|
| 100 |
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
"screen size: 1280 x 720\n"
|
| 104 |
-
"Current tab: 0\n"
|
| 105 |
-
"Available tabs:\n"
|
| 106 |
-
"- Tab 0 (active): https://www.google.com/\n"
|
| 107 |
-
"\n"
|
| 108 |
-
"url: https://www.google.com/\n"
|
| 109 |
-
"screenshot:\n"
|
| 110 |
-
)
|
| 111 |
messages = [
|
| 112 |
{"role": "system", "content": SYSTEM},
|
| 113 |
{"role": "user", "content": [
|
| 114 |
-
{"type": "text", "text": "<user_request>\n๊ตฌ๊ธ์์ KAIST ๋ฅผ ๊ฒ์ํด์ ๊ณต์ ํํ์ด์ง๋ฅผ ์ด์ด์ค.\n</user_request>\n" +
|
| 115 |
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{shot}"}},
|
| 116 |
{"type": "text", "text": "\n</observation>"},
|
| 117 |
]},
|
| 118 |
]
|
| 119 |
-
|
| 120 |
out = client.chat.completions.create(
|
| 121 |
-
model=
|
| 122 |
temperature=0.6, top_p=0.95, max_tokens=16384,
|
| 123 |
-
extra_body={"top_k": 20, "
|
| 124 |
-
"chat_template_kwargs": {"enable_thinking": True}},
|
| 125 |
)
|
| 126 |
-
print(out.choices[0].message.content)
|
| 127 |
```
|
| 128 |
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
```text
|
| 132 |
-
<think>
|
| 133 |
-
I can see the Google home page with an empty search box in the center. I have not searched yet.
|
| 134 |
-
My plan: 1. Click the search box. 2. Type "KAIST". 3. Press Enter and open the official site.
|
| 135 |
-
</think>
|
| 136 |
-
|
| 137 |
-
<tool_call>
|
| 138 |
-
<function=click>
|
| 139 |
-
<parameter=x>
|
| 140 |
-
500
|
| 141 |
-
</parameter>
|
| 142 |
-
<parameter=y>
|
| 143 |
-
427
|
| 144 |
-
</parameter>
|
| 145 |
-
</function>
|
| 146 |
-
</tool_call>
|
| 147 |
-
```
|
| 148 |
-
|
| 149 |
-
Convert the coordinates to pixels with `x / 1000 ร width` and `y / 1000 ร height`.
|
| 150 |
|
| 151 |
-
##
|
| 152 |
|
| 153 |
-
|
| 154 |
-
|
| 155 |
-
|
| 156 |
-
|
| 157 |
-
|
| 158 |
-
--task "๋ค๋์์์ 50๋ง ์ ์ดํ 10์ธ์น ์ด์ ํ๋ธ๋ฆฟ 3๊ฐ๋ฅผ ์ฐพ์ ๊ฐ๊ฒฉ์ ์๋ ค์ค" \
|
| 159 |
-
--start-url https://www.google.com/
|
| 160 |
```
|
| 161 |
|
| 162 |
-
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
|---|---|---|
|
| 166 |
-
| **Temperature** | **0.6** | All reported scores use it |
|
| 167 |
-
| **Top-p / Top-k** | **0.95 / 20** | |
|
| 168 |
-
| Repetition / presence penalty | 1.0 / 0.0 | Off |
|
| 169 |
-
| Thinking | **on** (`enable_thinking: true`) | The model was trained to reason before every action |
|
| 170 |
-
| `max_tokens` per step | 16384 | Leaves room for long `<think>` blocks |
|
| 171 |
-
| Context | 131072 | The full history is kept, so long episodes need it |
|
| 172 |
-
| Screenshots per request | **1 (current only)** | Matches training; past steps keep text only |
|
| 173 |
-
| Viewport | 1280 ร 720 | The most common size in training |
|
| 174 |
-
| Max steps | 50 | |
|
| 175 |
-
|
| 176 |
-
These sampling values are also written into `generation_config.json`, so vLLM and `transformers` pick them up by default.
|
| 177 |
-
|
| 178 |
-
## How to Talk to the Model
|
| 179 |
-
|
| 180 |
-
Inference needs **two things to match training**:
|
| 181 |
|
| 182 |
-
|
| 183 |
-
2. **The harness** ([`HARNESS.md`](HARNESS.md)): what goes inside the messages, which the template does not decide.
|
| 184 |
|
| 185 |
-
|
| 186 |
-
system system_prompt.md (+ 22 tools rendered by the chat template)
|
| 187 |
-
user <user_request>{task}</user_request> <observation> โฆ screenshot </observation>
|
| 188 |
-
assistant <think> โฆ </think> <tool_call><function=click> โฆ </function></tool_call>
|
| 189 |
-
tool Succeed: `click` on <button> "๊ฒ์" at (498, 101) executed. โฆ
|
| 190 |
-
<observation> screen size ยท tabs ยท url ยท screenshot (newest only) </observation>
|
| 191 |
-
โฆ
|
| 192 |
-
```
|
| 193 |
|
| 194 |
-
|
| 195 |
-
-
|
| 196 |
-
|
| 197 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 198 |
|
| 199 |
-
|
| 200 |
|
| 201 |
## Files
|
| 202 |
|
| 203 |
| File | Description |
|
| 204 |
|---|---|
|
| 205 |
-
| `chat_template.jinja` | Chat template
|
| 206 |
-
| `system_prompt.md` | System prompt, verbatim from
|
| 207 |
-
| `
|
| 208 |
-
| `
|
| 209 |
-
| `
|
| 210 |
-
| `config.json`, `generation_config.json`, `processor_config.json`, `tokenizer*.json` | Model, sampling and processor configs |
|
| 211 |
|
| 212 |
## ํ๊ตญ์ด ์์ฝ
|
| 213 |
|
| 214 |
-
**KORMo-Web-9B**๋ ์คํฌ๋ฆฐ์ท๋ง ๋ณด๊ณ ์น
|
| 215 |
|
| 216 |
-
- **
|
| 217 |
-
- **
|
| 218 |
-
- **
|
|
|
|
| 219 |
|
| 220 |
## Acknowledgements
|
| 221 |
|
|
|
|
| 17 |
|
| 18 |
<style>
|
| 19 |
#kormo-web-card { font-size: 16px; line-height: 1.6; overflow-wrap: break-word; }
|
| 20 |
+
#kormo-web-card h1 { margin: 0 0 6px; font-size: 32px; line-height: 1.2; letter-spacing: -0.02em; }
|
| 21 |
+
#kormo-web-card h2 { margin: 32px 0 10px; font-size: 22px; line-height: 1.3; letter-spacing: -0.012em; }
|
| 22 |
+
#kormo-web-card h3 { margin: 20px 0 8px; font-size: 17px; font-weight: 700; line-height: 1.4; }
|
| 23 |
+
#kormo-web-card h2 + h3 { margin-top: 12px; }
|
| 24 |
#kormo-web-card p { margin: 0 0 10px; }
|
| 25 |
#kormo-web-card ul, #kormo-web-card ol { margin: 6px 0 12px; padding-left: 22px; }
|
| 26 |
#kormo-web-card li { margin: 4px 0; line-height: 1.55; }
|
| 27 |
+
#kormo-web-card table { width: 100%; margin: 6px 0 12px; border-collapse: collapse; font-size: 14px; line-height: 1.45; font-variant-numeric: tabular-nums; }
|
| 28 |
#kormo-web-card th, #kormo-web-card td { padding: 7px 10px; vertical-align: middle; }
|
| 29 |
+
#kormo-web-card .perf th:not(:first-child), #kormo-web-card .perf td:not(:first-child) { text-align: right; }
|
| 30 |
#kormo-web-card blockquote { margin: 12px 0 16px; padding: 0 0 0 14px; font-size: 15px; }
|
| 31 |
#kormo-web-card blockquote p { margin: 0; }
|
| 32 |
#kormo-web-card pre { margin: 10px 0 14px; padding: 14px 16px; overflow-x: auto; line-height: 1.5; }
|
| 33 |
#kormo-web-card pre code { font-size: 13px; }
|
| 34 |
#kormo-web-card > p:has(> em:only-child) { margin: 4px 0 14px; font-size: 13px; }
|
| 35 |
+
#kormo-web-card .tagline { margin: 0 0 14px; font-size: 19px; font-weight: 600; opacity: .85; }
|
| 36 |
+
#kormo-web-card .stats { display: flex; flex-wrap: wrap; gap: 12px; margin: 16px 0 8px; }
|
| 37 |
+
#kormo-web-card .stat { flex: 1 1 150px; padding: 14px 16px; border: 1px solid rgba(127,127,127,.3); border-radius: 12px; }
|
| 38 |
+
#kormo-web-card .stat b { display: block; font-size: 30px; line-height: 1.1; letter-spacing: -0.02em; }
|
| 39 |
+
#kormo-web-card .stat span { font-size: 13px; opacity: .75; }
|
| 40 |
@media (max-width: 640px) {
|
| 41 |
+
#kormo-web-card h1 { font-size: 27px; }
|
| 42 |
+
#kormo-web-card .tagline { font-size: 17px; }
|
| 43 |
#kormo-web-card table { font-size: 13px; }
|
| 44 |
#kormo-web-card th, #kormo-web-card td { padding: 7px 6px; }
|
| 45 |
}
|
|
|
|
| 49 |
|
| 50 |
# KORMo-Web-9B
|
| 51 |
|
| 52 |
+
<p class="tagline">An open-weight web agent for Korean and English: the strongest open model in our live-web evaluation.</p>
|
| 53 |
+
|
| 54 |
+
**KORMo-Web-9B** is a 9B vision-language agent that completes real browser tasks **from screenshots alone**. Give it a request in Korean or English; it plans, clicks, types and moves between sites and tabs until the task is done. In the same live-web evaluation, it beats every open-weight web agent we compared, **by +18.3 points in English and +27.9 points in Korean**.
|
| 55 |
+
|
| 56 |
+
<div class="stats">
|
| 57 |
+
<div class="stat"><b>51.0%</b><span>Online-Mind2Web ยท English</span></div>
|
| 58 |
+
<div class="stat"><b>48.1%</b><span>Online-Mind2Web ยท Korean</span></div>
|
| 59 |
+
<div class="stat"><b>+26.3 / +31.4</b><span>points over Qwen3.5-9B (EN / KO)</span></div>
|
| 60 |
</div>
|
| 61 |
|
| 62 |
+
## Performance
|
| 63 |
|
| 64 |
+
All models below ran **the same 612 live-website tasks** (300 English, 312 Korean), each with its own prompt and action format, and were scored by **the same LLM judge**.
|
| 65 |
|
| 66 |
+
<div class="perf">
|
| 67 |
|
| 68 |
+
| Model | Size | Online-Mind2Web EN | Online-Mind2Web KO |
|
| 69 |
+
|---|---:|---:|---:|
|
| 70 |
+
| Qwen3.5-9B (base, no fine-tuning) | 9B | 24.7 | 16.7 |
|
| 71 |
+
| Fara-1.5-4B | 4B | 20.7 | 7.7 |
|
| 72 |
+
| UI-TARS-1.5-7B | 7B | 22.3 | 20.2 |
|
| 73 |
+
| OpenWebRL-4B | 4B | 32.7 | 15.4 |
|
| 74 |
+
| **KORMo-Web-9B (ours)** | **9B** | **51.0** | **48.1** |
|
| 75 |
+
| *Proprietary reference:* GPT-6-luna | โ | *63.0* | *65.7* |
|
| 76 |
+
| *Proprietary reference:* Gemini-3.8-flash | โ | *74.0* | *80.1* |
|
| 77 |
|
| 78 |
+
</div>
|
| 79 |
|
| 80 |
+
*Success rate (%). The denominator is always the full task count, so crashed or empty runs count as failures.*
|
| 81 |
|
| 82 |
+
- **Best open model in both languages.** It scores 51.0 in English and 48.1 in Korean. The strongest other open models score 32.7 in English (OpenWebRL-4B) and 20.2 in Korean (UI-TARS-1.5-7B).
|
| 83 |
+
- **Doubles the base model.** It scores 2.1ร Qwen3.5-9B in English and 2.9ร in Korean.
|
| 84 |
+
- **Korean is where it stands out.** The other open agents lose 2 to 17 points when moving from English to Korean. KORMo-Web-9B stays almost level (51.0 โ 48.1) and closes much of the gap to proprietary models.
|
|
|
|
| 85 |
|
| 86 |
+
## Why KORMo-Web
|
| 87 |
+
|
| 88 |
+
- **Korean and English, natural requests.** Handles short commands as well as long, conversational instructions in either language.
|
| 89 |
+
- **Screenshot-only.** It needs no DOM or accessibility tree; it acts on what is on screen, like a person, and checks each action's result on the next screenshot.
|
| 90 |
+
- **Thinks, then acts.** A short reasoning step precedes every action, so its behaviour is easy to inspect and debug.
|
| 91 |
+
- **Multi-site, multi-tab workflows.** It can search on one site, verify on another and write the result into a third, managing tabs on its own.
|
| 92 |
+
- **Open and ready to run.** Apache-2.0 weights, a reference browser harness, the tool schema and the exact inference settings are all included.
|
| 93 |
|
| 94 |
## Quickstart
|
| 95 |
|
| 96 |
+
Tested with **vLLM 0.23.0** and **transformers 5.12.1**. The model runs on a single GPU (TP=1).
|
| 97 |
+
|
| 98 |
+
**1. Install and download**
|
| 99 |
|
| 100 |
```bash
|
| 101 |
+
pip install -U "vllm>=0.23.0" openai playwright huggingface_hub
|
| 102 |
+
playwright install chromium
|
| 103 |
+
huggingface-cli download KORMo-VL/KORMo-Web-9B --local-dir KORMo-Web-9B
|
|
|
|
| 104 |
```
|
| 105 |
|
| 106 |
+
**2. Start the model server**
|
| 107 |
+
|
| 108 |
+
```bash
|
| 109 |
+
vllm serve ./KORMo-Web-9B --served-model-name KORMo-Web-9B \
|
| 110 |
+
--max-model-len 131072 --limit-mm-per-prompt '{"image":1,"video":0}'
|
| 111 |
+
```
|
| 112 |
+
|
| 113 |
+
**3. Run a browser task**
|
| 114 |
+
|
| 115 |
+
```bash
|
| 116 |
+
python KORMo-Web-9B/examples/agent_playwright.py --model KORMo-Web-9B \
|
| 117 |
+
--task "๋ค๋์์์ 50๋ง ์ ์ดํ, 10์ธ์น ์ด์ ํ๋ธ๋ฆฟ 3๊ฐ๋ฅผ ์ฐพ์์ ๊ฐ๊ฒฉ์ ์๋ ค์ค"
|
| 118 |
+
```
|
| 119 |
+
|
| 120 |
+
The agent opens Chromium, prints each action as it goes, and finishes with the model's answer.
|
| 121 |
+
|
| 122 |
+
<details>
|
| 123 |
+
<summary><b>Use the model from your own code (one step)</b></summary>
|
| 124 |
|
| 125 |
```python
|
| 126 |
import base64, json
|
| 127 |
from openai import OpenAI
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 128 |
|
| 129 |
+
SYSTEM = open("KORMo-Web-9B/system_prompt.md").read().strip()
|
| 130 |
+
TOOLS = json.load(open("KORMo-Web-9B/tools.json"))
|
| 131 |
shot = base64.b64encode(open("screen.png", "rb").read()).decode()
|
| 132 |
|
| 133 |
+
obs = ("<observation>\nscreen size: 1280 x 720\nCurrent tab: 0\nAvailable tabs:\n"
|
| 134 |
+
"- Tab 0 (active): https://www.google.com/\n\nurl: https://www.google.com/\nscreenshot:\n")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 135 |
messages = [
|
| 136 |
{"role": "system", "content": SYSTEM},
|
| 137 |
{"role": "user", "content": [
|
| 138 |
+
{"type": "text", "text": "<user_request>\n๊ตฌ๊ธ์์ KAIST ๋ฅผ ๊ฒ์ํด์ ๊ณต์ ํํ์ด์ง๋ฅผ ์ด์ด์ค.\n</user_request>\n" + obs},
|
| 139 |
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{shot}"}},
|
| 140 |
{"type": "text", "text": "\n</observation>"},
|
| 141 |
]},
|
| 142 |
]
|
| 143 |
+
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
| 144 |
out = client.chat.completions.create(
|
| 145 |
+
model="KORMo-Web-9B", messages=messages, tools=TOOLS,
|
| 146 |
temperature=0.6, top_p=0.95, max_tokens=16384,
|
| 147 |
+
extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": True}},
|
|
|
|
| 148 |
)
|
| 149 |
+
print(out.choices[0].message.content) # <think> โฆ </think> <tool_call><function=click> โฆ </tool_call>
|
| 150 |
```
|
| 151 |
|
| 152 |
+
Coordinates in the tool call are normalized to 0โ1000. Convert them to pixels with `x / 1000 ร width` and `y / 1000 ร height`.
|
| 153 |
+
</details>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 154 |
|
| 155 |
+
## How It Works
|
| 156 |
|
| 157 |
+
```
|
| 158 |
+
user <user_request> task </user_request> + observation (tabs, url, screenshot)
|
| 159 |
+
assistant <think> what I see ยท what is done ยท plan </think> <tool_call> one action </tool_call>
|
| 160 |
+
tool action result + new observation (newest screenshot only)
|
| 161 |
+
โฆ repeat until finished(answer)
|
|
|
|
|
|
|
| 162 |
```
|
| 163 |
|
| 164 |
+
- **Action space:** 22 browser tools: click, type, hotkey, scroll, drag, goto, tab control, `finished`, and others ([`tools.json`](tools.json)).
|
| 165 |
+
- **Memory:** the full text history of thoughts, actions and observations is kept. Only the current screenshot is sent.
|
| 166 |
+
- **Recommended settings:** temperature 0.6, top_p 0.95, top_k 20, thinking on, up to 16k tokens per step and 50 steps. These are already set in `generation_config.json`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 167 |
|
| 168 |
+
The exact message format, parsing rules and feedback strings are in **[HARNESS.md](HARNESS.md)**. Follow it if you build your own harness.
|
|
|
|
| 169 |
|
| 170 |
+
## Evaluation Details
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 171 |
|
| 172 |
+
| | |
|
| 173 |
+
|---|---|
|
| 174 |
+
| Benchmark | Online-Mind2Web, 300 English tasks, plus a Korean version with 312 tasks on Korean websites |
|
| 175 |
+
| Environment | Live websites, Chromium, 1280ร720 viewport for KORMo-Web-9B |
|
| 176 |
+
| Judge | One LLM judge for all models, using the WebVoyager judge prompt on the last 50 screenshots |
|
| 177 |
+
| Scoring | Success over the full task count; missing or failed runs count as failures |
|
| 178 |
+
| Budget | 50 steps (OpenWebRL-4B was run with its own 30-step setting) |
|
| 179 |
+
| Baselines | Each baseline used its own prompt, action format and viewport |
|
| 180 |
|
| 181 |
+
Live-web scores shift with site changes and bot protection, so all models in the table were run within the same evaluation campaign.
|
| 182 |
|
| 183 |
## Files
|
| 184 |
|
| 185 |
| File | Description |
|
| 186 |
|---|---|
|
| 187 |
+
| `chat_template.jinja` | Chat template (Qwen3.5 XML tool calls, thinking on) |
|
| 188 |
+
| `system_prompt.md` ยท `tools.json` | System prompt and 22-tool schema, verbatim from training |
|
| 189 |
+
| `HARNESS.md` | Full harness specification |
|
| 190 |
+
| `examples/agent_playwright.py` | Reference browser agent (Playwright + OpenAI-compatible API). It does not execute `upload`. |
|
| 191 |
+
| `config.json` ยท `generation_config.json` ยท `processor_config.json` ยท `tokenizer*.json` | Model, sampling and processor configs |
|
|
|
|
| 192 |
|
| 193 |
## ํ๊ตญ์ด ์์ฝ
|
| 194 |
|
| 195 |
+
**KORMo-Web-9B**๋ ์คํฌ๋ฆฐ์ท๋ง ๋ณด๊ณ ํ๊ตญ์ดยท์์ด ์น ์์
์ ์ํํ๋ 9B ๊ณต๊ฐ ์น ์์ด์ ํธ์
๋๋ค. ๊ฐ์ ์กฐ๊ฑด์ ์ค์ ์น ํ๊ฐ์์ ๋น๊ตํ ๊ณต๊ฐ ๋ชจ๋ธ ์ค ๊ฐ์ฅ ๋์ ์ ์๋ฅผ ๋ฐ์์ต๋๋ค.
|
| 196 |
|
| 197 |
+
- **Online-Mind2Web:** EN 51.0, KO 48.1
|
| 198 |
+
- **๋ฐํ ๋ชจ๋ธ(Qwen3.5-9B) ๋๋น:** EN +26.3, KO +31.4
|
| 199 |
+
- **๊ฐ์ฅ ๊ฐํ ๋ค๋ฅธ ๊ณต๊ฐ ๋ชจ๋ธ ๋๋น:** EN +18.3, KO +27.9
|
| 200 |
+
- ํ๊ตญ์ด์์ ์ฑ๋ฅ์ด ๊ฑฐ์ ๋จ์ด์ง์ง ์์ต๋๋ค(51.0 โ 48.1).
|
| 201 |
|
| 202 |
## Acknowledgements
|
| 203 |
|