File size: 14,151 Bytes
9da837f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
b37ee51
a5c05d6
9da837f
31866f8
9da837f
0676e9f
9da837f
0676e9f
9da837f
0676e9f
 
 
 
 
9da837f
 
0676e9f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9da837f
0676e9f
 
 
9da837f
0676e9f
9da837f
 
 
0676e9f
9da837f
54c629e
9da837f
0676e9f
9da837f
 
 
31866f8
9da837f
 
0676e9f
9da837f
0676e9f
9da837f
0676e9f
 
 
 
 
 
 
 
 
 
 
 
9da837f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31866f8
9da837f
 
 
 
 
 
 
31866f8
9da837f
 
 
 
5636a20
9da837f
 
 
 
 
5636a20
9da837f
 
 
5636a20
9da837f
 
 
 
 
 
 
 
 
 
5636a20
9da837f
 
5636a20
 
 
 
 
9da837f
 
 
 
 
5636a20
9da837f
 
5636a20
9da837f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31866f8
9da837f
 
 
 
 
31866f8
9da837f
 
 
 
 
 
 
 
31866f8
9da837f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2614b60
 
 
 
 
 
 
 
 
 
 
 
 
9da837f
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
---
license: apache-2.0
base_model:
- KORMo-VL/KORMo-VL
language:
- ko
- en
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- gui-agent
- web-agent
- computer-use
- browser
- korean
---

<style>
#kormo-web-card { font-size: 16px; line-height: 1.6; overflow-wrap: break-word; }
#kormo-web-card h1 { margin: 0 0 12px; font-size: 30px; line-height: 1.25; letter-spacing: -0.018em; }
#kormo-web-card h2 { margin: 30px 0 10px; font-size: 22px; line-height: 1.3; letter-spacing: -0.012em; }
#kormo-web-card h3 { margin: 22px 0 8px; font-size: 17px; font-weight: 700; line-height: 1.4; }
#kormo-web-card h2 + h3 { margin-top: 14px; }
#kormo-web-card p { margin: 0 0 10px; }
#kormo-web-card ul, #kormo-web-card ol { margin: 6px 0 12px; padding-left: 22px; }
#kormo-web-card li { margin: 4px 0; line-height: 1.55; }
#kormo-web-card table { width: 100%; margin: 6px 0 14px; border-collapse: collapse; font-size: 14px; line-height: 1.45; font-variant-numeric: tabular-nums; }
#kormo-web-card th, #kormo-web-card td { padding: 7px 10px; vertical-align: middle; }
#kormo-web-card blockquote { margin: 12px 0 16px; padding: 0 0 0 14px; font-size: 15px; }
#kormo-web-card blockquote p { margin: 0; }
#kormo-web-card pre { margin: 10px 0 14px; padding: 14px 16px; overflow-x: auto; line-height: 1.5; }
#kormo-web-card pre code { font-size: 13px; }
#kormo-web-card > p:has(> em:only-child) { margin: 4px 0 14px; font-size: 13px; }
#kormo-web-card .pills { margin: 4px 0 18px; }
#kormo-web-card .pill { display: inline-block; margin: 0 6px 6px 0; padding: 3px 10px; border-radius: 999px; border: 1px solid rgba(127,127,127,.35); font-size: 13px; }
@media (max-width: 640px) {
  #kormo-web-card h1 { font-size: 26px; }
  #kormo-web-card table { font-size: 13px; }
  #kormo-web-card th, #kormo-web-card td { padding: 7px 6px; }
}
</style>

<div id="kormo-web-card">
<video src="https://huggingface.co/KORMo-VL/KORMo-Web-11B/resolve/main/assets/gui_final_demo_maksed.mp4" controls autoplay muted loop playsinline width="100%"></video>


# KORMo-Web-11B

### One Model. Three Capabilities: LLM, VLM, and Web Agent.

**KORMo-Web-11B**๋Š” KAIST MLP ์—ฐ๊ตฌ์‹ค์ด ์ฒ˜์Œ๋ถ€ํ„ฐ ๊ตฌ์ถ•ํ•œ KORMo ํŒŒ์šด๋ฐ์ด์…˜ ๋ชจ๋ธ์„ ๊ธฐ๋ฐ˜์œผ๋กœ, **LLMยทVLMยทGUI Agent ๋Šฅ๋ ฅ์„ ๋ชจ๋‘ ๊ฐ–์ถ˜ ๋ฒ”์šฉ ๋ฉ€ํ‹ฐ๋ชจ๋‹ฌ AI ๋ชจ๋ธ**์ž…๋‹ˆ๋‹ค.

- **ํ•˜๋‚˜์˜ ๋ชจ๋ธ, ์„ธ ๊ฐ€์ง€ ๋Šฅ๋ ฅ:** ์ผ๋ฐ˜์ ์ธ ์งˆ๋ฌธ์—๋Š” LLM์œผ๋กœ ๋‹ต๋ณ€ํ•˜๊ณ , ์ด๋ฏธ์ง€ ์ดํ•ด๊ฐ€ ํ•„์š”ํ•œ ๊ฒฝ์šฐ VLM์œผ๋กœ, **์›น ์กฐ์ž‘์ด ํ•„์š”ํ•œ ์š”์ฒญ์—๋Š” GUI Agent๋กœ ์ž‘๋™**ํ•ฉ๋‹ˆ๋‹ค.
- **๋…์ž ํŒŒ์šด๋ฐ์ด์…˜ ๋ชจ๋ธ ๊ธฐ๋ฐ˜:** From-scratch๋กœ ํ•™์Šตํ•œ KORMo-10B๋ฅผ KORMo-VL๋กœ ํ™•์žฅํ•˜๊ณ , ๊ธฐ์กด ์–ธ์–ดยท์‹œ๊ฐ ์ดํ•ด ๋Šฅ๋ ฅ์„ ์œ ์ง€ํ•˜๋„๋ก ํ•™์Šตํ•œ ์›น ์—์ด์ „ํŠธ๋กœ **์ƒ๊ฐ\<Think\> ๋˜ํ•œ ์œ ์ผํ•˜๊ฒŒ ํ•œ๊ตญ์–ด**๋กœ ํ•˜๋Š” ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค.
- **๋Œ€๊ทœ๋ชจ ์›น ์‚ฌ์ „ํ•™์Šต:** 1,000๋งŒ ๊ฑด ์ด์ƒ์˜ ํ•œ์˜ ์›น ํ•™์Šต ์ƒ˜ํ”Œ๋กœ UI ์š”์†Œ์˜ ์œ„์น˜ยท์˜๋ฏธยท๊ตฌ์กฐยท๊ธฐ๋Šฅ ๋“ฑ์„ ์„ธ๋ฐ€ํ•˜๊ฒŒ ํ•™์Šตํ–ˆ์Šต๋‹ˆ๋‹ค.
- **์‹ค์ œ ์›น ํ–‰๋™ ํ•™์Šต:** 10๋งŒ ๊ฑด ์ด์ƒ์˜ ํ•œ์˜ ์›น ์‚ฌ์šฉ ๊ถค์ ์„ ํ™œ์šฉํ•ด ๋ณต์žกํ•œ ๋‹ค๋‹จ๊ณ„ ์›น ์ž‘์—…์„ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.
- **์™„์ „ ๊ณต๊ฐœ ๋ฐ ์ƒ์—…์  ํ™œ์šฉ:** ๋ชจ๋ธ ๊ฐ€์ค‘์น˜์™€ ํ•™์Šตยทํ‰๊ฐ€ ๋ฆฌ์†Œ์Šค๋ฅผ ๊ณต๊ฐœํ•˜๋ฉฐ, **Apache-2.0 ๋ผ์ด์„ ์Šค์— ๋”ฐ๋ผ ์ƒ์—…์  ์ด์šฉ๊ณผ ์ˆ˜์ •ยท์žฌ๋ฐฐํฌ๊ฐ€ ๊ฐ€๋Šฅ**ํ•ฉ๋‹ˆ๋‹ค.


**KORMo-Web-11B** is a general-purpose bilingual AI model built on the **KORMo foundation model family**, developed by KAIST MLP Lab starting from a language model trained entirely from scratch. Unlike our Qwen-based KORMo-Web-4B and 9B models, which focus on maximizing web task performance, **KORMo-Web-11B is designed to preserve its original LLM and VLM capabilities while learning to act as a GUI agent.** It can understand Korean and English text, interpret images, and complete real browser tasks by visually navigating webpagesโ€”**all within a single model.**

## What Makes KORMo-Web-11B Different?

- **Built on Our Own Foundation Model:** Developed from the KORMo-10B language model, trained from scratch by KAIST MLP Lab, and extended through KORMo-VL into a web agent.
- **Three Capabilities in One Model:** Designed to retain general **LLM reasoning and VLM understanding** while adding GUI interaction capabilities, rather than optimizing exclusively for web navigation.
- **10M+ Atomic-Level Web Pretraining Samples:** Large-scale Korean and English web understanding training, covering individual UI elements, their locations, layouts, functions, and states.
- **100K+ Web Interaction Trajectories:** Bilingual browser demonstrations for learning real-world, multi-step web tasks.
- **Open and Commercially Usable:** Model weights and supporting resources are openly released under **Apache-2.0**, allowing commercial use, modification, and redistribution subject to the license terms.

## From a Korean LLM to a General-Purpose Web Agent

KORMo-Web-11B extends our independently developed Koreanโ€“English foundation model through three stages:

| Stage | Model | Capabilities |
|---|---|---|
| 1. Language | [KORMo-10B](https://huggingface.co/KORMo-Team/KORMo-10B-sft) | Koreanโ€“English LLM trained from scratch with a Korean-optimized tokenizer |
| 2. Vision | [KORMo-VL](https://huggingface.co/KORMo-VL/KORMo-VL) | Multimodal understanding through a SigLIP vision encoder and KORMo language backbone |
| 3. Web Agent | **KORMo-Web-11B** | Large-scale web pretraining and action learning for visual browser interaction |

**The goal is not just a better web agent, but a versatile model that can understand language, interpret visual information, and interact with the web.**

## Performance

We evaluated KORMo-Web-11B on **612 live-website tasks** (300 English, 312 Korean) from Online-Mind2Web, using the same LLM judge across all evaluated models.

| Model | Size | Online-Mind2Web EN | Online-Mind2Web KO |
|---|---:|---:|---:|
| Qwen3.5-9B (base) | 9B | 24.7 | 16.7 |
| Fara-1.5-4B | 4B | 20.7 | 7.7 |
| UI-TARS-1.5-7B | 7B | 22.3 | 20.2 |
| OpenWebRL-4B | 4B | 32.7 | 15.4 |
| **KORMo-Web-11B (ours)** | **11B** | **29.0** | **30.1** |
| GPT-6-luna (proprietary) | โ€“ | 63.0 | 65.7 |

*Success rates are calculated over the full task set, including failed or incomplete runs. Each agent uses its own prompt and action format.*

KORMo-Web-11B achieves **30.1% Korean task success**, exceeding the strongest non-KORMo open-weight baseline listed above by **9.9 percentage points**.

While our specialized KORMo-Web-4B and 9B models achieve higher web task success, **KORMo-Web-11B prioritizes versatility across language, vision, and GUI interaction**, rather than web-only optimization.

## Why KORMo-Web-11B?

- **Korean from the Ground Up:** Built upon our independently pretrained Koreanโ€“English foundation model.
- **General-Purpose Multimodal AI:** Supports text-based language tasks, visual understanding, and browser interactions within one model.
- **Visual-First Web Interaction:** Operates primarily through screenshots without requiring full DOM or accessibility trees.
- **Think, Then Act:** Generates an inspectable reasoning step before executing browser actions.
- **Commercial Use Permitted:** Released under Apache-2.0, supporting academic research, commercial applications, fine-tuning, and redistribution.
- **Open and Ready to Run:** Includes model weights, Transformers 5 support, a vLLM inference path, browser harness, and reference inference settings.

**KORMo-Web-11B โ€” From a Korean Foundation Model to a General-Purpose Web Agent.**



## Quickstart

Tested with **transformers 5.12.1** and **vLLM 0.23.0**. The model runs on a single GPU.

transformers does not ship the KORMo text model, so this repo includes it (`_modeling_kormo.py`) together with a small loader, `kormo_load.py`:

- `kormo_load.register(path)` registers the KORMo text model. Call it **before** loading the processor or the model.
- `kormo_load.patch_vision(model)` feeds the SigLIP features *after* post-layernorm to the projector, as in training. Without it the model still runs but misreads screenshots.

### Inference example (one step, transformers)

```bash
pip install -U "transformers>=5.12" torch pillow huggingface_hub
huggingface-cli download KORMo-VL/KORMo-Web-11B --local-dir KORMo-Web-11B
```

```python
import json, sys, torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

M = "KORMo-Web-11B"
sys.path.insert(0, M)
import kormo_load
kormo_load.register(M)
processor = AutoProcessor.from_pretrained(M)
model = AutoModelForImageTextToText.from_pretrained(M, dtype=torch.bfloat16).to("cuda").eval()
kormo_load.patch_vision(model)

SYSTEM = open(f"{M}/system_prompt.md").read().strip()
TOOLS = json.load(open(f"{M}/tools.json"))
obs = ("<observation>\nscreen size: 1280 x 720\nCurrent tab: 0\nAvailable tabs:\n"
       "- Tab 0 (active): https://www.aladin.co.kr/\n\nurl: https://www.aladin.co.kr/\nscreenshot:\n")
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": [
        {"type": "text", "text": "<user_request>\n์•Œ๋ผ๋”˜ ์‚ฌ์ดํŠธ์—์„œ 'ํ•ด์ปค์Šค ํ† ์ต RC'๋ฅผ ๊ฒ€์ƒ‰ํ•œ ๋’ค ๊ฐ€์žฅ ๋จผ์ € ํ‘œ์‹œ๋˜๋Š” ๋„์„œ์˜ ์ถœ๊ฐ„์ผ์„ ์•Œ๋ ค์ค˜.\n</user_request>\n" + obs},
        {"type": "image", "image": Image.open("screen.png").convert("RGB")},
        {"type": "text", "text": "\n</observation>"},
    ]},
]
inputs = processor.apply_chat_template(messages, tools=TOOLS, add_generation_prompt=True,
                                       tokenize=True, return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=16384, do_sample=True, temperature=0.6, top_p=0.95, top_k=20)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))
```

A real first step from our Online-Mind2Web run (Aladin home page):

```text
<think>I see the Aladin homepage, with the search field labeled โ€œํ†ตํ•ฉ๊ฒ€์ƒ‰โ€ near the top. I have not yet searched for the book, so I still need to enter โ€œํ•ด์ปค์Šค ํ† ์ต RC,โ€ inspect the first resultโ€™s publication date, and report that date. My plan:
1. Click the search field
2. Enter โ€œํ•ด์ปค์Šค ํ† ์ต RCโ€ and submit the search
3. Read the publication date of the first listed book
This click targets the visible search field and is a safe, relevant first action.
</think>

<tool_call>
<function=click>
<parameter=x>
482
</parameter>
<parameter=y>
212
</parameter>
</function>
</tool_call>
```

Coordinates are normalized to 0โ€“1000. Convert them to pixels with `x / 1000 ร— width` and `y / 1000 ร— height`.

### Run the agent (vLLM)

vLLM does not include the KORMo architecture either. The KORMo text layers have the Llama structure, so `vllm/convert_to_vllm.py` renames two parameter names and writes a folder vLLM loads natively. `vllm/sitecustomize.py` turns SigLIP post-layernorm back on inside vLLM.

**1. Convert (once)**

```bash
pip install -U "vllm>=0.23.0" openai playwright safetensors
playwright install chromium
python KORMo-Web-11B/vllm/convert_to_vllm.py KORMo-Web-11B KORMo-Web-11B-vllm
```

**2. Start the model server**

```bash
PYTHONPATH=$PWD/KORMo-Web-11B/vllm vllm serve ./KORMo-Web-11B-vllm --served-model-name KORMo-Web-11B \
  --max-model-len 65536 --limit-mm-per-prompt '{"image":1,"video":0}'
```

Check that the server log prints `[kormo-vllm] SigLIP post_layernorm enabled`.

**3. Run a browser task**

```bash
python KORMo-Web-11B/examples/agent_playwright.py --model KORMo-Web-11B \
  --task "๋‹ค๋‚˜์™€์—์„œ 50๋งŒ ์› ์ดํ•˜, 10์ธ์น˜ ์ด์ƒ ํƒœ๋ธ”๋ฆฟ 3๊ฐœ๋ฅผ ์ฐพ์•„์„œ ๊ฐ€๊ฒฉ์„ ์•Œ๋ ค์ค˜"
```

The agent opens Chromium, prints each action as it goes, and finishes with the model's answer.

## How It Works

```
user       <user_request> task </user_request>  + observation (tabs, url, screenshot)
assistant  <think> what I see ยท what is done ยท plan </think>  <tool_call> one action </tool_call>
tool       action result  + new observation (newest screenshot only)
โ€ฆ          repeat until finished(answer)
```

- **Action space:** 22 browser tools: click, type, hotkey, scroll, drag, goto, tab control, `finished`, and others ([`tools.json`](tools.json)).
- **Memory:** the full text history of thoughts, actions and observations is kept. Only the current screenshot is sent.
- **Recommended settings:** temperature 0.6, top_p 0.95, top_k 20, thinking on, up to 16k tokens per step and 50 steps. These are already set in `generation_config.json`.

The exact message format, parsing rules and feedback strings are in **[HARNESS.md](HARNESS.md)**. Follow it if you build your own harness.

## Evaluation Details

| | |
|---|---|
| Tasks | Online-Mind2Web: 300 English, 312 Korean |
| Judge | GPT-6 |

## Files

| File | Description |
|---|---|
| `model.safetensors` ยท `config.json` | Weights and config (LLaVA-OneVision: SigLIP + KORMo-10B text model) |
| `_configuration_kormo.py` ยท `_modeling_kormo.py` | KORMo text model code for transformers 5 |
| `kormo_load.py` | `register()` and `patch_vision()` |
| `chat_template.jinja` | Chat template (XML tool calls, thinking on) |
| `system_prompt.md` ยท `tools.json` | System prompt and 22-tool schema |
| `HARNESS.md` | Full harness specification |
| `examples/agent_playwright.py` | Reference browser agent (Playwright + OpenAI-compatible API). It does not execute `upload`. |
| `vllm/convert_to_vllm.py` ยท `vllm/sitecustomize.py` | Convert for vLLM and the post-layernorm fix |
| `generation_config.json` ยท `*processor_config.json` ยท `tokenizer*.json` ยท `special_tokens_map.json` | Sampling, processor and tokenizer configs |


## Acknowledgements

Built on **[KORMo-VL](https://huggingface.co/KORMo-VL/KORMo-VL)** and **[KORMo-10B](https://huggingface.co/KORMo-Team/KORMo-10B-sft)** by KAIST MLP Lab. Evaluated on **Online-Mind2Web** and its Korean counterpart.

## Contact
- KyungTae Lim, Professor at KAIST. `ktlim@kaist.ac.kr`

## Contributor
- Hangyeol Yoo (hg.yoo@kaist.ac.kr)
- Minkyu Kim (minkyukim@kaist.ac.kr)
- Minkyung Cho (minkyung.cho@kaist.ac.kr)
- Yerin Nam (yerinnam@kaist.ac.kr)
- Seungwoo Song (sswoo@kaist.ac.kr)
- Gyun Lee (gyunini@kaist.ac.kr)
- KyungTae Lim (ktlim@kaist.ac.kr)


</div>