KORMo-Web-11B / README.md
PerRing's picture
Replace demo video with gui_final_demo_maksed.mp4
b37ee51 verified
|
Raw History Blame Contribute Delete
13.8 kB
---
license: apache-2.0
base_model:
- KORMo-VL/KORMo-VL
language:
- ko
- en
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- gui-agent
- web-agent
- computer-use
- browser
- korean
---
<style>
#kormo-web-card { font-size: 16px; line-height: 1.6; overflow-wrap: break-word; }
#kormo-web-card h1 { margin: 0 0 12px; font-size: 30px; line-height: 1.25; letter-spacing: -0.018em; }
#kormo-web-card h2 { margin: 30px 0 10px; font-size: 22px; line-height: 1.3; letter-spacing: -0.012em; }
#kormo-web-card h3 { margin: 22px 0 8px; font-size: 17px; font-weight: 700; line-height: 1.4; }
#kormo-web-card h2 + h3 { margin-top: 14px; }
#kormo-web-card p { margin: 0 0 10px; }
#kormo-web-card ul, #kormo-web-card ol { margin: 6px 0 12px; padding-left: 22px; }
#kormo-web-card li { margin: 4px 0; line-height: 1.55; }
#kormo-web-card table { width: 100%; margin: 6px 0 14px; border-collapse: collapse; font-size: 14px; line-height: 1.45; font-variant-numeric: tabular-nums; }
#kormo-web-card th, #kormo-web-card td { padding: 7px 10px; vertical-align: middle; }
#kormo-web-card blockquote { margin: 12px 0 16px; padding: 0 0 0 14px; font-size: 15px; }
#kormo-web-card blockquote p { margin: 0; }
#kormo-web-card pre { margin: 10px 0 14px; padding: 14px 16px; overflow-x: auto; line-height: 1.5; }
#kormo-web-card pre code { font-size: 13px; }
#kormo-web-card > p:has(> em:only-child) { margin: 4px 0 14px; font-size: 13px; }
#kormo-web-card .pills { margin: 4px 0 18px; }
#kormo-web-card .pill { display: inline-block; margin: 0 6px 6px 0; padding: 3px 10px; border-radius: 999px; border: 1px solid rgba(127,127,127,.35); font-size: 13px; }
@media (max-width: 640px) {
#kormo-web-card h1 { font-size: 26px; }
#kormo-web-card table { font-size: 13px; }
#kormo-web-card th, #kormo-web-card td { padding: 7px 6px; }
}
</style>
<div id="kormo-web-card">
<video src="https://huggingface.co/KORMo-VL/KORMo-Web-11B/resolve/main/assets/gui_final_demo_maksed.mp4" controls autoplay muted loop playsinline width="100%"></video>
# KORMo-Web-11B
### One Model. Three Capabilities: LLM, VLM, and Web Agent.
**KORMo-Web-11B**๋Š” KAIST MLP ์—ฐ๊ตฌ์‹ค์ด ์ฒ˜์Œ๋ถ€ํ„ฐ ๊ตฌ์ถ•ํ•œ KORMo ํŒŒ์šด๋ฐ์ด์…˜ ๋ชจ๋ธ์„ ๊ธฐ๋ฐ˜์œผ๋กœ, **LLMยทVLMยทGUI Agent ๋Šฅ๋ ฅ์„ ๋ชจ๋‘ ๊ฐ–์ถ˜ ๋ฒ”์šฉ ๋ฉ€ํ‹ฐ๋ชจ๋‹ฌ AI ๋ชจ๋ธ**์ž…๋‹ˆ๋‹ค.
- **ํ•˜๋‚˜์˜ ๋ชจ๋ธ, ์„ธ ๊ฐ€์ง€ ๋Šฅ๋ ฅ:** ์ผ๋ฐ˜์ ์ธ ์งˆ๋ฌธ์—๋Š” LLM์œผ๋กœ ๋‹ต๋ณ€ํ•˜๊ณ , ์ด๋ฏธ์ง€ ์ดํ•ด๊ฐ€ ํ•„์š”ํ•œ ๊ฒฝ์šฐ VLM์œผ๋กœ, **์›น ์กฐ์ž‘์ด ํ•„์š”ํ•œ ์š”์ฒญ์—๋Š” GUI Agent๋กœ ์ž‘๋™**ํ•ฉ๋‹ˆ๋‹ค.
- **๋…์ž ํŒŒ์šด๋ฐ์ด์…˜ ๋ชจ๋ธ ๊ธฐ๋ฐ˜:** From-scratch๋กœ ํ•™์Šตํ•œ KORMo-10B๋ฅผ KORMo-VL๋กœ ํ™•์žฅํ•˜๊ณ , ๊ธฐ์กด ์–ธ์–ดยท์‹œ๊ฐ ์ดํ•ด ๋Šฅ๋ ฅ์„ ์œ ์ง€ํ•˜๋„๋ก ํ•™์Šตํ•œ ์›น ์—์ด์ „ํŠธ๋กœ **์ƒ๊ฐ\<Think\> ๋˜ํ•œ ์œ ์ผํ•˜๊ฒŒ ํ•œ๊ตญ์–ด**๋กœ ํ•˜๋Š” ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค.
- **๋Œ€๊ทœ๋ชจ ์›น ์‚ฌ์ „ํ•™์Šต:** 1,000๋งŒ ๊ฑด ์ด์ƒ์˜ ํ•œ์˜ ์›น ํ•™์Šต ์ƒ˜ํ”Œ๋กœ UI ์š”์†Œ์˜ ์œ„์น˜ยท์˜๋ฏธยท๊ตฌ์กฐยท๊ธฐ๋Šฅ ๋“ฑ์„ ์„ธ๋ฐ€ํ•˜๊ฒŒ ํ•™์Šตํ–ˆ์Šต๋‹ˆ๋‹ค.
- **์‹ค์ œ ์›น ํ–‰๋™ ํ•™์Šต:** 10๋งŒ ๊ฑด ์ด์ƒ์˜ ํ•œ์˜ ์›น ์‚ฌ์šฉ ๊ถค์ ์„ ํ™œ์šฉํ•ด ๋ณต์žกํ•œ ๋‹ค๋‹จ๊ณ„ ์›น ์ž‘์—…์„ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.
- **์™„์ „ ๊ณต๊ฐœ ๋ฐ ์ƒ์—…์  ํ™œ์šฉ:** ๋ชจ๋ธ ๊ฐ€์ค‘์น˜์™€ ํ•™์Šตยทํ‰๊ฐ€ ๋ฆฌ์†Œ์Šค๋ฅผ ๊ณต๊ฐœํ•˜๋ฉฐ, **Apache-2.0 ๋ผ์ด์„ ์Šค์— ๋”ฐ๋ผ ์ƒ์—…์  ์ด์šฉ๊ณผ ์ˆ˜์ •ยท์žฌ๋ฐฐํฌ๊ฐ€ ๊ฐ€๋Šฅ**ํ•ฉ๋‹ˆ๋‹ค.
**KORMo-Web-11B** is a general-purpose bilingual AI model built on the **KORMo foundation model family**, developed by KAIST MLP Lab starting from a language model trained entirely from scratch. Unlike our Qwen-based KORMo-Web-4B and 9B models, which focus on maximizing web task performance, **KORMo-Web-11B is designed to preserve its original LLM and VLM capabilities while learning to act as a GUI agent.** It can understand Korean and English text, interpret images, and complete real browser tasks by visually navigating webpagesโ€”**all within a single model.**
## What Makes KORMo-Web-11B Different?
- **Built on Our Own Foundation Model:** Developed from the KORMo-10B language model, trained from scratch by KAIST MLP Lab, and extended through KORMo-VL into a web agent.
- **Three Capabilities in One Model:** Designed to retain general **LLM reasoning and VLM understanding** while adding GUI interaction capabilities, rather than optimizing exclusively for web navigation.
- **10M+ Atomic-Level Web Pretraining Samples:** Large-scale Korean and English web understanding training, covering individual UI elements, their locations, layouts, functions, and states.
- **100K+ Web Interaction Trajectories:** Bilingual browser demonstrations for learning real-world, multi-step web tasks.
- **Open and Commercially Usable:** Model weights and supporting resources are openly released under **Apache-2.0**, allowing commercial use, modification, and redistribution subject to the license terms.
## From a Korean LLM to a General-Purpose Web Agent
KORMo-Web-11B extends our independently developed Koreanโ€“English foundation model through three stages:
| Stage | Model | Capabilities |
|---|---|---|
| 1. Language | [KORMo-10B](https://huggingface.co/KORMo-Team/KORMo-10B-sft) | Koreanโ€“English LLM trained from scratch with a Korean-optimized tokenizer |
| 2. Vision | [KORMo-VL](https://huggingface.co/KORMo-VL/KORMo-VL) | Multimodal understanding through a SigLIP vision encoder and KORMo language backbone |
| 3. Web Agent | **KORMo-Web-11B** | Large-scale web pretraining and action learning for visual browser interaction |
**The goal is not just a better web agent, but a versatile model that can understand language, interpret visual information, and interact with the web.**
## Performance
We evaluated KORMo-Web-11B on **612 live-website tasks** (300 English, 312 Korean) from Online-Mind2Web, using the same LLM judge across all evaluated models.
| Model | Size | English SR (%) | Korean SR (%) |
|---|---:|---:|---:|
| Qwen3.5-9B (base) | 9B | 24.7 | 16.7 |
| Fara-1.5-4B | 4B | 20.7 | 7.7 |
| UI-TARS-1.5-7B | 7B | 22.3 | 20.2 |
| OpenWebRL-4B | 4B | 32.7 | 15.4 |
| **KORMo-Web-11B (ours)** | **11B** | **29.0** | **30.1** |
| GPT-6-luna (proprietary) | โ€“ | 63.0 | 65.7 |
*Success rates are calculated over the full task set, including failed or incomplete runs. Each agent uses its own prompt and action format.*
KORMo-Web-11B achieves **30.1% Korean task success**, exceeding the strongest non-KORMo open-weight baseline listed above by **9.9 percentage points**.
While our specialized KORMo-Web-4B and 9B models achieve higher web task success, **KORMo-Web-11B prioritizes versatility across language, vision, and GUI interaction**, rather than web-only optimization.
## Why KORMo-Web-11B?
- **Korean from the Ground Up:** Built upon our independently pretrained Koreanโ€“English foundation model.
- **General-Purpose Multimodal AI:** Supports text-based language tasks, visual understanding, and browser interactions within one model.
- **Visual-First Web Interaction:** Operates primarily through screenshots without requiring full DOM or accessibility trees.
- **Think, Then Act:** Generates an inspectable reasoning step before executing browser actions.
- **Commercial Use Permitted:** Released under Apache-2.0, supporting academic research, commercial applications, fine-tuning, and redistribution.
- **Open and Ready to Run:** Includes model weights, Transformers 5 support, a vLLM inference path, browser harness, and reference inference settings.
**KORMo-Web-11B โ€” From a Korean Foundation Model to a General-Purpose Web Agent.**
## Quickstart
Tested with **transformers 5.12.1** and **vLLM 0.23.0**. The model runs on a single GPU.
transformers does not ship the KORMo text model, so this repo includes it (`_modeling_kormo.py`) together with a small loader, `kormo_load.py`:
- `kormo_load.register(path)` registers the KORMo text model. Call it **before** loading the processor or the model.
- `kormo_load.patch_vision(model)` feeds the SigLIP features *after* post-layernorm to the projector, as in training. Without it the model still runs but misreads screenshots.
### Inference example (one step, transformers)
```bash
pip install -U "transformers>=5.12" torch pillow huggingface_hub
huggingface-cli download KORMo-VL/KORMo-Web-11B --local-dir KORMo-Web-11B
```
```python
import json, sys, torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor
M = "KORMo-Web-11B"
sys.path.insert(0, M)
import kormo_load
kormo_load.register(M)
processor = AutoProcessor.from_pretrained(M)
model = AutoModelForImageTextToText.from_pretrained(M, dtype=torch.bfloat16).to("cuda").eval()
kormo_load.patch_vision(model)
SYSTEM = open(f"{M}/system_prompt.md").read().strip()
TOOLS = json.load(open(f"{M}/tools.json"))
obs = ("<observation>\nscreen size: 1280 x 720\nCurrent tab: 0\nAvailable tabs:\n"
"- Tab 0 (active): https://www.aladin.co.kr/\n\nurl: https://www.aladin.co.kr/\nscreenshot:\n")
messages = [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": [
{"type": "text", "text": "<user_request>\n์•Œ๋ผ๋”˜ ์‚ฌ์ดํŠธ์—์„œ 'ํ•ด์ปค์Šค ํ† ์ต RC'๋ฅผ ๊ฒ€์ƒ‰ํ•œ ๋’ค ๊ฐ€์žฅ ๋จผ์ € ํ‘œ์‹œ๋˜๋Š” ๋„์„œ์˜ ์ถœ๊ฐ„์ผ์„ ์•Œ๋ ค์ค˜.\n</user_request>\n" + obs},
{"type": "image", "image": Image.open("screen.png").convert("RGB")},
{"type": "text", "text": "\n</observation>"},
]},
]
inputs = processor.apply_chat_template(messages, tools=TOOLS, add_generation_prompt=True,
tokenize=True, return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=16384, do_sample=True, temperature=0.6, top_p=0.95, top_k=20)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))
```
A real first step from our Online-Mind2Web run (Aladin home page):
```text
<think>I see the Aladin homepage, with the search field labeled โ€œํ†ตํ•ฉ๊ฒ€์ƒ‰โ€ near the top. I have not yet searched for the book, so I still need to enter โ€œํ•ด์ปค์Šค ํ† ์ต RC,โ€ inspect the first resultโ€™s publication date, and report that date. My plan:
1. Click the search field
2. Enter โ€œํ•ด์ปค์Šค ํ† ์ต RCโ€ and submit the search
3. Read the publication date of the first listed book
This click targets the visible search field and is a safe, relevant first action.
</think>
<tool_call>
<function=click>
<parameter=x>
482
</parameter>
<parameter=y>
212
</parameter>
</function>
</tool_call>
```
Coordinates are normalized to 0โ€“1000. Convert them to pixels with `x / 1000 ร— width` and `y / 1000 ร— height`.
### Run the agent (vLLM)
vLLM does not include the KORMo architecture either. The KORMo text layers have the Llama structure, so `vllm/convert_to_vllm.py` renames two parameter names and writes a folder vLLM loads natively. `vllm/sitecustomize.py` turns SigLIP post-layernorm back on inside vLLM.
**1. Convert (once)**
```bash
pip install -U "vllm>=0.23.0" openai playwright safetensors
playwright install chromium
python KORMo-Web-11B/vllm/convert_to_vllm.py KORMo-Web-11B KORMo-Web-11B-vllm
```
**2. Start the model server**
```bash
PYTHONPATH=$PWD/KORMo-Web-11B/vllm vllm serve ./KORMo-Web-11B-vllm --served-model-name KORMo-Web-11B \
--max-model-len 65536 --limit-mm-per-prompt '{"image":1,"video":0}'
```
Check that the server log prints `[kormo-vllm] SigLIP post_layernorm enabled`.
**3. Run a browser task**
```bash
python KORMo-Web-11B/examples/agent_playwright.py --model KORMo-Web-11B \
--task "๋‹ค๋‚˜์™€์—์„œ 50๋งŒ ์› ์ดํ•˜, 10์ธ์น˜ ์ด์ƒ ํƒœ๋ธ”๋ฆฟ 3๊ฐœ๋ฅผ ์ฐพ์•„์„œ ๊ฐ€๊ฒฉ์„ ์•Œ๋ ค์ค˜"
```
The agent opens Chromium, prints each action as it goes, and finishes with the model's answer.
## How It Works
```
user <user_request> task </user_request> + observation (tabs, url, screenshot)
assistant <think> what I see ยท what is done ยท plan </think> <tool_call> one action </tool_call>
tool action result + new observation (newest screenshot only)
โ€ฆ repeat until finished(answer)
```
- **Action space:** 22 browser tools: click, type, hotkey, scroll, drag, goto, tab control, `finished`, and others ([`tools.json`](tools.json)).
- **Memory:** the full text history of thoughts, actions and observations is kept. Only the current screenshot is sent.
- **Recommended settings:** temperature 0.6, top_p 0.95, top_k 20, thinking on, up to 16k tokens per step and 50 steps. These are already set in `generation_config.json`.
The exact message format, parsing rules and feedback strings are in **[HARNESS.md](HARNESS.md)**. Follow it if you build your own harness.
## Evaluation Details
| | |
|---|---|
| Tasks | Online-Mind2Web: 300 English, 312 Korean |
| Judge | GPT-6 |
## Files
| File | Description |
|---|---|
| `model.safetensors` ยท `config.json` | Weights and config (LLaVA-OneVision: SigLIP + KORMo-10B text model) |
| `_configuration_kormo.py` ยท `_modeling_kormo.py` | KORMo text model code for transformers 5 |
| `kormo_load.py` | `register()` and `patch_vision()` |
| `chat_template.jinja` | Chat template (XML tool calls, thinking on) |
| `system_prompt.md` ยท `tools.json` | System prompt and 22-tool schema |
| `HARNESS.md` | Full harness specification |
| `examples/agent_playwright.py` | Reference browser agent (Playwright + OpenAI-compatible API). It does not execute `upload`. |
| `vllm/convert_to_vllm.py` ยท `vllm/sitecustomize.py` | Convert for vLLM and the post-layernorm fix |
| `generation_config.json` ยท `*processor_config.json` ยท `tokenizer*.json` ยท `special_tokens_map.json` | Sampling, processor and tokenizer configs |
## Acknowledgements
Built on **[KORMo-VL](https://huggingface.co/KORMo-VL/KORMo-VL)** and **[KORMo-10B](https://huggingface.co/KORMo-Team/KORMo-10B-sft)** by KAIST MLP Lab. Evaluated on **Online-Mind2Web** and its Korean counterpart.
</div>