Image-Text-to-Text
Transformers
Safetensors
Korean
English
llava_onevision
gui-agent
web-agent
computer-use
browser
korean
conversational
Instructions to use KORMo-VL/KORMo-Web-11B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KORMo-VL/KORMo-Web-11B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="KORMo-VL/KORMo-Web-11B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("KORMo-VL/KORMo-Web-11B") model = AutoModelForMultimodalLM.from_pretrained("KORMo-VL/KORMo-Web-11B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use KORMo-VL/KORMo-Web-11B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "KORMo-VL/KORMo-Web-11B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-11B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/KORMo-VL/KORMo-Web-11B
- SGLang
How to use KORMo-VL/KORMo-Web-11B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-11B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-11B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-11B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-11B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use KORMo-VL/KORMo-Web-11B with Docker Model Runner:
docker model run hf.co/KORMo-VL/KORMo-Web-11B
File size: 14,151 Bytes
9da837f b37ee51 a5c05d6 9da837f 31866f8 9da837f 0676e9f 9da837f 0676e9f 9da837f 0676e9f 9da837f 0676e9f 9da837f 0676e9f 9da837f 0676e9f 9da837f 0676e9f 9da837f 54c629e 9da837f 0676e9f 9da837f 31866f8 9da837f 0676e9f 9da837f 0676e9f 9da837f 0676e9f 9da837f 31866f8 9da837f 31866f8 9da837f 5636a20 9da837f 5636a20 9da837f 5636a20 9da837f 5636a20 9da837f 5636a20 9da837f 5636a20 9da837f 5636a20 9da837f 31866f8 9da837f 31866f8 9da837f 31866f8 9da837f 2614b60 9da837f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 | ---
license: apache-2.0
base_model:
- KORMo-VL/KORMo-VL
language:
- ko
- en
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- gui-agent
- web-agent
- computer-use
- browser
- korean
---
<style>
#kormo-web-card { font-size: 16px; line-height: 1.6; overflow-wrap: break-word; }
#kormo-web-card h1 { margin: 0 0 12px; font-size: 30px; line-height: 1.25; letter-spacing: -0.018em; }
#kormo-web-card h2 { margin: 30px 0 10px; font-size: 22px; line-height: 1.3; letter-spacing: -0.012em; }
#kormo-web-card h3 { margin: 22px 0 8px; font-size: 17px; font-weight: 700; line-height: 1.4; }
#kormo-web-card h2 + h3 { margin-top: 14px; }
#kormo-web-card p { margin: 0 0 10px; }
#kormo-web-card ul, #kormo-web-card ol { margin: 6px 0 12px; padding-left: 22px; }
#kormo-web-card li { margin: 4px 0; line-height: 1.55; }
#kormo-web-card table { width: 100%; margin: 6px 0 14px; border-collapse: collapse; font-size: 14px; line-height: 1.45; font-variant-numeric: tabular-nums; }
#kormo-web-card th, #kormo-web-card td { padding: 7px 10px; vertical-align: middle; }
#kormo-web-card blockquote { margin: 12px 0 16px; padding: 0 0 0 14px; font-size: 15px; }
#kormo-web-card blockquote p { margin: 0; }
#kormo-web-card pre { margin: 10px 0 14px; padding: 14px 16px; overflow-x: auto; line-height: 1.5; }
#kormo-web-card pre code { font-size: 13px; }
#kormo-web-card > p:has(> em:only-child) { margin: 4px 0 14px; font-size: 13px; }
#kormo-web-card .pills { margin: 4px 0 18px; }
#kormo-web-card .pill { display: inline-block; margin: 0 6px 6px 0; padding: 3px 10px; border-radius: 999px; border: 1px solid rgba(127,127,127,.35); font-size: 13px; }
@media (max-width: 640px) {
#kormo-web-card h1 { font-size: 26px; }
#kormo-web-card table { font-size: 13px; }
#kormo-web-card th, #kormo-web-card td { padding: 7px 6px; }
}
</style>
<div id="kormo-web-card">
<video src="https://huggingface.co/KORMo-VL/KORMo-Web-11B/resolve/main/assets/gui_final_demo_maksed.mp4" controls autoplay muted loop playsinline width="100%"></video>
# KORMo-Web-11B
### One Model. Three Capabilities: LLM, VLM, and Web Agent.
**KORMo-Web-11B**๋ KAIST MLP ์ฐ๊ตฌ์ค์ด ์ฒ์๋ถํฐ ๊ตฌ์ถํ KORMo ํ์ด๋ฐ์ด์
๋ชจ๋ธ์ ๊ธฐ๋ฐ์ผ๋ก, **LLMยทVLMยทGUI Agent ๋ฅ๋ ฅ์ ๋ชจ๋ ๊ฐ์ถ ๋ฒ์ฉ ๋ฉํฐ๋ชจ๋ฌ AI ๋ชจ๋ธ**์
๋๋ค.
- **ํ๋์ ๋ชจ๋ธ, ์ธ ๊ฐ์ง ๋ฅ๋ ฅ:** ์ผ๋ฐ์ ์ธ ์ง๋ฌธ์๋ LLM์ผ๋ก ๋ต๋ณํ๊ณ , ์ด๋ฏธ์ง ์ดํด๊ฐ ํ์ํ ๊ฒฝ์ฐ VLM์ผ๋ก, **์น ์กฐ์์ด ํ์ํ ์์ฒญ์๋ GUI Agent๋ก ์๋**ํฉ๋๋ค.
- **๋
์ ํ์ด๋ฐ์ด์
๋ชจ๋ธ ๊ธฐ๋ฐ:** From-scratch๋ก ํ์ตํ KORMo-10B๋ฅผ KORMo-VL๋ก ํ์ฅํ๊ณ , ๊ธฐ์กด ์ธ์ดยท์๊ฐ ์ดํด ๋ฅ๋ ฅ์ ์ ์งํ๋๋ก ํ์ตํ ์น ์์ด์ ํธ๋ก **์๊ฐ\<Think\> ๋ํ ์ ์ผํ๊ฒ ํ๊ตญ์ด**๋ก ํ๋ ๋ชจ๋ธ์
๋๋ค.
- **๋๊ท๋ชจ ์น ์ฌ์ ํ์ต:** 1,000๋ง ๊ฑด ์ด์์ ํ์ ์น ํ์ต ์ํ๋ก UI ์์์ ์์นยท์๋ฏธยท๊ตฌ์กฐยท๊ธฐ๋ฅ ๋ฑ์ ์ธ๋ฐํ๊ฒ ํ์ตํ์ต๋๋ค.
- **์ค์ ์น ํ๋ ํ์ต:** 10๋ง ๊ฑด ์ด์์ ํ์ ์น ์ฌ์ฉ ๊ถค์ ์ ํ์ฉํด ๋ณต์กํ ๋ค๋จ๊ณ ์น ์์
์ ์ํํฉ๋๋ค.
- **์์ ๊ณต๊ฐ ๋ฐ ์์
์ ํ์ฉ:** ๋ชจ๋ธ ๊ฐ์ค์น์ ํ์ตยทํ๊ฐ ๋ฆฌ์์ค๋ฅผ ๊ณต๊ฐํ๋ฉฐ, **Apache-2.0 ๋ผ์ด์ ์ค์ ๋ฐ๋ผ ์์
์ ์ด์ฉ๊ณผ ์์ ยท์ฌ๋ฐฐํฌ๊ฐ ๊ฐ๋ฅ**ํฉ๋๋ค.
**KORMo-Web-11B** is a general-purpose bilingual AI model built on the **KORMo foundation model family**, developed by KAIST MLP Lab starting from a language model trained entirely from scratch. Unlike our Qwen-based KORMo-Web-4B and 9B models, which focus on maximizing web task performance, **KORMo-Web-11B is designed to preserve its original LLM and VLM capabilities while learning to act as a GUI agent.** It can understand Korean and English text, interpret images, and complete real browser tasks by visually navigating webpagesโ**all within a single model.**
## What Makes KORMo-Web-11B Different?
- **Built on Our Own Foundation Model:** Developed from the KORMo-10B language model, trained from scratch by KAIST MLP Lab, and extended through KORMo-VL into a web agent.
- **Three Capabilities in One Model:** Designed to retain general **LLM reasoning and VLM understanding** while adding GUI interaction capabilities, rather than optimizing exclusively for web navigation.
- **10M+ Atomic-Level Web Pretraining Samples:** Large-scale Korean and English web understanding training, covering individual UI elements, their locations, layouts, functions, and states.
- **100K+ Web Interaction Trajectories:** Bilingual browser demonstrations for learning real-world, multi-step web tasks.
- **Open and Commercially Usable:** Model weights and supporting resources are openly released under **Apache-2.0**, allowing commercial use, modification, and redistribution subject to the license terms.
## From a Korean LLM to a General-Purpose Web Agent
KORMo-Web-11B extends our independently developed KoreanโEnglish foundation model through three stages:
| Stage | Model | Capabilities |
|---|---|---|
| 1. Language | [KORMo-10B](https://huggingface.co/KORMo-Team/KORMo-10B-sft) | KoreanโEnglish LLM trained from scratch with a Korean-optimized tokenizer |
| 2. Vision | [KORMo-VL](https://huggingface.co/KORMo-VL/KORMo-VL) | Multimodal understanding through a SigLIP vision encoder and KORMo language backbone |
| 3. Web Agent | **KORMo-Web-11B** | Large-scale web pretraining and action learning for visual browser interaction |
**The goal is not just a better web agent, but a versatile model that can understand language, interpret visual information, and interact with the web.**
## Performance
We evaluated KORMo-Web-11B on **612 live-website tasks** (300 English, 312 Korean) from Online-Mind2Web, using the same LLM judge across all evaluated models.
| Model | Size | Online-Mind2Web EN | Online-Mind2Web KO |
|---|---:|---:|---:|
| Qwen3.5-9B (base) | 9B | 24.7 | 16.7 |
| Fara-1.5-4B | 4B | 20.7 | 7.7 |
| UI-TARS-1.5-7B | 7B | 22.3 | 20.2 |
| OpenWebRL-4B | 4B | 32.7 | 15.4 |
| **KORMo-Web-11B (ours)** | **11B** | **29.0** | **30.1** |
| GPT-6-luna (proprietary) | โ | 63.0 | 65.7 |
*Success rates are calculated over the full task set, including failed or incomplete runs. Each agent uses its own prompt and action format.*
KORMo-Web-11B achieves **30.1% Korean task success**, exceeding the strongest non-KORMo open-weight baseline listed above by **9.9 percentage points**.
While our specialized KORMo-Web-4B and 9B models achieve higher web task success, **KORMo-Web-11B prioritizes versatility across language, vision, and GUI interaction**, rather than web-only optimization.
## Why KORMo-Web-11B?
- **Korean from the Ground Up:** Built upon our independently pretrained KoreanโEnglish foundation model.
- **General-Purpose Multimodal AI:** Supports text-based language tasks, visual understanding, and browser interactions within one model.
- **Visual-First Web Interaction:** Operates primarily through screenshots without requiring full DOM or accessibility trees.
- **Think, Then Act:** Generates an inspectable reasoning step before executing browser actions.
- **Commercial Use Permitted:** Released under Apache-2.0, supporting academic research, commercial applications, fine-tuning, and redistribution.
- **Open and Ready to Run:** Includes model weights, Transformers 5 support, a vLLM inference path, browser harness, and reference inference settings.
**KORMo-Web-11B โ From a Korean Foundation Model to a General-Purpose Web Agent.**
## Quickstart
Tested with **transformers 5.12.1** and **vLLM 0.23.0**. The model runs on a single GPU.
transformers does not ship the KORMo text model, so this repo includes it (`_modeling_kormo.py`) together with a small loader, `kormo_load.py`:
- `kormo_load.register(path)` registers the KORMo text model. Call it **before** loading the processor or the model.
- `kormo_load.patch_vision(model)` feeds the SigLIP features *after* post-layernorm to the projector, as in training. Without it the model still runs but misreads screenshots.
### Inference example (one step, transformers)
```bash
pip install -U "transformers>=5.12" torch pillow huggingface_hub
huggingface-cli download KORMo-VL/KORMo-Web-11B --local-dir KORMo-Web-11B
```
```python
import json, sys, torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor
M = "KORMo-Web-11B"
sys.path.insert(0, M)
import kormo_load
kormo_load.register(M)
processor = AutoProcessor.from_pretrained(M)
model = AutoModelForImageTextToText.from_pretrained(M, dtype=torch.bfloat16).to("cuda").eval()
kormo_load.patch_vision(model)
SYSTEM = open(f"{M}/system_prompt.md").read().strip()
TOOLS = json.load(open(f"{M}/tools.json"))
obs = ("<observation>\nscreen size: 1280 x 720\nCurrent tab: 0\nAvailable tabs:\n"
"- Tab 0 (active): https://www.aladin.co.kr/\n\nurl: https://www.aladin.co.kr/\nscreenshot:\n")
messages = [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": [
{"type": "text", "text": "<user_request>\n์๋ผ๋ ์ฌ์ดํธ์์ 'ํด์ปค์ค ํ ์ต RC'๋ฅผ ๊ฒ์ํ ๋ค ๊ฐ์ฅ ๋จผ์ ํ์๋๋ ๋์์ ์ถ๊ฐ์ผ์ ์๋ ค์ค.\n</user_request>\n" + obs},
{"type": "image", "image": Image.open("screen.png").convert("RGB")},
{"type": "text", "text": "\n</observation>"},
]},
]
inputs = processor.apply_chat_template(messages, tools=TOOLS, add_generation_prompt=True,
tokenize=True, return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=16384, do_sample=True, temperature=0.6, top_p=0.95, top_k=20)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))
```
A real first step from our Online-Mind2Web run (Aladin home page):
```text
<think>I see the Aladin homepage, with the search field labeled โํตํฉ๊ฒ์โ near the top. I have not yet searched for the book, so I still need to enter โํด์ปค์ค ํ ์ต RC,โ inspect the first resultโs publication date, and report that date. My plan:
1. Click the search field
2. Enter โํด์ปค์ค ํ ์ต RCโ and submit the search
3. Read the publication date of the first listed book
This click targets the visible search field and is a safe, relevant first action.
</think>
<tool_call>
<function=click>
<parameter=x>
482
</parameter>
<parameter=y>
212
</parameter>
</function>
</tool_call>
```
Coordinates are normalized to 0โ1000. Convert them to pixels with `x / 1000 ร width` and `y / 1000 ร height`.
### Run the agent (vLLM)
vLLM does not include the KORMo architecture either. The KORMo text layers have the Llama structure, so `vllm/convert_to_vllm.py` renames two parameter names and writes a folder vLLM loads natively. `vllm/sitecustomize.py` turns SigLIP post-layernorm back on inside vLLM.
**1. Convert (once)**
```bash
pip install -U "vllm>=0.23.0" openai playwright safetensors
playwright install chromium
python KORMo-Web-11B/vllm/convert_to_vllm.py KORMo-Web-11B KORMo-Web-11B-vllm
```
**2. Start the model server**
```bash
PYTHONPATH=$PWD/KORMo-Web-11B/vllm vllm serve ./KORMo-Web-11B-vllm --served-model-name KORMo-Web-11B \
--max-model-len 65536 --limit-mm-per-prompt '{"image":1,"video":0}'
```
Check that the server log prints `[kormo-vllm] SigLIP post_layernorm enabled`.
**3. Run a browser task**
```bash
python KORMo-Web-11B/examples/agent_playwright.py --model KORMo-Web-11B \
--task "๋ค๋์์์ 50๋ง ์ ์ดํ, 10์ธ์น ์ด์ ํ๋ธ๋ฆฟ 3๊ฐ๋ฅผ ์ฐพ์์ ๊ฐ๊ฒฉ์ ์๋ ค์ค"
```
The agent opens Chromium, prints each action as it goes, and finishes with the model's answer.
## How It Works
```
user <user_request> task </user_request> + observation (tabs, url, screenshot)
assistant <think> what I see ยท what is done ยท plan </think> <tool_call> one action </tool_call>
tool action result + new observation (newest screenshot only)
โฆ repeat until finished(answer)
```
- **Action space:** 22 browser tools: click, type, hotkey, scroll, drag, goto, tab control, `finished`, and others ([`tools.json`](tools.json)).
- **Memory:** the full text history of thoughts, actions and observations is kept. Only the current screenshot is sent.
- **Recommended settings:** temperature 0.6, top_p 0.95, top_k 20, thinking on, up to 16k tokens per step and 50 steps. These are already set in `generation_config.json`.
The exact message format, parsing rules and feedback strings are in **[HARNESS.md](HARNESS.md)**. Follow it if you build your own harness.
## Evaluation Details
| | |
|---|---|
| Tasks | Online-Mind2Web: 300 English, 312 Korean |
| Judge | GPT-6 |
## Files
| File | Description |
|---|---|
| `model.safetensors` ยท `config.json` | Weights and config (LLaVA-OneVision: SigLIP + KORMo-10B text model) |
| `_configuration_kormo.py` ยท `_modeling_kormo.py` | KORMo text model code for transformers 5 |
| `kormo_load.py` | `register()` and `patch_vision()` |
| `chat_template.jinja` | Chat template (XML tool calls, thinking on) |
| `system_prompt.md` ยท `tools.json` | System prompt and 22-tool schema |
| `HARNESS.md` | Full harness specification |
| `examples/agent_playwright.py` | Reference browser agent (Playwright + OpenAI-compatible API). It does not execute `upload`. |
| `vllm/convert_to_vllm.py` ยท `vllm/sitecustomize.py` | Convert for vLLM and the post-layernorm fix |
| `generation_config.json` ยท `*processor_config.json` ยท `tokenizer*.json` ยท `special_tokens_map.json` | Sampling, processor and tokenizer configs |
## Acknowledgements
Built on **[KORMo-VL](https://huggingface.co/KORMo-VL/KORMo-VL)** and **[KORMo-10B](https://huggingface.co/KORMo-Team/KORMo-10B-sft)** by KAIST MLP Lab. Evaluated on **Online-Mind2Web** and its Korean counterpart.
## Contact
- KyungTae Lim, Professor at KAIST. `ktlim@kaist.ac.kr`
## Contributor
- Hangyeol Yoo (hg.yoo@kaist.ac.kr)
- Minkyu Kim (minkyukim@kaist.ac.kr)
- Minkyung Cho (minkyung.cho@kaist.ac.kr)
- Yerin Nam (yerinnam@kaist.ac.kr)
- Seungwoo Song (sswoo@kaist.ac.kr)
- Gyun Lee (gyunini@kaist.ac.kr)
- KyungTae Lim (ktlim@kaist.ac.kr)
</div>
|