Instructions to use KORMo-VL/KORMo-Web-11B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KORMo-VL/KORMo-Web-11B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="KORMo-VL/KORMo-Web-11B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("KORMo-VL/KORMo-Web-11B") model = AutoModelForMultimodalLM.from_pretrained("KORMo-VL/KORMo-Web-11B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use KORMo-VL/KORMo-Web-11B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "KORMo-VL/KORMo-Web-11B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-11B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/KORMo-VL/KORMo-Web-11B
- SGLang
How to use KORMo-VL/KORMo-Web-11B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-11B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-11B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-11B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-11B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use KORMo-VL/KORMo-Web-11B with Docker Model Runner:
docker model run hf.co/KORMo-VL/KORMo-Web-11B
Download README.md from KORMo-VL/KORMo-Web-11B: direct link, hf CLI and curl.
- Browser
- Download file 13.8 kB
-
https://huggingface.co/KORMo-VL/KORMo-Web-11B/resolve/main/README.md
- Command line
-
hf download hf://KORMo-VL/KORMo-Web-11B/README.md
-
curl -L -o README.md https://huggingface.co/KORMo-VL/KORMo-Web-11B/resolve/main/README.md
license: apache-2.0
base_model:
- KORMo-VL/KORMo-VL
language:
- ko
- en
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- gui-agent
- web-agent
- computer-use
- browser
- korean
KORMo-Web-11B
One Model. Three Capabilities: LLM, VLM, and Web Agent.
KORMo-Web-11B๋ KAIST MLP ์ฐ๊ตฌ์ค์ด ์ฒ์๋ถํฐ ๊ตฌ์ถํ KORMo ํ์ด๋ฐ์ด์ ๋ชจ๋ธ์ ๊ธฐ๋ฐ์ผ๋ก, LLMยทVLMยทGUI Agent ๋ฅ๋ ฅ์ ๋ชจ๋ ๊ฐ์ถ ๋ฒ์ฉ ๋ฉํฐ๋ชจ๋ฌ AI ๋ชจ๋ธ์ ๋๋ค.
- ํ๋์ ๋ชจ๋ธ, ์ธ ๊ฐ์ง ๋ฅ๋ ฅ: ์ผ๋ฐ์ ์ธ ์ง๋ฌธ์๋ LLM์ผ๋ก ๋ต๋ณํ๊ณ , ์ด๋ฏธ์ง ์ดํด๊ฐ ํ์ํ ๊ฒฝ์ฐ VLM์ผ๋ก, ์น ์กฐ์์ด ํ์ํ ์์ฒญ์๋ GUI Agent๋ก ์๋ํฉ๋๋ค.
- ๋ ์ ํ์ด๋ฐ์ด์ ๋ชจ๋ธ ๊ธฐ๋ฐ: From-scratch๋ก ํ์ตํ KORMo-10B๋ฅผ KORMo-VL๋ก ํ์ฅํ๊ณ , ๊ธฐ์กด ์ธ์ดยท์๊ฐ ์ดํด ๋ฅ๋ ฅ์ ์ ์งํ๋๋ก ํ์ตํ ์น ์์ด์ ํธ๋ก ์๊ฐ<Think> ๋ํ ์ ์ผํ๊ฒ ํ๊ตญ์ด๋ก ํ๋ ๋ชจ๋ธ์ ๋๋ค.
- ๋๊ท๋ชจ ์น ์ฌ์ ํ์ต: 1,000๋ง ๊ฑด ์ด์์ ํ์ ์น ํ์ต ์ํ๋ก UI ์์์ ์์นยท์๋ฏธยท๊ตฌ์กฐยท๊ธฐ๋ฅ ๋ฑ์ ์ธ๋ฐํ๊ฒ ํ์ตํ์ต๋๋ค.
- ์ค์ ์น ํ๋ ํ์ต: 10๋ง ๊ฑด ์ด์์ ํ์ ์น ์ฌ์ฉ ๊ถค์ ์ ํ์ฉํด ๋ณต์กํ ๋ค๋จ๊ณ ์น ์์ ์ ์ํํฉ๋๋ค.
- ์์ ๊ณต๊ฐ ๋ฐ ์์ ์ ํ์ฉ: ๋ชจ๋ธ ๊ฐ์ค์น์ ํ์ตยทํ๊ฐ ๋ฆฌ์์ค๋ฅผ ๊ณต๊ฐํ๋ฉฐ, Apache-2.0 ๋ผ์ด์ ์ค์ ๋ฐ๋ผ ์์ ์ ์ด์ฉ๊ณผ ์์ ยท์ฌ๋ฐฐํฌ๊ฐ ๊ฐ๋ฅํฉ๋๋ค.
KORMo-Web-11B is a general-purpose bilingual AI model built on the KORMo foundation model family, developed by KAIST MLP Lab starting from a language model trained entirely from scratch. Unlike our Qwen-based KORMo-Web-4B and 9B models, which focus on maximizing web task performance, KORMo-Web-11B is designed to preserve its original LLM and VLM capabilities while learning to act as a GUI agent. It can understand Korean and English text, interpret images, and complete real browser tasks by visually navigating webpagesโall within a single model.
What Makes KORMo-Web-11B Different?
- Built on Our Own Foundation Model: Developed from the KORMo-10B language model, trained from scratch by KAIST MLP Lab, and extended through KORMo-VL into a web agent.
- Three Capabilities in One Model: Designed to retain general LLM reasoning and VLM understanding while adding GUI interaction capabilities, rather than optimizing exclusively for web navigation.
- 10M+ Atomic-Level Web Pretraining Samples: Large-scale Korean and English web understanding training, covering individual UI elements, their locations, layouts, functions, and states.
- 100K+ Web Interaction Trajectories: Bilingual browser demonstrations for learning real-world, multi-step web tasks.
- Open and Commercially Usable: Model weights and supporting resources are openly released under Apache-2.0, allowing commercial use, modification, and redistribution subject to the license terms.
From a Korean LLM to a General-Purpose Web Agent
KORMo-Web-11B extends our independently developed KoreanโEnglish foundation model through three stages:
| Stage | Model | Capabilities |
|---|---|---|
| 1. Language | KORMo-10B | KoreanโEnglish LLM trained from scratch with a Korean-optimized tokenizer |
| 2. Vision | KORMo-VL | Multimodal understanding through a SigLIP vision encoder and KORMo language backbone |
| 3. Web Agent | KORMo-Web-11B | Large-scale web pretraining and action learning for visual browser interaction |
The goal is not just a better web agent, but a versatile model that can understand language, interpret visual information, and interact with the web.
Performance
We evaluated KORMo-Web-11B on 612 live-website tasks (300 English, 312 Korean) from Online-Mind2Web, using the same LLM judge across all evaluated models.
| Model | Size | English SR (%) | Korean SR (%) |
|---|---|---|---|
| Qwen3.5-9B (base) | 9B | 24.7 | 16.7 |
| Fara-1.5-4B | 4B | 20.7 | 7.7 |
| UI-TARS-1.5-7B | 7B | 22.3 | 20.2 |
| OpenWebRL-4B | 4B | 32.7 | 15.4 |
| KORMo-Web-11B (ours) | 11B | 29.0 | 30.1 |
| GPT-6-luna (proprietary) | โ | 63.0 | 65.7 |
Success rates are calculated over the full task set, including failed or incomplete runs. Each agent uses its own prompt and action format.
KORMo-Web-11B achieves 30.1% Korean task success, exceeding the strongest non-KORMo open-weight baseline listed above by 9.9 percentage points.
While our specialized KORMo-Web-4B and 9B models achieve higher web task success, KORMo-Web-11B prioritizes versatility across language, vision, and GUI interaction, rather than web-only optimization.
Why KORMo-Web-11B?
- Korean from the Ground Up: Built upon our independently pretrained KoreanโEnglish foundation model.
- General-Purpose Multimodal AI: Supports text-based language tasks, visual understanding, and browser interactions within one model.
- Visual-First Web Interaction: Operates primarily through screenshots without requiring full DOM or accessibility trees.
- Think, Then Act: Generates an inspectable reasoning step before executing browser actions.
- Commercial Use Permitted: Released under Apache-2.0, supporting academic research, commercial applications, fine-tuning, and redistribution.
- Open and Ready to Run: Includes model weights, Transformers 5 support, a vLLM inference path, browser harness, and reference inference settings.
KORMo-Web-11B โ From a Korean Foundation Model to a General-Purpose Web Agent.
Quickstart
Tested with transformers 5.12.1 and vLLM 0.23.0. The model runs on a single GPU.
transformers does not ship the KORMo text model, so this repo includes it (_modeling_kormo.py) together with a small loader, kormo_load.py:
kormo_load.register(path)registers the KORMo text model. Call it before loading the processor or the model.kormo_load.patch_vision(model)feeds the SigLIP features after post-layernorm to the projector, as in training. Without it the model still runs but misreads screenshots.
Inference example (one step, transformers)
pip install -U "transformers>=5.12" torch pillow huggingface_hub
huggingface-cli download KORMo-VL/KORMo-Web-11B --local-dir KORMo-Web-11B
import json, sys, torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor
M = "KORMo-Web-11B"
sys.path.insert(0, M)
import kormo_load
kormo_load.register(M)
processor = AutoProcessor.from_pretrained(M)
model = AutoModelForImageTextToText.from_pretrained(M, dtype=torch.bfloat16).to("cuda").eval()
kormo_load.patch_vision(model)
SYSTEM = open(f"{M}/system_prompt.md").read().strip()
TOOLS = json.load(open(f"{M}/tools.json"))
obs = ("<observation>\nscreen size: 1280 x 720\nCurrent tab: 0\nAvailable tabs:\n"
"- Tab 0 (active): https://www.aladin.co.kr/\n\nurl: https://www.aladin.co.kr/\nscreenshot:\n")
messages = [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": [
{"type": "text", "text": "<user_request>\n์๋ผ๋ ์ฌ์ดํธ์์ 'ํด์ปค์ค ํ ์ต RC'๋ฅผ ๊ฒ์ํ ๋ค ๊ฐ์ฅ ๋จผ์ ํ์๋๋ ๋์์ ์ถ๊ฐ์ผ์ ์๋ ค์ค.\n</user_request>\n" + obs},
{"type": "image", "image": Image.open("screen.png").convert("RGB")},
{"type": "text", "text": "\n</observation>"},
]},
]
inputs = processor.apply_chat_template(messages, tools=TOOLS, add_generation_prompt=True,
tokenize=True, return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=16384, do_sample=True, temperature=0.6, top_p=0.95, top_k=20)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))
A real first step from our Online-Mind2Web run (Aladin home page):
<think>I see the Aladin homepage, with the search field labeled โํตํฉ๊ฒ์โ near the top. I have not yet searched for the book, so I still need to enter โํด์ปค์ค ํ ์ต RC,โ inspect the first resultโs publication date, and report that date. My plan:
1. Click the search field
2. Enter โํด์ปค์ค ํ ์ต RCโ and submit the search
3. Read the publication date of the first listed book
This click targets the visible search field and is a safe, relevant first action.
</think>
<tool_call>
<function=click>
<parameter=x>
482
</parameter>
<parameter=y>
212
</parameter>
</function>
</tool_call>
Coordinates are normalized to 0โ1000. Convert them to pixels with x / 1000 ร width and y / 1000 ร height.
Run the agent (vLLM)
vLLM does not include the KORMo architecture either. The KORMo text layers have the Llama structure, so vllm/convert_to_vllm.py renames two parameter names and writes a folder vLLM loads natively. vllm/sitecustomize.py turns SigLIP post-layernorm back on inside vLLM.
1. Convert (once)
pip install -U "vllm>=0.23.0" openai playwright safetensors
playwright install chromium
python KORMo-Web-11B/vllm/convert_to_vllm.py KORMo-Web-11B KORMo-Web-11B-vllm
2. Start the model server
PYTHONPATH=$PWD/KORMo-Web-11B/vllm vllm serve ./KORMo-Web-11B-vllm --served-model-name KORMo-Web-11B \
--max-model-len 65536 --limit-mm-per-prompt '{"image":1,"video":0}'
Check that the server log prints [kormo-vllm] SigLIP post_layernorm enabled.
3. Run a browser task
python KORMo-Web-11B/examples/agent_playwright.py --model KORMo-Web-11B \
--task "๋ค๋์์์ 50๋ง ์ ์ดํ, 10์ธ์น ์ด์ ํ๋ธ๋ฆฟ 3๊ฐ๋ฅผ ์ฐพ์์ ๊ฐ๊ฒฉ์ ์๋ ค์ค"
The agent opens Chromium, prints each action as it goes, and finishes with the model's answer.
How It Works
user <user_request> task </user_request> + observation (tabs, url, screenshot)
assistant <think> what I see ยท what is done ยท plan </think> <tool_call> one action </tool_call>
tool action result + new observation (newest screenshot only)
โฆ repeat until finished(answer)
- Action space: 22 browser tools: click, type, hotkey, scroll, drag, goto, tab control,
finished, and others (tools.json). - Memory: the full text history of thoughts, actions and observations is kept. Only the current screenshot is sent.
- Recommended settings: temperature 0.6, top_p 0.95, top_k 20, thinking on, up to 16k tokens per step and 50 steps. These are already set in
generation_config.json.
The exact message format, parsing rules and feedback strings are in HARNESS.md. Follow it if you build your own harness.
Evaluation Details
| Tasks | Online-Mind2Web: 300 English, 312 Korean |
| Judge | GPT-6 |
Files
| File | Description |
|---|---|
model.safetensors ยท config.json |
Weights and config (LLaVA-OneVision: SigLIP + KORMo-10B text model) |
_configuration_kormo.py ยท _modeling_kormo.py |
KORMo text model code for transformers 5 |
kormo_load.py |
register() and patch_vision() |
chat_template.jinja |
Chat template (XML tool calls, thinking on) |
system_prompt.md ยท tools.json |
System prompt and 22-tool schema |
HARNESS.md |
Full harness specification |
examples/agent_playwright.py |
Reference browser agent (Playwright + OpenAI-compatible API). It does not execute upload. |
vllm/convert_to_vllm.py ยท vllm/sitecustomize.py |
Convert for vLLM and the post-layernorm fix |
generation_config.json ยท *processor_config.json ยท tokenizer*.json ยท special_tokens_map.json |
Sampling, processor and tokenizer configs |
Acknowledgements
Built on KORMo-VL and KORMo-10B by KAIST MLP Lab. Evaluated on Online-Mind2Web and its Korean counterpart.