How to use from the
Use from the
Transformers library
# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="KORMo-VL/KORMo-Web-11B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("KORMo-VL/KORMo-Web-11B")
model = AutoModelForMultimodalLM.from_pretrained("KORMo-VL/KORMo-Web-11B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))
Quick Links

KORMo-Web-11B

One Model. Three Capabilities: LLM, VLM, and Web Agent.

KORMo-Web-11B๋Š” KAIST MLP ์—ฐ๊ตฌ์‹ค์ด ์ฒ˜์Œ๋ถ€ํ„ฐ ๊ตฌ์ถ•ํ•œ KORMo ํŒŒ์šด๋ฐ์ด์…˜ ๋ชจ๋ธ์„ ๊ธฐ๋ฐ˜์œผ๋กœ, LLMยทVLMยทGUI Agent ๋Šฅ๋ ฅ์„ ๋ชจ๋‘ ๊ฐ–์ถ˜ ๋ฒ”์šฉ ๋ฉ€ํ‹ฐ๋ชจ๋‹ฌ AI ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค.

  • ํ•˜๋‚˜์˜ ๋ชจ๋ธ, ์„ธ ๊ฐ€์ง€ ๋Šฅ๋ ฅ: ์ผ๋ฐ˜์ ์ธ ์งˆ๋ฌธ์—๋Š” LLM์œผ๋กœ ๋‹ต๋ณ€ํ•˜๊ณ , ์ด๋ฏธ์ง€ ์ดํ•ด๊ฐ€ ํ•„์š”ํ•œ ๊ฒฝ์šฐ VLM์œผ๋กœ, ์›น ์กฐ์ž‘์ด ํ•„์š”ํ•œ ์š”์ฒญ์—๋Š” GUI Agent๋กœ ์ž‘๋™ํ•ฉ๋‹ˆ๋‹ค.
  • ๋…์ž ํŒŒ์šด๋ฐ์ด์…˜ ๋ชจ๋ธ ๊ธฐ๋ฐ˜: From-scratch๋กœ ํ•™์Šตํ•œ KORMo-10B๋ฅผ KORMo-VL๋กœ ํ™•์žฅํ•˜๊ณ , ๊ธฐ์กด ์–ธ์–ดยท์‹œ๊ฐ ์ดํ•ด ๋Šฅ๋ ฅ์„ ์œ ์ง€ํ•˜๋„๋ก ํ•™์Šตํ•œ ์›น ์—์ด์ „ํŠธ๋กœ ์ƒ๊ฐ<Think> ๋˜ํ•œ ์œ ์ผํ•˜๊ฒŒ ํ•œ๊ตญ์–ด๋กœ ํ•˜๋Š” ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค.
  • ๋Œ€๊ทœ๋ชจ ์›น ์‚ฌ์ „ํ•™์Šต: 1,000๋งŒ ๊ฑด ์ด์ƒ์˜ ํ•œ์˜ ์›น ํ•™์Šต ์ƒ˜ํ”Œ๋กœ UI ์š”์†Œ์˜ ์œ„์น˜ยท์˜๋ฏธยท๊ตฌ์กฐยท๊ธฐ๋Šฅ ๋“ฑ์„ ์„ธ๋ฐ€ํ•˜๊ฒŒ ํ•™์Šตํ–ˆ์Šต๋‹ˆ๋‹ค.
  • ์‹ค์ œ ์›น ํ–‰๋™ ํ•™์Šต: 10๋งŒ ๊ฑด ์ด์ƒ์˜ ํ•œ์˜ ์›น ์‚ฌ์šฉ ๊ถค์ ์„ ํ™œ์šฉํ•ด ๋ณต์žกํ•œ ๋‹ค๋‹จ๊ณ„ ์›น ์ž‘์—…์„ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.
  • ์™„์ „ ๊ณต๊ฐœ ๋ฐ ์ƒ์—…์  ํ™œ์šฉ: ๋ชจ๋ธ ๊ฐ€์ค‘์น˜์™€ ํ•™์Šตยทํ‰๊ฐ€ ๋ฆฌ์†Œ์Šค๋ฅผ ๊ณต๊ฐœํ•˜๋ฉฐ, Apache-2.0 ๋ผ์ด์„ ์Šค์— ๋”ฐ๋ผ ์ƒ์—…์  ์ด์šฉ๊ณผ ์ˆ˜์ •ยท์žฌ๋ฐฐํฌ๊ฐ€ ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค.

KORMo-Web-11B is a general-purpose bilingual AI model built on the KORMo foundation model family, developed by KAIST MLP Lab starting from a language model trained entirely from scratch. Unlike our Qwen-based KORMo-Web-4B and 9B models, which focus on maximizing web task performance, KORMo-Web-11B is designed to preserve its original LLM and VLM capabilities while learning to act as a GUI agent. It can understand Korean and English text, interpret images, and complete real browser tasks by visually navigating webpagesโ€”all within a single model.

What Makes KORMo-Web-11B Different?

  • Built on Our Own Foundation Model: Developed from the KORMo-10B language model, trained from scratch by KAIST MLP Lab, and extended through KORMo-VL into a web agent.
  • Three Capabilities in One Model: Designed to retain general LLM reasoning and VLM understanding while adding GUI interaction capabilities, rather than optimizing exclusively for web navigation.
  • 10M+ Atomic-Level Web Pretraining Samples: Large-scale Korean and English web understanding training, covering individual UI elements, their locations, layouts, functions, and states.
  • 100K+ Web Interaction Trajectories: Bilingual browser demonstrations for learning real-world, multi-step web tasks.
  • Open and Commercially Usable: Model weights and supporting resources are openly released under Apache-2.0, allowing commercial use, modification, and redistribution subject to the license terms.

From a Korean LLM to a General-Purpose Web Agent

KORMo-Web-11B extends our independently developed Koreanโ€“English foundation model through three stages:

Stage Model Capabilities
1. Language KORMo-10B Koreanโ€“English LLM trained from scratch with a Korean-optimized tokenizer
2. Vision KORMo-VL Multimodal understanding through a SigLIP vision encoder and KORMo language backbone
3. Web Agent KORMo-Web-11B Large-scale web pretraining and action learning for visual browser interaction

The goal is not just a better web agent, but a versatile model that can understand language, interpret visual information, and interact with the web.

Performance

We evaluated KORMo-Web-11B on 612 live-website tasks (300 English, 312 Korean) from Online-Mind2Web, using the same LLM judge across all evaluated models.

Model Size English SR (%) Korean SR (%)
Qwen3.5-9B (base) 9B 24.7 16.7
Fara-1.5-4B 4B 20.7 7.7
UI-TARS-1.5-7B 7B 22.3 20.2
OpenWebRL-4B 4B 32.7 15.4
KORMo-Web-11B (ours) 11B 29.0 30.1
GPT-6-luna (proprietary) โ€“ 63.0 65.7

Success rates are calculated over the full task set, including failed or incomplete runs. Each agent uses its own prompt and action format.

KORMo-Web-11B achieves 30.1% Korean task success, exceeding the strongest non-KORMo open-weight baseline listed above by 9.9 percentage points.

While our specialized KORMo-Web-4B and 9B models achieve higher web task success, KORMo-Web-11B prioritizes versatility across language, vision, and GUI interaction, rather than web-only optimization.

Why KORMo-Web-11B?

  • Korean from the Ground Up: Built upon our independently pretrained Koreanโ€“English foundation model.
  • General-Purpose Multimodal AI: Supports text-based language tasks, visual understanding, and browser interactions within one model.
  • Visual-First Web Interaction: Operates primarily through screenshots without requiring full DOM or accessibility trees.
  • Think, Then Act: Generates an inspectable reasoning step before executing browser actions.
  • Commercial Use Permitted: Released under Apache-2.0, supporting academic research, commercial applications, fine-tuning, and redistribution.
  • Open and Ready to Run: Includes model weights, Transformers 5 support, a vLLM inference path, browser harness, and reference inference settings.

KORMo-Web-11B โ€” From a Korean Foundation Model to a General-Purpose Web Agent.

Quickstart

Tested with transformers 5.12.1 and vLLM 0.23.0. The model runs on a single GPU.

transformers does not ship the KORMo text model, so this repo includes it (_modeling_kormo.py) together with a small loader, kormo_load.py:

  • kormo_load.register(path) registers the KORMo text model. Call it before loading the processor or the model.
  • kormo_load.patch_vision(model) feeds the SigLIP features after post-layernorm to the projector, as in training. Without it the model still runs but misreads screenshots.

Inference example (one step, transformers)

pip install -U "transformers>=5.12" torch pillow huggingface_hub
huggingface-cli download KORMo-VL/KORMo-Web-11B --local-dir KORMo-Web-11B
import json, sys, torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

M = "KORMo-Web-11B"
sys.path.insert(0, M)
import kormo_load
kormo_load.register(M)
processor = AutoProcessor.from_pretrained(M)
model = AutoModelForImageTextToText.from_pretrained(M, dtype=torch.bfloat16).to("cuda").eval()
kormo_load.patch_vision(model)

SYSTEM = open(f"{M}/system_prompt.md").read().strip()
TOOLS = json.load(open(f"{M}/tools.json"))
obs = ("<observation>\nscreen size: 1280 x 720\nCurrent tab: 0\nAvailable tabs:\n"
       "- Tab 0 (active): https://www.aladin.co.kr/\n\nurl: https://www.aladin.co.kr/\nscreenshot:\n")
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": [
        {"type": "text", "text": "<user_request>\n์•Œ๋ผ๋”˜ ์‚ฌ์ดํŠธ์—์„œ 'ํ•ด์ปค์Šค ํ† ์ต RC'๋ฅผ ๊ฒ€์ƒ‰ํ•œ ๋’ค ๊ฐ€์žฅ ๋จผ์ € ํ‘œ์‹œ๋˜๋Š” ๋„์„œ์˜ ์ถœ๊ฐ„์ผ์„ ์•Œ๋ ค์ค˜.\n</user_request>\n" + obs},
        {"type": "image", "image": Image.open("screen.png").convert("RGB")},
        {"type": "text", "text": "\n</observation>"},
    ]},
]
inputs = processor.apply_chat_template(messages, tools=TOOLS, add_generation_prompt=True,
                                       tokenize=True, return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=16384, do_sample=True, temperature=0.6, top_p=0.95, top_k=20)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))

A real first step from our Online-Mind2Web run (Aladin home page):

<think>I see the Aladin homepage, with the search field labeled โ€œํ†ตํ•ฉ๊ฒ€์ƒ‰โ€ near the top. I have not yet searched for the book, so I still need to enter โ€œํ•ด์ปค์Šค ํ† ์ต RC,โ€ inspect the first resultโ€™s publication date, and report that date. My plan:
1. Click the search field
2. Enter โ€œํ•ด์ปค์Šค ํ† ์ต RCโ€ and submit the search
3. Read the publication date of the first listed book
This click targets the visible search field and is a safe, relevant first action.
</think>

<tool_call>
<function=click>
<parameter=x>
482
</parameter>
<parameter=y>
212
</parameter>
</function>
</tool_call>

Coordinates are normalized to 0โ€“1000. Convert them to pixels with x / 1000 ร— width and y / 1000 ร— height.

Run the agent (vLLM)

vLLM does not include the KORMo architecture either. The KORMo text layers have the Llama structure, so vllm/convert_to_vllm.py renames two parameter names and writes a folder vLLM loads natively. vllm/sitecustomize.py turns SigLIP post-layernorm back on inside vLLM.

1. Convert (once)

pip install -U "vllm>=0.23.0" openai playwright safetensors
playwright install chromium
python KORMo-Web-11B/vllm/convert_to_vllm.py KORMo-Web-11B KORMo-Web-11B-vllm

2. Start the model server

PYTHONPATH=$PWD/KORMo-Web-11B/vllm vllm serve ./KORMo-Web-11B-vllm --served-model-name KORMo-Web-11B \
  --max-model-len 65536 --limit-mm-per-prompt '{"image":1,"video":0}'

Check that the server log prints [kormo-vllm] SigLIP post_layernorm enabled.

3. Run a browser task

python KORMo-Web-11B/examples/agent_playwright.py --model KORMo-Web-11B \
  --task "๋‹ค๋‚˜์™€์—์„œ 50๋งŒ ์› ์ดํ•˜, 10์ธ์น˜ ์ด์ƒ ํƒœ๋ธ”๋ฆฟ 3๊ฐœ๋ฅผ ์ฐพ์•„์„œ ๊ฐ€๊ฒฉ์„ ์•Œ๋ ค์ค˜"

The agent opens Chromium, prints each action as it goes, and finishes with the model's answer.

How It Works

user       <user_request> task </user_request>  + observation (tabs, url, screenshot)
assistant  <think> what I see ยท what is done ยท plan </think>  <tool_call> one action </tool_call>
tool       action result  + new observation (newest screenshot only)
โ€ฆ          repeat until finished(answer)
  • Action space: 22 browser tools: click, type, hotkey, scroll, drag, goto, tab control, finished, and others (tools.json).
  • Memory: the full text history of thoughts, actions and observations is kept. Only the current screenshot is sent.
  • Recommended settings: temperature 0.6, top_p 0.95, top_k 20, thinking on, up to 16k tokens per step and 50 steps. These are already set in generation_config.json.

The exact message format, parsing rules and feedback strings are in HARNESS.md. Follow it if you build your own harness.

Evaluation Details

Tasks Online-Mind2Web: 300 English, 312 Korean
Judge GPT-6

Files

File Description
model.safetensors ยท config.json Weights and config (LLaVA-OneVision: SigLIP + KORMo-10B text model)
_configuration_kormo.py ยท _modeling_kormo.py KORMo text model code for transformers 5
kormo_load.py register() and patch_vision()
chat_template.jinja Chat template (XML tool calls, thinking on)
system_prompt.md ยท tools.json System prompt and 22-tool schema
HARNESS.md Full harness specification
examples/agent_playwright.py Reference browser agent (Playwright + OpenAI-compatible API). It does not execute upload.
vllm/convert_to_vllm.py ยท vllm/sitecustomize.py Convert for vLLM and the post-layernorm fix
generation_config.json ยท *processor_config.json ยท tokenizer*.json ยท special_tokens_map.json Sampling, processor and tokenizer configs

Acknowledgements

Built on KORMo-VL and KORMo-10B by KAIST MLP Lab. Evaluated on Online-Mind2Web and its Korean counterpart.

Downloads last month
2
Safetensors
Model size
11B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for KORMo-VL/KORMo-Web-11B

Finetuned
(1)
this model

Collection including KORMo-VL/KORMo-Web-11B