Image-Text-to-Text
Transformers
Safetensors
Korean
English
llava_onevision
gui-agent
web-agent
computer-use
browser
korean
conversational
Instructions to use KORMo-VL/KORMo-Web-11B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KORMo-VL/KORMo-Web-11B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="KORMo-VL/KORMo-Web-11B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("KORMo-VL/KORMo-Web-11B") model = AutoModelForMultimodalLM.from_pretrained("KORMo-VL/KORMo-Web-11B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use KORMo-VL/KORMo-Web-11B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "KORMo-VL/KORMo-Web-11B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-11B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/KORMo-VL/KORMo-Web-11B
- SGLang
How to use KORMo-VL/KORMo-Web-11B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-11B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-11B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-11B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-11B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use KORMo-VL/KORMo-Web-11B with Docker Model Runner:
docker model run hf.co/KORMo-VL/KORMo-Web-11B
|
Download README.md from KORMo-VL/KORMo-Web-11B: direct link, hf CLI and curl.
- Browser
- Download file 13.8 kB
-
https://huggingface.co/KORMo-VL/KORMo-Web-11B/resolve/main/README.md
- Command line
-
hf download hf://KORMo-VL/KORMo-Web-11B/README.md
-
curl -L -o README.md https://huggingface.co/KORMo-VL/KORMo-Web-11B/resolve/main/README.md
13.8 kB
| license: apache-2.0 | |
| base_model: | |
| - KORMo-VL/KORMo-VL | |
| language: | |
| - ko | |
| - en | |
| pipeline_tag: image-text-to-text | |
| library_name: transformers | |
| tags: | |
| - gui-agent | |
| - web-agent | |
| - computer-use | |
| - browser | |
| - korean | |
| <style> | |
| #kormo-web-card { font-size: 16px; line-height: 1.6; overflow-wrap: break-word; } | |
| #kormo-web-card h1 { margin: 0 0 12px; font-size: 30px; line-height: 1.25; letter-spacing: -0.018em; } | |
| #kormo-web-card h2 { margin: 30px 0 10px; font-size: 22px; line-height: 1.3; letter-spacing: -0.012em; } | |
| #kormo-web-card h3 { margin: 22px 0 8px; font-size: 17px; font-weight: 700; line-height: 1.4; } | |
| #kormo-web-card h2 + h3 { margin-top: 14px; } | |
| #kormo-web-card p { margin: 0 0 10px; } | |
| #kormo-web-card ul, #kormo-web-card ol { margin: 6px 0 12px; padding-left: 22px; } | |
| #kormo-web-card li { margin: 4px 0; line-height: 1.55; } | |
| #kormo-web-card table { width: 100%; margin: 6px 0 14px; border-collapse: collapse; font-size: 14px; line-height: 1.45; font-variant-numeric: tabular-nums; } | |
| #kormo-web-card th, #kormo-web-card td { padding: 7px 10px; vertical-align: middle; } | |
| #kormo-web-card blockquote { margin: 12px 0 16px; padding: 0 0 0 14px; font-size: 15px; } | |
| #kormo-web-card blockquote p { margin: 0; } | |
| #kormo-web-card pre { margin: 10px 0 14px; padding: 14px 16px; overflow-x: auto; line-height: 1.5; } | |
| #kormo-web-card pre code { font-size: 13px; } | |
| #kormo-web-card > p:has(> em:only-child) { margin: 4px 0 14px; font-size: 13px; } | |
| #kormo-web-card .pills { margin: 4px 0 18px; } | |
| #kormo-web-card .pill { display: inline-block; margin: 0 6px 6px 0; padding: 3px 10px; border-radius: 999px; border: 1px solid rgba(127,127,127,.35); font-size: 13px; } | |
| @media (max-width: 640px) { | |
| #kormo-web-card h1 { font-size: 26px; } | |
| #kormo-web-card table { font-size: 13px; } | |
| #kormo-web-card th, #kormo-web-card td { padding: 7px 6px; } | |
| } | |
| </style> | |
| <div id="kormo-web-card"> | |
| <video src="https://huggingface.co/KORMo-VL/KORMo-Web-11B/resolve/main/assets/gui_final_demo_maksed.mp4" controls autoplay muted loop playsinline width="100%"></video> | |
| # KORMo-Web-11B | |
| ### One Model. Three Capabilities: LLM, VLM, and Web Agent. | |
| **KORMo-Web-11B**๋ KAIST MLP ์ฐ๊ตฌ์ค์ด ์ฒ์๋ถํฐ ๊ตฌ์ถํ KORMo ํ์ด๋ฐ์ด์ ๋ชจ๋ธ์ ๊ธฐ๋ฐ์ผ๋ก, **LLMยทVLMยทGUI Agent ๋ฅ๋ ฅ์ ๋ชจ๋ ๊ฐ์ถ ๋ฒ์ฉ ๋ฉํฐ๋ชจ๋ฌ AI ๋ชจ๋ธ**์ ๋๋ค. | |
| - **ํ๋์ ๋ชจ๋ธ, ์ธ ๊ฐ์ง ๋ฅ๋ ฅ:** ์ผ๋ฐ์ ์ธ ์ง๋ฌธ์๋ LLM์ผ๋ก ๋ต๋ณํ๊ณ , ์ด๋ฏธ์ง ์ดํด๊ฐ ํ์ํ ๊ฒฝ์ฐ VLM์ผ๋ก, **์น ์กฐ์์ด ํ์ํ ์์ฒญ์๋ GUI Agent๋ก ์๋**ํฉ๋๋ค. | |
| - **๋ ์ ํ์ด๋ฐ์ด์ ๋ชจ๋ธ ๊ธฐ๋ฐ:** From-scratch๋ก ํ์ตํ KORMo-10B๋ฅผ KORMo-VL๋ก ํ์ฅํ๊ณ , ๊ธฐ์กด ์ธ์ดยท์๊ฐ ์ดํด ๋ฅ๋ ฅ์ ์ ์งํ๋๋ก ํ์ตํ ์น ์์ด์ ํธ๋ก **์๊ฐ\<Think\> ๋ํ ์ ์ผํ๊ฒ ํ๊ตญ์ด**๋ก ํ๋ ๋ชจ๋ธ์ ๋๋ค. | |
| - **๋๊ท๋ชจ ์น ์ฌ์ ํ์ต:** 1,000๋ง ๊ฑด ์ด์์ ํ์ ์น ํ์ต ์ํ๋ก UI ์์์ ์์นยท์๋ฏธยท๊ตฌ์กฐยท๊ธฐ๋ฅ ๋ฑ์ ์ธ๋ฐํ๊ฒ ํ์ตํ์ต๋๋ค. | |
| - **์ค์ ์น ํ๋ ํ์ต:** 10๋ง ๊ฑด ์ด์์ ํ์ ์น ์ฌ์ฉ ๊ถค์ ์ ํ์ฉํด ๋ณต์กํ ๋ค๋จ๊ณ ์น ์์ ์ ์ํํฉ๋๋ค. | |
| - **์์ ๊ณต๊ฐ ๋ฐ ์์ ์ ํ์ฉ:** ๋ชจ๋ธ ๊ฐ์ค์น์ ํ์ตยทํ๊ฐ ๋ฆฌ์์ค๋ฅผ ๊ณต๊ฐํ๋ฉฐ, **Apache-2.0 ๋ผ์ด์ ์ค์ ๋ฐ๋ผ ์์ ์ ์ด์ฉ๊ณผ ์์ ยท์ฌ๋ฐฐํฌ๊ฐ ๊ฐ๋ฅ**ํฉ๋๋ค. | |
| **KORMo-Web-11B** is a general-purpose bilingual AI model built on the **KORMo foundation model family**, developed by KAIST MLP Lab starting from a language model trained entirely from scratch. Unlike our Qwen-based KORMo-Web-4B and 9B models, which focus on maximizing web task performance, **KORMo-Web-11B is designed to preserve its original LLM and VLM capabilities while learning to act as a GUI agent.** It can understand Korean and English text, interpret images, and complete real browser tasks by visually navigating webpagesโ**all within a single model.** | |
| ## What Makes KORMo-Web-11B Different? | |
| - **Built on Our Own Foundation Model:** Developed from the KORMo-10B language model, trained from scratch by KAIST MLP Lab, and extended through KORMo-VL into a web agent. | |
| - **Three Capabilities in One Model:** Designed to retain general **LLM reasoning and VLM understanding** while adding GUI interaction capabilities, rather than optimizing exclusively for web navigation. | |
| - **10M+ Atomic-Level Web Pretraining Samples:** Large-scale Korean and English web understanding training, covering individual UI elements, their locations, layouts, functions, and states. | |
| - **100K+ Web Interaction Trajectories:** Bilingual browser demonstrations for learning real-world, multi-step web tasks. | |
| - **Open and Commercially Usable:** Model weights and supporting resources are openly released under **Apache-2.0**, allowing commercial use, modification, and redistribution subject to the license terms. | |
| ## From a Korean LLM to a General-Purpose Web Agent | |
| KORMo-Web-11B extends our independently developed KoreanโEnglish foundation model through three stages: | |
| | Stage | Model | Capabilities | | |
| |---|---|---| | |
| | 1. Language | [KORMo-10B](https://huggingface.co/KORMo-Team/KORMo-10B-sft) | KoreanโEnglish LLM trained from scratch with a Korean-optimized tokenizer | | |
| | 2. Vision | [KORMo-VL](https://huggingface.co/KORMo-VL/KORMo-VL) | Multimodal understanding through a SigLIP vision encoder and KORMo language backbone | | |
| | 3. Web Agent | **KORMo-Web-11B** | Large-scale web pretraining and action learning for visual browser interaction | | |
| **The goal is not just a better web agent, but a versatile model that can understand language, interpret visual information, and interact with the web.** | |
| ## Performance | |
| We evaluated KORMo-Web-11B on **612 live-website tasks** (300 English, 312 Korean) from Online-Mind2Web, using the same LLM judge across all evaluated models. | |
| | Model | Size | English SR (%) | Korean SR (%) | | |
| |---|---:|---:|---:| | |
| | Qwen3.5-9B (base) | 9B | 24.7 | 16.7 | | |
| | Fara-1.5-4B | 4B | 20.7 | 7.7 | | |
| | UI-TARS-1.5-7B | 7B | 22.3 | 20.2 | | |
| | OpenWebRL-4B | 4B | 32.7 | 15.4 | | |
| | **KORMo-Web-11B (ours)** | **11B** | **29.0** | **30.1** | | |
| | GPT-6-luna (proprietary) | โ | 63.0 | 65.7 | | |
| *Success rates are calculated over the full task set, including failed or incomplete runs. Each agent uses its own prompt and action format.* | |
| KORMo-Web-11B achieves **30.1% Korean task success**, exceeding the strongest non-KORMo open-weight baseline listed above by **9.9 percentage points**. | |
| While our specialized KORMo-Web-4B and 9B models achieve higher web task success, **KORMo-Web-11B prioritizes versatility across language, vision, and GUI interaction**, rather than web-only optimization. | |
| ## Why KORMo-Web-11B? | |
| - **Korean from the Ground Up:** Built upon our independently pretrained KoreanโEnglish foundation model. | |
| - **General-Purpose Multimodal AI:** Supports text-based language tasks, visual understanding, and browser interactions within one model. | |
| - **Visual-First Web Interaction:** Operates primarily through screenshots without requiring full DOM or accessibility trees. | |
| - **Think, Then Act:** Generates an inspectable reasoning step before executing browser actions. | |
| - **Commercial Use Permitted:** Released under Apache-2.0, supporting academic research, commercial applications, fine-tuning, and redistribution. | |
| - **Open and Ready to Run:** Includes model weights, Transformers 5 support, a vLLM inference path, browser harness, and reference inference settings. | |
| **KORMo-Web-11B โ From a Korean Foundation Model to a General-Purpose Web Agent.** | |
| ## Quickstart | |
| Tested with **transformers 5.12.1** and **vLLM 0.23.0**. The model runs on a single GPU. | |
| transformers does not ship the KORMo text model, so this repo includes it (`_modeling_kormo.py`) together with a small loader, `kormo_load.py`: | |
| - `kormo_load.register(path)` registers the KORMo text model. Call it **before** loading the processor or the model. | |
| - `kormo_load.patch_vision(model)` feeds the SigLIP features *after* post-layernorm to the projector, as in training. Without it the model still runs but misreads screenshots. | |
| ### Inference example (one step, transformers) | |
| ```bash | |
| pip install -U "transformers>=5.12" torch pillow huggingface_hub | |
| huggingface-cli download KORMo-VL/KORMo-Web-11B --local-dir KORMo-Web-11B | |
| ``` | |
| ```python | |
| import json, sys, torch | |
| from PIL import Image | |
| from transformers import AutoModelForImageTextToText, AutoProcessor | |
| M = "KORMo-Web-11B" | |
| sys.path.insert(0, M) | |
| import kormo_load | |
| kormo_load.register(M) | |
| processor = AutoProcessor.from_pretrained(M) | |
| model = AutoModelForImageTextToText.from_pretrained(M, dtype=torch.bfloat16).to("cuda").eval() | |
| kormo_load.patch_vision(model) | |
| SYSTEM = open(f"{M}/system_prompt.md").read().strip() | |
| TOOLS = json.load(open(f"{M}/tools.json")) | |
| obs = ("<observation>\nscreen size: 1280 x 720\nCurrent tab: 0\nAvailable tabs:\n" | |
| "- Tab 0 (active): https://www.aladin.co.kr/\n\nurl: https://www.aladin.co.kr/\nscreenshot:\n") | |
| messages = [ | |
| {"role": "system", "content": SYSTEM}, | |
| {"role": "user", "content": [ | |
| {"type": "text", "text": "<user_request>\n์๋ผ๋ ์ฌ์ดํธ์์ 'ํด์ปค์ค ํ ์ต RC'๋ฅผ ๊ฒ์ํ ๋ค ๊ฐ์ฅ ๋จผ์ ํ์๋๋ ๋์์ ์ถ๊ฐ์ผ์ ์๋ ค์ค.\n</user_request>\n" + obs}, | |
| {"type": "image", "image": Image.open("screen.png").convert("RGB")}, | |
| {"type": "text", "text": "\n</observation>"}, | |
| ]}, | |
| ] | |
| inputs = processor.apply_chat_template(messages, tools=TOOLS, add_generation_prompt=True, | |
| tokenize=True, return_dict=True, return_tensors="pt").to(model.device) | |
| out = model.generate(**inputs, max_new_tokens=16384, do_sample=True, temperature=0.6, top_p=0.95, top_k=20) | |
| print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False)) | |
| ``` | |
| A real first step from our Online-Mind2Web run (Aladin home page): | |
| ```text | |
| <think>I see the Aladin homepage, with the search field labeled โํตํฉ๊ฒ์โ near the top. I have not yet searched for the book, so I still need to enter โํด์ปค์ค ํ ์ต RC,โ inspect the first resultโs publication date, and report that date. My plan: | |
| 1. Click the search field | |
| 2. Enter โํด์ปค์ค ํ ์ต RCโ and submit the search | |
| 3. Read the publication date of the first listed book | |
| This click targets the visible search field and is a safe, relevant first action. | |
| </think> | |
| <tool_call> | |
| <function=click> | |
| <parameter=x> | |
| 482 | |
| </parameter> | |
| <parameter=y> | |
| 212 | |
| </parameter> | |
| </function> | |
| </tool_call> | |
| ``` | |
| Coordinates are normalized to 0โ1000. Convert them to pixels with `x / 1000 ร width` and `y / 1000 ร height`. | |
| ### Run the agent (vLLM) | |
| vLLM does not include the KORMo architecture either. The KORMo text layers have the Llama structure, so `vllm/convert_to_vllm.py` renames two parameter names and writes a folder vLLM loads natively. `vllm/sitecustomize.py` turns SigLIP post-layernorm back on inside vLLM. | |
| **1. Convert (once)** | |
| ```bash | |
| pip install -U "vllm>=0.23.0" openai playwright safetensors | |
| playwright install chromium | |
| python KORMo-Web-11B/vllm/convert_to_vllm.py KORMo-Web-11B KORMo-Web-11B-vllm | |
| ``` | |
| **2. Start the model server** | |
| ```bash | |
| PYTHONPATH=$PWD/KORMo-Web-11B/vllm vllm serve ./KORMo-Web-11B-vllm --served-model-name KORMo-Web-11B \ | |
| --max-model-len 65536 --limit-mm-per-prompt '{"image":1,"video":0}' | |
| ``` | |
| Check that the server log prints `[kormo-vllm] SigLIP post_layernorm enabled`. | |
| **3. Run a browser task** | |
| ```bash | |
| python KORMo-Web-11B/examples/agent_playwright.py --model KORMo-Web-11B \ | |
| --task "๋ค๋์์์ 50๋ง ์ ์ดํ, 10์ธ์น ์ด์ ํ๋ธ๋ฆฟ 3๊ฐ๋ฅผ ์ฐพ์์ ๊ฐ๊ฒฉ์ ์๋ ค์ค" | |
| ``` | |
| The agent opens Chromium, prints each action as it goes, and finishes with the model's answer. | |
| ## How It Works | |
| ``` | |
| user <user_request> task </user_request> + observation (tabs, url, screenshot) | |
| assistant <think> what I see ยท what is done ยท plan </think> <tool_call> one action </tool_call> | |
| tool action result + new observation (newest screenshot only) | |
| โฆ repeat until finished(answer) | |
| ``` | |
| - **Action space:** 22 browser tools: click, type, hotkey, scroll, drag, goto, tab control, `finished`, and others ([`tools.json`](tools.json)). | |
| - **Memory:** the full text history of thoughts, actions and observations is kept. Only the current screenshot is sent. | |
| - **Recommended settings:** temperature 0.6, top_p 0.95, top_k 20, thinking on, up to 16k tokens per step and 50 steps. These are already set in `generation_config.json`. | |
| The exact message format, parsing rules and feedback strings are in **[HARNESS.md](HARNESS.md)**. Follow it if you build your own harness. | |
| ## Evaluation Details | |
| | | | | |
| |---|---| | |
| | Tasks | Online-Mind2Web: 300 English, 312 Korean | | |
| | Judge | GPT-6 | | |
| ## Files | |
| | File | Description | | |
| |---|---| | |
| | `model.safetensors` ยท `config.json` | Weights and config (LLaVA-OneVision: SigLIP + KORMo-10B text model) | | |
| | `_configuration_kormo.py` ยท `_modeling_kormo.py` | KORMo text model code for transformers 5 | | |
| | `kormo_load.py` | `register()` and `patch_vision()` | | |
| | `chat_template.jinja` | Chat template (XML tool calls, thinking on) | | |
| | `system_prompt.md` ยท `tools.json` | System prompt and 22-tool schema | | |
| | `HARNESS.md` | Full harness specification | | |
| | `examples/agent_playwright.py` | Reference browser agent (Playwright + OpenAI-compatible API). It does not execute `upload`. | | |
| | `vllm/convert_to_vllm.py` ยท `vllm/sitecustomize.py` | Convert for vLLM and the post-layernorm fix | | |
| | `generation_config.json` ยท `*processor_config.json` ยท `tokenizer*.json` ยท `special_tokens_map.json` | Sampling, processor and tokenizer configs | | |
| ## Acknowledgements | |
| Built on **[KORMo-VL](https://huggingface.co/KORMo-VL/KORMo-VL)** and **[KORMo-10B](https://huggingface.co/KORMo-Team/KORMo-10B-sft)** by KAIST MLP Lab. Evaluated on **Online-Mind2Web** and its Korean counterpart. | |
| </div> | |