Image-Text-to-Text
Transformers
Safetensors
Korean
English
qwen3_5
gui-agent
web-agent
computer-use
browser
korean
conversational
Instructions to use KORMo-VL/KORMo-Web-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KORMo-VL/KORMo-Web-9B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="KORMo-VL/KORMo-Web-9B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("KORMo-VL/KORMo-Web-9B") model = AutoModelForMultimodalLM.from_pretrained("KORMo-VL/KORMo-Web-9B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use KORMo-VL/KORMo-Web-9B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "KORMo-VL/KORMo-Web-9B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-9B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/KORMo-VL/KORMo-Web-9B
- SGLang
How to use KORMo-VL/KORMo-Web-9B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-9B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-9B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-9B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-9B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use KORMo-VL/KORMo-Web-9B with Docker Model Runner:
docker model run hf.co/KORMo-VL/KORMo-Web-9B
README: simplify training and benchmark descriptions
Browse files
README.md
CHANGED
|
@@ -52,15 +52,14 @@ tags:
|
|
| 52 |
<span class="pill">9B · Qwen3.5</span>
|
| 53 |
</div>
|
| 54 |
|
| 55 |
-
**KORMo-Web-9B** is a **9B-parameter vision-language web agent** that completes real browser tasks from **screenshots alone**. It reads the user's request and the current screen, reasons in a `<think>` block, and emits **one browser action at a time** from a fixed set of **22 tools**. The model is built on **Qwen3.5-9B**
|
| 56 |
|
| 57 |
> It is built for the long, multi-site tasks people actually do. In the example below, the model asks ChatGPT a question, checks the matching notice on a university portal, and writes the result into Notion, in one session and in Korean.
|
| 58 |
|
| 59 |
## Highlights
|
| 60 |
|
| 61 |
-
- **Korean-first web agent.** More than half of the training trajectories run on real Korean websites, including university portals, Korean shopping and price-comparison sites, Notion, Google Workspace and AI chat services. The tasks are written in natural, casual Korean.
|
| 62 |
- **Grounded, verifiable actions.** The model sees no DOM and no accessibility tree. Every pointer action is a coordinate normalized to 0–1000 on the screenshot, and the system prompt tells the model to check each result in the next observation before it moves on.
|
| 63 |
-
- **Robust to
|
| 64 |
- **Cross-site and multi-tab.** The model has native tab tools (`new_tab`, `switch_tab`, `close_tab`), find-in-page, drag-to-copy, and `upload` for file pickers.
|
| 65 |
|
| 66 |
## Benchmarks
|
|
@@ -72,9 +71,7 @@ tags:
|
|
| 72 |
| Online-Mind2Web (EN) | 300 | 153 | **51.0** |
|
| 73 |
| Online-Mind2Web (KO) | 312 | 150 | **48.1** |
|
| 74 |
|
| 75 |
-
*
|
| 76 |
-
|
| 77 |
-
> Live-web scores move with site changes, bot protection, pop-ups and judge versions. Compare models only under the same harness and on the same dates.
|
| 78 |
|
| 79 |
## Quickstart
|
| 80 |
|
|
@@ -214,15 +211,12 @@ See **[HARNESS.md](HARNESS.md)** for the exact observation format, the parsing r
|
|
| 214 |
|
| 215 |
## Training
|
| 216 |
|
| 217 |
-
| Stage | Data |
|
| 218 |
-
|---|---|
|
| 219 |
-
|
|
| 220 |
-
|
|
| 221 |
-
| 2. Agent SFT | **1,259 trajectories / 29,770 steps** (KO 726 · EN 533) on real websites | Full fine-tuning · 3 epochs · LR 1e-5 (vision 2e-6) · batch 64 · warmup 3% · max length 90k tokens · 32 GPUs |
|
| 222 |
|
| 223 |
- **Supervision:** each training sample is one step. Only the final assistant turn (its `<think>` and tool call) is trained, with the full preceding history as context.
|
| 224 |
-
- **Instruction rewriting:** for half of the trajectories, the task text was replaced by a rewritten version. The rewrite is more conversational and explains the user's intent and context, while keeping every concrete requirement. The other half keeps the original instruction.
|
| 225 |
-
- **Domains:** Korean university portals, Notion, Google Sheets and Docs, ChatGPT and Gemini, shopping and price-comparison sites, search engines, and English counterparts. Many tasks span several sites and tabs.
|
| 226 |
|
| 227 |
## Limitations
|
| 228 |
|
|
@@ -233,11 +227,11 @@ See **[HARNESS.md](HARNESS.md)** for the exact observation format, the parsing r
|
|
| 233 |
|
| 234 |
## 한국어 요약
|
| 235 |
|
| 236 |
-
**KORMo-Web-9B**는 스크린샷만 보고 웹 브라우저 작업을 수행하는 9B 비전-언어 에이전트입니다. Qwen3.5-9B를 웹
|
| 237 |
|
| 238 |
- **추론 설정:** temperature 0.6 · top_p 0.95 · top_k 20 · thinking 켬 · 스크린샷은 현재 화면 1장 · 이력은 전부 유지
|
| 239 |
- **필요한 파일:** `chat_template.jinja`, `system_prompt.md`, `tools.json`. 메시지 조립 규칙은 [`HARNESS.md`](HARNESS.md)에 있습니다.
|
| 240 |
-
- **성능:** Online-Mind2Web EN 51.0 · KO 48.1
|
| 241 |
|
| 242 |
## Acknowledgements
|
| 243 |
|
|
|
|
| 52 |
<span class="pill">9B · Qwen3.5</span>
|
| 53 |
</div>
|
| 54 |
|
| 55 |
+
**KORMo-Web-9B** is a **9B-parameter vision-language web agent** that completes real browser tasks from **screenshots alone**. It reads the user's request and the current screen, reasons in a `<think>` block, and emits **one browser action at a time** from a fixed set of **22 tools**. The model is built on **Qwen3.5-9B** and trained on **large-scale web understanding data and web trajectories**.
|
| 56 |
|
| 57 |
> It is built for the long, multi-site tasks people actually do. In the example below, the model asks ChatGPT a question, checks the matching notice on a university portal, and writes the result into Notion, in one session and in Korean.
|
| 58 |
|
| 59 |
## Highlights
|
| 60 |
|
|
|
|
| 61 |
- **Grounded, verifiable actions.** The model sees no DOM and no accessibility tree. Every pointer action is a coordinate normalized to 0–1000 on the screenshot, and the system prompt tells the model to check each result in the next observation before it moves on.
|
| 62 |
+
- **Robust to phrasing.** Handles both short commands and long, conversational requests.
|
| 63 |
- **Cross-site and multi-tab.** The model has native tab tools (`new_tab`, `switch_tab`, `close_tab`), find-in-page, drag-to-copy, and `upload` for file pickers.
|
| 64 |
|
| 65 |
## Benchmarks
|
|
|
|
| 71 |
| Online-Mind2Web (EN) | 300 | 153 | **51.0** |
|
| 72 |
| Online-Mind2Web (KO) | 312 | 150 | **48.1** |
|
| 73 |
|
| 74 |
+
*Evaluated on live websites. KO is a Korean version of Online-Mind2Web.*
|
|
|
|
|
|
|
| 75 |
|
| 76 |
## Quickstart
|
| 77 |
|
|
|
|
| 211 |
|
| 212 |
## Training
|
| 213 |
|
| 214 |
+
| Stage | Data |
|
| 215 |
+
|---|---|
|
| 216 |
+
| Base | Qwen3.5-9B |
|
| 217 |
+
| Training | Large-scale web understanding data + web trajectories (Korean and English) |
|
|
|
|
| 218 |
|
| 219 |
- **Supervision:** each training sample is one step. Only the final assistant turn (its `<think>` and tool call) is trained, with the full preceding history as context.
|
|
|
|
|
|
|
| 220 |
|
| 221 |
## Limitations
|
| 222 |
|
|
|
|
| 227 |
|
| 228 |
## 한국어 요약
|
| 229 |
|
| 230 |
+
**KORMo-Web-9B**는 스크린샷만 보고 웹 브라우저 작업을 수행하는 9B 비전-언어 에이전트입니다. Qwen3.5-9B를 바탕으로 대규모 웹 이해 데이터와 웹 궤적(trajectory)으로 학습했습니다. 매 스텝 `<think>`로 생각한 뒤 22개 도구 중 하나를 호출합니다. 좌표는 0–1000으로 정규화합니다.
|
| 231 |
|
| 232 |
- **추론 설정:** temperature 0.6 · top_p 0.95 · top_k 20 · thinking 켬 · 스크린샷은 현재 화면 1장 · 이력은 전부 유지
|
| 233 |
- **필요한 파일:** `chat_template.jinja`, `system_prompt.md`, `tools.json`. 메시지 조립 규칙은 [`HARNESS.md`](HARNESS.md)에 있습니다.
|
| 234 |
+
- **성능:** Online-Mind2Web EN 51.0 · KO 48.1
|
| 235 |
|
| 236 |
## Acknowledgements
|
| 237 |
|