Instructions to use KORMo-VL/KORMo-Web-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use KORMo-VL/KORMo-Web-4B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="KORMo-VL/KORMo-Web-4B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("KORMo-VL/KORMo-Web-4B")
model = AutoModelForMultimodalLM.from_pretrained("KORMo-VL/KORMo-Web-4B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use KORMo-VL/KORMo-Web-4B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "KORMo-VL/KORMo-Web-4B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KORMo-VL/KORMo-Web-4B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/KORMo-VL/KORMo-Web-4B

SGLang

How to use KORMo-VL/KORMo-Web-4B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "KORMo-VL/KORMo-Web-4B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KORMo-VL/KORMo-Web-4B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "KORMo-VL/KORMo-Web-4B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KORMo-VL/KORMo-Web-4B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use KORMo-VL/KORMo-Web-4B with Docker Model Runner:
```
docker model run hf.co/KORMo-VL/KORMo-Web-4B
```

KORMo-Web-4B / HARNESS.md

JLee0

Model card, harness spec, prompt, tools and example in the KORMo-Web-9B format

5bdd6d1 verified about 2 hours ago

preview code

Raw History Blame Contribute Delete

7.89 kB

KORMo-Web-4B Harness Specification

Running a GUI agent requires more than the chat template. This file covers the parts the template does not: what goes into each message, how the screenshot and history are managed, how actions are parsed and executed, and how feedback is written back to the model.

Match this spec as closely as you can. The model was trained on this exact layout, and deviating from it (for example by sending several past screenshots, dropping the tab list, or using pixel coordinates) measurably hurts it.

A runnable reference implementation is in examples/agent_playwright.py.

1. What you need

Piece	File	Role
Weights + tokenizer/processor	`model.safetensors`, `tokenizer*.json`, `processor_config.json`	Qwen3.5 VL architecture
Chat template	`chat_template.jinja`	Renders tools, `<think>`, XML tool calls, `<tool_response>`
System prompt	`system_prompt.md`	Sent verbatim as the `system` message
Tool schema	`tools.json`	The 22 tools, passed as `tools=`
Sampling	`generation_config.json`	`temperature 0.6 · top_p 0.95 · top_k 20`
Harness	this file	Message layout, parsing, execution, feedback

2. Conversation layout

system     system_prompt.md                     (tools are injected by the chat template)
user       <user_request>{task}</user_request>
           <observation>…</observation>         ← screenshot only if this is the current step
assistant  <think>…</think>\n\n<tool_call>…</tool_call>
tool       {feedback}\n<observation>…</observation>
assistant  …
tool       {feedback}\n<observation> … screenshot: <image> </observation>   ← newest step only

Full history is kept. Every previous assistant turn stays in the context with its full <think> and <tool_call>, and so does every tool feedback and observation. Nothing is truncated; the 50-step episodes in our evaluation stayed inside a 131k-token context.
Only one screenshot is sent: the newest one. When a new observation is appended, the screenshot is removed from the previous observation, but its text is kept.
Tool results use role: "tool". The chat template wraps consecutive tool messages into a single user turn as <tool_response>…</tool_response>.
Thinking is on. Pass chat_template_kwargs={"enable_thinking": true}; the generation prompt then opens with <think>\n.

2.1 Observation block

<observation>
screen size: 1280 x 720
Current tab: 0
Available tabs:
- Tab 0 (active): https://www.google.com/
- Tab 1: https://chatgpt.com/

url: https://www.google.com/
screenshot:
<image>
</observation>

screen size is the viewport size in pixels, which is also the screenshot size.
List every open tab with its 0-based index, and mark the active tab with (active).
Cut URLs longer than 300 characters to their first 300 characters followed by …. Non-ASCII characters, such as Korean, are shown decoded rather than percent-encoded.
For past observations, drop the screenshot: line and the image. The block then ends right after the url: line.

2.2 First user message

<user_request>
{task text}
</user_request>
<observation>
…
</observation>

3. Model output and parsing

Each assistant turn contains an optional reasoning block followed by exactly one tool call in Qwen XML form:

<think>
I can see the search box at the top of the page. I have not searched yet …
My plan: 1. … 2. …
</think>

<tool_call>
<function=click>
<parameter=x>
498
</parameter>
<parameter=y>
101
</parameter>
</function>
</tool_call>

Parsing rules:

Remove everything up to and including </think>.
The remaining text must contain exactly one <tool_call>…</tool_call>.
Read <function=NAME> and each <parameter=KEY>\nVALUE\n</parameter>.
If the schema types a parameter as string (for example text, answer, url, path), keep its value as raw text. Decode every other value with json.loads (numbers, the keys array, index).
Check the parsed call against tools.json (required parameters, no unknown names). If parsing fails, do not act; reply with the failure as feedback (see §5).

Do not enable vLLM's server-side tool parser unless it supports this XML format. Parsing the raw text yourself is the simplest option.

4. Action space (22 tools)

Coordinates x, y, x2, y2 are normalized to 0–1000 relative to the screenshot. Convert them to pixels with px = x / 1000 × width and py = y / 1000 × height. Scroll deltas are signed pixels and are not normalized.

Tool	Arguments	Execute as
`click` / `double_click` / `right_click`	`x, y`	mouse click at the point
`hover`	`x, y`	move the mouse without clicking
`focus`	`x, y`	focus the element at the point
`drag`	`x, y, x2, y2`	press, move, release
`type`	`text`	type into the currently focused input; never clears it first
`clear`	–	clear the currently focused input (Ctrl+A, Backspace)
`hotkey`	`keys: [..]`	press the keys together, modifiers first, e.g. `["Control","a"]`
`scroll`	`scroll_dy`, `scroll_dx?`	mouse wheel by those pixels (positive = down / right)
`wait`	`seconds`	sleep
`goto`	`url`	navigate the active tab
`go_back` / `go_forward`	–	browser history
`new_tab` / `switch_tab(index)` / `close_tab`	– / `index` / –	tab management
`upload`	`path`	set the file on the open file picker
`finished`	`answer?`	terminal – the task is done; `answer` is the final answer
`fail`	–	terminal – the task cannot be done
`EXIT`	–	terminal – explicit stop
`unknown`	–	no-op; reply `No action executed.`

Two execution details from training:

If an action opens a new tab, for example a target=_blank link, switch to the new tab automatically.
type and clear work only on the focused element. The model learned to click or focus the input first.

5. Feedback strings

The text before <observation> in each tool message tells the model what happened. These are the patterns used in training; staying close to them helps.

Succeed: `click` on <input> type=text "Search" at (498, 101) executed. Note: no visible navigation or new tab detected.
Succeed: `click` on <h3> "Product title …" at (294, 314) executed. Page navigated to https://example.com/item/1.
Succeed: `click` on <span> "Menu" at (139, 342) executed. New tab opened (tab 1).
Succeed: `type` typed "KAIST graduate credit exchange".
Succeed: `hotkey` ['Enter'] executed. Page navigated to https://…
Succeed: `scroll` by (dx=0, dy=504) executed.
Succeed: waited 10.0 seconds.
Succeed: `goto` navigated to https://chatgpt.com/.
Succeed: `new_tab` opened blank tab (tab 1).
Succeed: `switch_tab` switched to tab 0 (https://…).
Failed: `goto` execution failed: Page.goto: Timeout 60000ms exceeded.

Coordinates in feedback are the model's normalized values, not pixels.
The element description (<tag> type=… role=… "text") is optional, but it helps.
On failure, use Failed: \{tool}` execution failed: {reason}`.

6. Recommended settings

Setting	Value
Sampling	`temperature 0.6`, `top_p 0.95`, `top_k 20`, `repetition_penalty 1.0`, `presence_penalty 0`
`max_tokens` per step	16384 (thinking on)
Context (`--max-model-len`)	131072
Viewport	1280 × 720 (77% of training trajectories; the rest were 1280×1000, 1366×768, 1440×900 and 1536×864)
Images per request	1 (`--limit-mm-per-prompt '{"image":1,"video":0}'`)
Max steps	50

All scores in the README were measured with these settings.