Image-Text-to-Text
MLX
Safetensors
Korean
English
qwen3_5
gui-agent
web-agent
computer-use
browser
korean
conversational
4-bit precision
Instructions to use KORMo-VL/KORMo-Web-4B-mlx-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use KORMo-VL/KORMo-Web-4B-mlx-4bit with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("KORMo-VL/KORMo-Web-4B-mlx-4bit") config = load_config("KORMo-VL/KORMo-Web-4B-mlx-4bit") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use KORMo-VL/KORMo-Web-4B-mlx-4bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "KORMo-VL/KORMo-Web-4B-mlx-4bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "KORMo-VL/KORMo-Web-4B-mlx-4bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use KORMo-VL/KORMo-Web-4B-mlx-4bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "KORMo-VL/KORMo-Web-4B-mlx-4bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default KORMo-VL/KORMo-Web-4B-mlx-4bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use KORMo-VL/KORMo-Web-4B-mlx-4bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "KORMo-VL/KORMo-Web-4B-mlx-4bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "KORMo-VL/KORMo-Web-4B-mlx-4bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
|
Download README.md from KORMo-VL/KORMo-Web-4B-mlx-4bit: direct link, hf CLI and curl.
- Browser
- Download file 10.7 kB
-
https://huggingface.co/KORMo-VL/KORMo-Web-4B-mlx-4bit/resolve/main/README.md
- Command line
-
hf download hf://KORMo-VL/KORMo-Web-4B-mlx-4bit/README.md
-
curl -L -o README.md https://huggingface.co/KORMo-VL/KORMo-Web-4B-mlx-4bit/resolve/main/README.md
10.7 kB
| license: apache-2.0 | |
| base_model: | |
| - KORMo-VL/KORMo-Web-4B | |
| base_model_relation: quantized | |
| language: | |
| - ko | |
| - en | |
| pipeline_tag: image-text-to-text | |
| library_name: mlx | |
| tags: | |
| - mlx | |
| - gui-agent | |
| - web-agent | |
| - computer-use | |
| - browser | |
| - korean | |
| <style> | |
| #kormo-web-card { font-size: 16px; line-height: 1.6; overflow-wrap: break-word; } | |
| #kormo-web-card h1 { margin: 0 0 12px; font-size: 30px; line-height: 1.25; letter-spacing: -0.018em; } | |
| #kormo-web-card h2 { margin: 30px 0 10px; font-size: 22px; line-height: 1.3; letter-spacing: -0.012em; } | |
| #kormo-web-card h3 { margin: 22px 0 8px; font-size: 17px; font-weight: 700; line-height: 1.4; } | |
| #kormo-web-card h2 + h3 { margin-top: 14px; } | |
| #kormo-web-card p { margin: 0 0 10px; } | |
| #kormo-web-card ul, #kormo-web-card ol { margin: 6px 0 12px; padding-left: 22px; } | |
| #kormo-web-card li { margin: 4px 0; line-height: 1.55; } | |
| #kormo-web-card table { width: 100%; margin: 6px 0 14px; border-collapse: collapse; font-size: 14px; line-height: 1.45; font-variant-numeric: tabular-nums; } | |
| #kormo-web-card th, #kormo-web-card td { padding: 7px 10px; vertical-align: middle; } | |
| #kormo-web-card blockquote { margin: 12px 0 16px; padding: 0 0 0 14px; font-size: 15px; } | |
| #kormo-web-card blockquote p { margin: 0; } | |
| #kormo-web-card pre { margin: 10px 0 14px; padding: 14px 16px; overflow-x: auto; line-height: 1.5; } | |
| #kormo-web-card pre code { font-size: 13px; } | |
| #kormo-web-card > p:has(> em:only-child) { margin: 4px 0 14px; font-size: 13px; } | |
| #kormo-web-card .pills { margin: 4px 0 18px; } | |
| #kormo-web-card .pill { display: inline-block; margin: 0 6px 6px 0; padding: 3px 10px; border-radius: 999px; border: 1px solid rgba(127,127,127,.35); font-size: 13px; } | |
| @media (max-width: 640px) { | |
| #kormo-web-card h1 { font-size: 26px; } | |
| #kormo-web-card table { font-size: 13px; } | |
| #kormo-web-card th, #kormo-web-card td { padding: 7px 6px; } | |
| } | |
| </style> | |
| <div id="kormo-web-card"> | |
| <video src="https://huggingface.co/KORMo-VL/KORMo-Web-4B-mlx-4bit/resolve/main/assets/gui_final_demo_maksed.mp4" controls autoplay muted loop playsinline width="100%"></video> | |
| # KORMo-Web-4B | |
| **KORMo-Web-4B** is a 4B vision-language agent that completes real browser tasks **from screenshots alone**. Give it a request in Korean or English; it plans, clicks, types and moves between sites and tabs until the task is done. In the same live-web evaluation, it beats every open-weight web agent we compared, **by +9.6 points in English and +22.1 points in Korean**. | |
| On **Online-Mind2Web** it reaches **42.3% in English** and **42.3% in Korean**, which is **+22.3 / +28.8 points** over the base Qwen3.5-4B. | |
| ## What Makes KORMo-Web Different? | |
| - **10M+ Atomic-Level Web Pretraining Samples.** Large-scale Korean and English web understanding pretraining teaches the model what individual UI elements **are, where they are, and what they do**โcovering text, location, appearance, layout, function, and state. | |
| - **100K+ Web Interaction Trajectories.** Bilingual browser demonstrations teach the model to translate fine-grained web understanding into real-world, multi-step actions. | |
| - **Open Models and Data.** Releasing model weights, large-scale bilingual web pretraining data, interaction trajectories, and evaluation benchmarks to support reproducible web agent research. | |
| ## Performance | |
| All models below ran **the same 612 live-website tasks** (300 English, 312 Korean), each with its own prompt and action format, and were scored by **the same LLM judge**. | |
| | Model | Size | Online-Mind2Web EN | Online-Mind2Web KO | | |
| |---|---:|---:|---:| | |
| | Qwen3.5-4B (base, no fine-tuning) | 4B | 20.0 | 13.5 | | |
| | Fara-1.5-4B | 4B | 20.7 | 7.7 | | |
| | UI-TARS-1.5-7B | 7B | 22.3 | 20.2 | | |
| | OpenWebRL-4B | 4B | 32.7 | 15.4 | | |
| | **KORMo-Web-4B (ours)** | **4B** | **42.3** | **42.3** | | |
| | KORMo-Web-9B (ours) | 9B | 51.0 | 48.1 | | |
| Success rate (%). The denominator is always the full task count, so crashed or empty runs count as failures. | |
| KORMo-Web-4B outperforms open web agents of similar size in both English and Korean. | |
| ## Why KORMo-Web | |
| - **Korean and English, natural requests.** Handles short commands as well as long, conversational instructions in either language. | |
| - **Screenshot-only.** It needs no DOM or accessibility tree; it acts on what is on screen, like a person, and checks each action's result on the next screenshot. | |
| - **Thinks, then acts.** A short reasoning step precedes every action, so its behaviour is easy to inspect and debug. | |
| - **Multi-site, multi-tab workflows.** It can search on one site, verify on another and write the result into a third, managing tabs on its own. | |
| - **Open and ready to run.** Apache-2.0 weights, a reference browser harness, the tool schema and the exact inference settings are all included. | |
| ## Quickstart | |
| This repository holds the **4-bit MLX** weights of KORMo-Web-4B for Apple Silicon Macs. Serve it with **[vllm-mlx](https://github.com/waybarrios/vllm-mlx)** (0.6.0 or later), which exposes the same OpenAI-compatible API as vLLM. | |
| ### Run the agent | |
| **1. Install and download** | |
| ```bash | |
| pip install -U "vllm-mlx>=0.6.0" openai playwright huggingface_hub | |
| playwright install chromium | |
| huggingface-cli download KORMo-VL/KORMo-Web-4B-mlx-4bit --local-dir KORMo-Web-4B-mlx-4bit | |
| # system prompt, tool schema and reference harness (shared with the full-precision repo) | |
| huggingface-cli download KORMo-VL/KORMo-Web-4B system_prompt.md tools.json HARNESS.md examples/agent_playwright.py \ | |
| --local-dir KORMo-Web-4B-mlx-4bit | |
| ``` | |
| **2. Start the model server** | |
| ```bash | |
| vllm-mlx serve ./KORMo-Web-4B-mlx-4bit --served-model-name KORMo-Web-4B --mllm --port 8000 | |
| ``` | |
| Do not enable `--enable-auto-tool-choice` or `--reasoning-parser`: the harness parses the raw `<think>` and `<tool_call>` text from the message content. | |
| **3. Run a browser task** | |
| ```bash | |
| python KORMo-Web-4B-mlx-4bit/examples/agent_playwright.py --model KORMo-Web-4B \ | |
| --endpoint http://localhost:8000/v1 \ | |
| --task "๋ค๋ฐฉ์์ ์ํ์ญ ๊ทผ์ฒ ๋ณด์ฆ๊ธ ์ฒ๋ง์ ์ดํ๋ก ์ค์ ํ๋ค ์๋ฃธ 3๊ฐ์ ๋ ์ฐพ์์ ์ดํด๋ณด๊ณ ๋คํ๋ฉด ์ฐพ์ ๋ด์ฉ์ ์นด์ด์คํธ ๋ฉ์ผ๋ก mlplab@kaist.ac.kr ํํ ์ ๋ฆฌํด์ ๋ณด๋ด์ค" | |
| ``` | |
| The agent opens Chromium, prints each action as it goes, and finishes with the model's answer. | |
| ### Inference example (one step) | |
| Send one screenshot with the request and get back one action: | |
| ```python | |
| import base64, json | |
| from openai import OpenAI | |
| SYSTEM = open("KORMo-Web-4B-mlx-4bit/system_prompt.md").read().strip() | |
| TOOLS = json.load(open("KORMo-Web-4B-mlx-4bit/tools.json")) | |
| shot = base64.b64encode(open("screen.png", "rb").read()).decode() | |
| obs = ("<observation>\nscreen size: 1280 x 720\nCurrent tab: 0\nAvailable tabs:\n" | |
| "- Tab 0 (active): https://www.google.com/\n\nurl: https://www.google.com/\nscreenshot:\n") | |
| messages = [ | |
| {"role": "system", "content": SYSTEM}, | |
| {"role": "user", "content": [ | |
| {"type": "text", "text": "<user_request>\n๊ตฌ๊ธ์์ KAIST ๋ฅผ ๊ฒ์ํด์ ๊ณต์ ํํ์ด์ง๋ฅผ ์ด์ด์ค.\n</user_request>\n" + obs}, | |
| {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{shot}"}}, | |
| {"type": "text", "text": "\n</observation>"}, | |
| ]}, | |
| ] | |
| client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") # vllm-mlx server | |
| out = client.chat.completions.create( | |
| model="KORMo-Web-4B", messages=messages, tools=TOOLS, | |
| temperature=0.6, top_p=0.95, max_tokens=16384, | |
| extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": True}}, | |
| ) | |
| print(out.choices[0].message.content) | |
| ``` | |
| The output looks like this (illustrative): | |
| ```text | |
| <think> | |
| I can see the Google home page with an empty search box in the center. I have not searched yet. | |
| My plan: 1. Click the search box. 2. Type "KAIST". 3. Press Enter and open the official site. | |
| </think> | |
| <tool_call> | |
| <function=click> | |
| <parameter=x> | |
| 500 | |
| </parameter> | |
| <parameter=y> | |
| 427 | |
| </parameter> | |
| </function> | |
| </tool_call> | |
| ``` | |
| Coordinates in the tool call are normalized to 0โ1000. Convert them to pixels with `x / 1000 ร width` and `y / 1000 ร height`. | |
| ## How It Works | |
| ``` | |
| user <user_request> task </user_request> + observation (tabs, url, screenshot) | |
| assistant <think> what I see ยท what is done ยท plan </think> <tool_call> one action </tool_call> | |
| tool action result + new observation (newest screenshot only) | |
| โฆ repeat until finished(answer) | |
| ``` | |
| - **Action space:** 22 browser tools: click, type, hotkey, scroll, drag, goto, tab control, `finished`, and others ([`tools.json`](tools.json)). | |
| - **Memory:** the full text history of thoughts, actions and observations is kept. Only the current screenshot is sent. | |
| - **Recommended settings:** temperature 0.6, top_p 0.95, top_k 20, thinking on, up to 16k tokens per step and 50 steps. These are already set in `generation_config.json`. | |
| The exact message format, parsing rules and feedback strings are in **[HARNESS.md](HARNESS.md)**. Follow it if you build your own harness. | |
| ## Evaluation Details | |
| | | | | |
| |---|---| | |
| | Tasks | Online-Mind2Web: 300 English, 312 Korean | | |
| | Judge | GPT-6 | | |
| ## Files | |
| | File | Description | | |
| |---|---| | |
| | `chat_template.jinja` | Chat template (Qwen3.5 XML tool calls, thinking on) | | |
| | `system_prompt.md` ยท `tools.json` | System prompt and 22-tool schema, verbatim from training | | |
| | `HARNESS.md` | Full harness specification | | |
| | `examples/agent_playwright.py` | Reference browser agent (Playwright + OpenAI-compatible API). It does not execute `upload`. | | |
| | `config.json` ยท `generation_config.json` ยท `processor_config.json` ยท `tokenizer*.json` | Model, sampling and processor configs | | |
| ## ํ๊ตญ์ด ์์ฝ | |
| **KORMo-Web-4B**๋ ์ฌ๋์ฒ๋ผ ์น ํ๋ฉด์ ๋ณด๊ณ ํด๋ฆญยท์ ๋ ฅยทํ์์ ์ํํ๋ **4B ๊ท๋ชจ์ ํ์ ์๊ฐ ๊ธฐ๋ฐ ์น ์์ด์ ํธ**์ ๋๋ค. | |
| - **๋๊ท๋ชจ ์น ์ฌ์ ํ์ต:** 1,000๋ง ๊ฑด ์ด์์ ํ์ ์น ํ์ต ์ํ๋ก UI ์์์ ์์นยท์๋ฏธยท๊ตฌ์กฐยท๊ธฐ๋ฅ ๋ฑ์ ์ธ๋ฐํ๊ฒ ํ์ต(Atomic-Level Pretraining) | |
| - **์ค์ ์น ํ๋ ํ์ต:** 10๋ง ๊ฑด ์ด์์ ํ๊ตญ์ธ ํ์ ์น ์ฌ์ฉ ๊ถค์ ์ ํ์ตํด ์ฌ๋ฌ ์น์ฌ์ดํธ์ ํญ์ ๋๋๋๋ ๋ณต์กํ ์์ ์ํ | |
| - **๋ฐ์ด๋ ์ฑ๋ฅ:** Online-Mind2Web์์ ์์ดยทํ๊ตญ์ด ๋ชจ๋ **42.3% ์ฑ๊ณต๋ฅ **, ๋น๊ตํ ๊ณต๊ฐ ๋ชจ๋ธ ์ค ์ต๊ณ ์ฑ๋ฅ(๊ธฐ์กด ์ต๊ณ ๋๋น +9.6%p / +22.1%p) | |
| - **์๊ฐ ์ค์ฌ ์น ์กฐ์:** ๋ณ๋์ DOM ์ ๋ณด ์์ด ์คํฌ๋ฆฐ์ท๋ง์ผ๋ก ์น์ ์ดํดํ๊ณ , ํ๋ ๊ฒฐ๊ณผ๋ฅผ ํ์ธํ๋ฉฐ ๋ค์ ์์ ์ํ | |
| - **๋ชจ๋ธยท๋ฐ์ดํฐ ๊ณต๊ฐ:** Apache-2.0 ๋ชจ๋ธ ๊ฐ์ค์น, ๋๊ท๋ชจ ํ์ต ๋ฐ์ดํฐ, ์น ์ฌ์ฉ ๊ถค์ , ํ๊ฐ ๋ฒค์น๋งํฌ ๋ฐ ์คํ ์ฝ๋ ๊ณต๊ฐ | |
| ## Acknowledgements | |
| Built on **[Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B)**. Evaluated on **Online-Mind2Web** and its Korean counterpart. | |
| </div> | |