Image-Text-to-Text
Transformers
Safetensors
minicpmv4_7
webbrain
minicpm-v
browser-agent
tool-calling
gui-grounding
decision-model
webbrain-compass
conversational
Instructions to use webbrain-one/webbrain-compass-vld-tiny-v5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use webbrain-one/webbrain-compass-vld-tiny-v5 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="webbrain-one/webbrain-compass-vld-tiny-v5") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("webbrain-one/webbrain-compass-vld-tiny-v5") model = AutoModelForMultimodalLM.from_pretrained("webbrain-one/webbrain-compass-vld-tiny-v5", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use webbrain-one/webbrain-compass-vld-tiny-v5 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "webbrain-one/webbrain-compass-vld-tiny-v5" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "webbrain-one/webbrain-compass-vld-tiny-v5", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/webbrain-one/webbrain-compass-vld-tiny-v5
- SGLang
How to use webbrain-one/webbrain-compass-vld-tiny-v5 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "webbrain-one/webbrain-compass-vld-tiny-v5" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "webbrain-one/webbrain-compass-vld-tiny-v5", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "webbrain-one/webbrain-compass-vld-tiny-v5" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "webbrain-one/webbrain-compass-vld-tiny-v5", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use webbrain-one/webbrain-compass-vld-tiny-v5 with Docker Model Runner:
docker model run hf.co/webbrain-one/webbrain-compass-vld-tiny-v5
|
Download README.md from webbrain-one/webbrain-compass-vld-tiny-v5: direct link, hf CLI and curl.
- Browser
- Download file 8.83 kB
-
https://huggingface.co/webbrain-one/webbrain-compass-vld-tiny-v5/resolve/main/README.md
- Command line
-
hf download hf://webbrain-one/webbrain-compass-vld-tiny-v5/README.md
-
curl -L -o README.md https://huggingface.co/webbrain-one/webbrain-compass-vld-tiny-v5/resolve/main/README.md
8.83 kB
| license: other | |
| license_name: webbrain-noncommercial-research-restrictions | |
| base_model: openbmb/MiniCPM-V-4.7-1B | |
| pipeline_tag: image-text-to-text | |
| library_name: transformers | |
| tags: | |
| - webbrain | |
| - minicpm-v | |
| - browser-agent | |
| - tool-calling | |
| - gui-grounding | |
| - decision-model | |
| - webbrain-compass | |
| # WebBrain Compass VLD Tiny v5 | |
| `openbmb/MiniCPM-V-4.7-1B` fine-tuned in two rounds as one on-device model for WebBrain: Compass text and tool | |
| calls, VL (the visual fallback) and D (decisions). It covers three jobs: | |
| - **page-state decisions:** 120/126 on d1 with 0 false completions, 0.94 s per request in the browser | |
| - **the local visual fallback:** better grounding than LFM2.5-VL-450M on WebBrain's VL-3 validation | |
| - **offline text and tool calls:** close to Compass Tiny v2.1 on the routing benchmark | |
| This repository holds the **merged float32 weights**. Companion repositories: | |
| - [`webbrain-one/webbrain-compass-vld-tiny-v5-lora`](https://huggingface.co/webbrain-one/webbrain-compass-vld-tiny-v5-lora): the LoRA adapter and the merger weights. | |
| - [`webbrain-one/webbrain-compass-vld-tiny-v5-onnx`](https://huggingface.co/webbrain-one/webbrain-compass-vld-tiny-v5-onnx): the WebGPU ONNX bundle. | |
| Round 1 is kept as `webbrain-one/minicpmv47-webbrain-v1`. | |
| ## Training | |
| Two rounds on one RTX 5090 (about 20 hours in total). | |
| **Base:** `openbmb/MiniCPM-V-4.7-1B` (revision `379f120c2ff3588e6e06559ec6b9ec15cc05114e`): a SigLIP vision tower, a | |
| 16x vision-to-language merger, and a Qwen3.5-0.8B-style language model (18 Gated DeltaNet + 6 attention layers). | |
| 1. **Round 1** (`minicpmv47-webbrain-v1`): one pass over 344,139 samples: WebBrain-VL-4 (335,572 draws of VL-2/curated VL-3 observations, | |
| MolmoPoint GUI grounding, WebChain trajectories) plus the WebBrain Compass v3 maximal text release (8,583 rows). | |
| - Trainable: LoRA r = 64, alpha = 64 on all 186 language-model linear layers (43.3M parameters, fp32) and the | |
| full merger (26.0M, fp32). The vision tower is frozen. | |
| - AdamW with LoRA LR 1e-4 and merger LR 2e-5; 5,144 steps. | |
| 2. **Round 2 (this model):** continues the round 1 adapter and merger with a fresh optimizer (LoRA LR 5e-5, merger LR 1e-5; | |
| 3% warmup, cosine to 10%; effective batch 64; 784 steps; 51,811 samples). The data: | |
| - **Decision data:** the d1 balanced-V3 **train** split rendered in WebBrain's decision protocol, repeated 6x. | |
| It has 1,206 samples (918 verify Yes/No continuations and 288 option readouts), token-exact with the extension's | |
| prompts. | |
| - **Text data:** the Compass v3 release once more (17% of samples, up from 2.5%). | |
| - **Replay data:** a 36,000-draw VL-4 replay (WebChain 12k, VL-2 12k, curated VL-3 3k, MolmoPoint 9k). | |
| Loss covers completion tokens only (per-sample mean), with thinking off and a 32K maximum length. | |
| The d1 validation split was used only for the eval loss and probe selection. The d1 final test split never fed | |
| training or selection. | |
| Final validation loss (full sets, round 1 → round 2): VL 0.295 → 0.288, Compass 1.138 → 1.116. On d1 validation decision | |
| samples the loss fell from 4.00 (verify) / 2.32 (readout) at the start of round 2 (first 96 samples) to | |
| 0.066 / 0.067 at the end (all 594). | |
| ## Results | |
| ### Page-state decisions: d1 balanced-V3 final test (126 categorical questions) | |
| Scored through WebBrain's decision protocol in the browser (WebGPU) on an RTX 5090. One request is a screenshot, | |
| its page state and its questions. The d1 final test split was never trained on. | |
| | Model | Correct | Completions found | False completions | Unknown recognized | Time per request | | |
| |---|---:|---:|---:|---:|---:| | |
| | **v5, WebGPU q8 + probe** | **120/126** | **24/24** | **0/48** | **24/24** | 0.94 s | | |
| | **v5, WebGPU q4 (lite) + probe** | **120/126** | 24/24 | 0/48 | 24/24 | 0.94 s | | |
| | Round 1, WebGPU q8 + probe | 103/126 | 22/24 | 11/48 | 13/24 | 0.85 s | | |
| | MiniCPM-V-4.7-1B untuned, WebGPU q8 + probe | 104/126 | 23/24 | 8/48 | 16/24 | n/a | | |
| | Bonsai 2 2B Vision + probe | 110/126 | 21/24 | 4/48 | 20/24 | 2.2 s | | |
| | d1-omni-600M | 92/126 | n/a | 1/48 | n/a | 0.75 s | | |
| - The probe is no longer needed. Without it, v5's raw Yes − No verify margins score 121/126 in PyTorch and 120/126 | |
| in WebGPU q8. In PyTorch the standard and conservative probes (fit on d1 train, chosen on validation) also score | |
| 121/126, and the generated option readout alone scores 120/126. | |
| - A caution on scope: d1 is a synthetic benchmark. v5 was fine-tuned on d1's train families, which are disjoint from | |
| the test families but follow the same design. Bonsai's backbone was not trained on d1, only its probe was. | |
| Validate v5 on real extension decisions before relying on these margins. | |
| ### Visual fallback: WebBrain VL-3 frozen validation, same 1,022 rows | |
| Greedy generation, scored with the VL-3 pipeline's own scorer. The rows are every MolmoPoint grounding row except | |
| actionable inventory (first 20), the first 80 curated VL-3 rows and every 20th VL-2 row. LFM2.5-VL-450M (VL-3, | |
| step 3750) is WebBrain's current visual fallback. | |
| | Metric | v5 | Round 1 | LFM2.5-VL-450M | | |
| |---|---:|---:|---:| | |
| | Schema-valid outputs | **96.2%** | 95.8% | 92.7% | | |
| | Point within 0.05 (510 rows) | **65.5%** | 58.8% | 45.9% | | |
| | Mean point distance | **0.156** | 0.194 | 0.161 | | |
| | Grounded interaction: point within 0.05 (255) | **57.3%** | 51.0% | 29.8% | | |
| | Intent → point/box: point within 0.05 (255) | **73.7%** | 66.7% | 62.1% | | |
| | Box IoU | **0.175** | 0.113 | 0.050 | | |
| | Point inside the target box | **33.3%** | 23.3% | 10.4% | | |
| | Six-section observation valid, VL-2 / VL-3 | 98.7% / **72.5%** | 99.4% / 65.0% | 97.5% / 38.8% | | |
| | Actionable inventory valid (first 20) | 25% | 30% | 10% | | |
| On intent → point/box the mean point distance is still worse than LFM's (0.190 vs 0.132; round 1 0.248), although more | |
| answers land within 0.05: a minority of far misses. Long inventory lists often do not terminate, for every model. | |
| ### Text and tool calls: WebBrain `test/llm` routing benchmark (compact tier) | |
| Served from PyTorch BF16 with seed 3407 and thinking off. The table counts the 89 scenarios scored in the Compass | |
| Tiny v2.1 run: strict = `ideal`, loose = `ideal` + `ideal_name`. | |
| | Model | First-turn structured calls | Strict exact action | Loose tool family | Discouraged actions | | |
| |---|---:|---:|---:|---:| | |
| | **v5** (0.8B language model) | 95/100 | 9/89 | **48/89** | 3 | | |
| | Round 1 | 97/100 | 10/89 | 38/89 | 5 | | |
| | MiniCPM-V-4.7-1B, untuned | 98/100 | 3/89 | 30/89 | 2 | | |
| | Compass Tiny v2.1 (2.6B, WebGPU q4f16) | 98/100 | **15/89** | 43/89 | 3 | | |
| ## Usage | |
| ```python | |
| import torch | |
| from PIL import Image | |
| from transformers import AutoModelForImageTextToText, AutoProcessor | |
| repo = "webbrain-one/webbrain-compass-vld-tiny-v5" | |
| processor = AutoProcessor.from_pretrained(repo) | |
| model = AutoModelForImageTextToText.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda").eval() | |
| messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Describe this page."}]}] | |
| prompt = processor.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, tokenize=False) | |
| inputs = processor(text=[prompt], images=[Image.open("screenshot.png").convert("RGB")], return_tensors="pt").to(model.device) | |
| out = model.generate(**inputs, do_sample=False, max_new_tokens=512) | |
| print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)) | |
| ``` | |
| Tool calls use the chat template's native format: `<tool_call>`, then `<function=NAME>`, then one `<parameter=KEY>` | |
| block per argument. Pass OpenAI-style `tools=` to `apply_chat_template`, and parse the reply with a Qwen3.5 / | |
| Qwen3-coder style tool parser that types each argument from the tool's JSON schema. | |
| For decisions, use WebBrain's verify protocol: one "Proposed answer: X ... Is the proposed answer correct? Answer Yes or No." continuation per option, compared by the Yes − No logit. See `minicpmv-decisions.mjs` in the ONNX repository. | |
| ## License | |
| The base model is Apache-2.0. This fine-tune was trained on data that includes noncommercial-licensed sources (for | |
| example the WebLINX-derived Compass subset), so the fine-tuned weights are released for **noncommercial research | |
| only** under the WebBrain noncommercial research restrictions. Private hosting does not expand those rights. | |
| Model output is never proof that a browser action succeeded. Keep WebBrain's schema validation, grounding checks and | |
| confirmations in front of any action. | |
| Training data includes WebChain v2 (`webagentlab/webchain`, WebChain Academic Research License v1.0): Fan, Sicheng, Rui Wan, Yifei Leng, Gaoning Liang, Li Ling, Yanyi Shang, and Dehan Kong. "WebChain: A Large-Scale Human-Annotated Dataset of Real-World Web Interaction Traces." arXiv:2603.05295, 2026. It also includes WebLINX (McGill-NLP, CC BY-NC-SA 4.0) and MolmoPoint-GUISyn (Ai2, ODC-BY). | |