Instructions to use webbrain-one/webbrain-compass-vld-tiny-v5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use webbrain-one/webbrain-compass-vld-tiny-v5 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="webbrain-one/webbrain-compass-vld-tiny-v5")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("webbrain-one/webbrain-compass-vld-tiny-v5")
model = AutoModelForMultimodalLM.from_pretrained("webbrain-one/webbrain-compass-vld-tiny-v5", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use webbrain-one/webbrain-compass-vld-tiny-v5 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "webbrain-one/webbrain-compass-vld-tiny-v5"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "webbrain-one/webbrain-compass-vld-tiny-v5",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/webbrain-one/webbrain-compass-vld-tiny-v5

SGLang

How to use webbrain-one/webbrain-compass-vld-tiny-v5 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "webbrain-one/webbrain-compass-vld-tiny-v5" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "webbrain-one/webbrain-compass-vld-tiny-v5",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "webbrain-one/webbrain-compass-vld-tiny-v5" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "webbrain-one/webbrain-compass-vld-tiny-v5",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use webbrain-one/webbrain-compass-vld-tiny-v5 with Docker Model Runner:
```
docker model run hf.co/webbrain-one/webbrain-compass-vld-tiny-v5
```

webbrain-one commited on 7 days ago

Commit

9d7b87f

verified ·

1 Parent(s): be6e007

Rename to WebBrain Compass VLD Tiny v5; attribution

Browse files

Files changed (1) hide show

README.md +23 -19

README.md CHANGED Viewed

@@ -11,11 +11,13 @@ tags:
 - tool-calling
 - gui-grounding
 - decision-model
 ---
-# MiniCPM-V-4.7 WebBrain v2
-MiniCPM-V-4.7-1B fine-tuned in two rounds as one on-device model for WebBrain. It covers three jobs:
 - **page-state decisions:** 120/126 on d1 with 0 false completions, 0.94 s per request in the browser
 - **the local visual fallback:** better grounding than LFM2.5-VL-450M on WebBrain's VL-3 validation
@@ -23,10 +25,10 @@ MiniCPM-V-4.7-1B fine-tuned in two rounds as one on-device model for WebBrain. I
 This repository holds the **merged float32 weights**. Companion repositories:
-- [`webbrain-one/minicpmv47-webbrain-v2-lora`](https://huggingface.co/webbrain-one/minicpmv47-webbrain-v2-lora): the LoRA adapter and the merger weights.
-- [`webbrain-one/minicpmv47-webbrain-v2-onnx`](https://huggingface.co/webbrain-one/minicpmv47-webbrain-v2-onnx): the WebGPU ONNX bundle.
-The previous round is kept as `webbrain-one/minicpmv47-webbrain-v1`.
 ## Training
@@ -34,12 +36,12 @@ Two rounds on one RTX 5090 (about 20 hours in total).
 **Base:** `openbmb/MiniCPM-V-4.7-1B` (revision `379f120c2ff3588e6e06559ec6b9ec15cc05114e`): a SigLIP vision tower, a
 16x vision-to-language merger, and a Qwen3.5-0.8B-style language model (18 Gated DeltaNet + 6 attention layers).
-1. **Round 1 (v1):** one pass over 344,139 samples: WebBrain-VL-4 (335,572 draws of VL-2/curated VL-3 observations,
    MolmoPoint GUI grounding, WebChain trajectories) plus the WebBrain Compass v3 maximal text release (8,583 rows).
    - Trainable: LoRA r = 64, alpha = 64 on all 186 language-model linear layers (43.3M parameters, fp32) and the
      full merger (26.0M, fp32). The vision tower is frozen.
    - AdamW with LoRA LR 1e-4 and merger LR 2e-5; 5,144 steps.
-2. **Round 2 (v2, this model):** continues the v1 adapter and merger with a fresh optimizer (LoRA LR 5e-5, merger LR 1e-5;
    3% warmup, cosine to 10%; effective batch 64; 784 steps; 51,811 samples). The data:
    - **Decision data:** the d1 balanced-V3 **train** split rendered in WebBrain's decision protocol, repeated 6x.
      It has 1,206 samples (918 verify Yes/No continuations and 288 option readouts), token-exact with the extension's
@@ -51,7 +53,7 @@ Loss covers completion tokens only (per-sample mean), with thinking off and a 32
 The d1 validation split was used only for the eval loss and probe selection. The d1 final test split never fed
 training or selection.
-Final validation loss (full sets, v1 → v2): VL 0.295 → 0.288, Compass 1.138 → 1.116. On d1 validation decision
 samples the loss fell from 4.00 (verify) / 2.32 (readout) at the start of round 2 (first 96 samples) to
 0.066 / 0.067 at the end (all 594).
@@ -64,19 +66,19 @@ its page state and its questions. The d1 final test split was never trained on.
 | Model | Correct | Completions found | False completions | Unknown recognized | Time per request |
 |---|---:|---:|---:|---:|---:|
-| **v2, WebGPU q8 + probe** | **120/126** | **24/24** | **0/48** | **24/24** | 0.94 s |
-| **v2, WebGPU q4 (lite) + probe** | **120/126** | 24/24 | 0/48 | 24/24 | 0.94 s |
-| v1, WebGPU q8 + probe | 103/126 | 22/24 | 11/48 | 13/24 | 0.85 s |
 | MiniCPM-V-4.7-1B untuned, WebGPU q8 + probe | 104/126 | 23/24 | 8/48 | 16/24 | n/a |
 | Bonsai 2 2B Vision + probe | 110/126 | 21/24 | 4/48 | 20/24 | 2.2 s |
 | d1-omni-600M | 92/126 | n/a | 1/48 | n/a | 0.75 s |
-- The probe is no longer needed. Without it, v2's raw Yes − No verify margins score 121/126 in PyTorch and 120/126
   in WebGPU q8. In PyTorch the standard and conservative probes (fit on d1 train, chosen on validation) also score
   121/126, and the generated option readout alone scores 120/126.
-- A caution on scope: d1 is a synthetic benchmark. v2 was fine-tuned on d1's train families, which are disjoint from
   the test families but follow the same design. Bonsai's backbone was not trained on d1, only its probe was.
-  Validate v2 on real extension decisions before relying on these margins.
 ### Visual fallback: WebBrain VL-3 frozen validation, same 1,022 rows
@@ -84,7 +86,7 @@ Greedy generation, scored with the VL-3 pipeline's own scorer. The rows are ever
 actionable inventory (first 20), the first 80 curated VL-3 rows and every 20th VL-2 row. LFM2.5-VL-450M (VL-3,
 step 3750) is WebBrain's current visual fallback.
-| Metric | v2 | v1 | LFM2.5-VL-450M |
 |---|---:|---:|---:|
 | Schema-valid outputs | **96.2%** | 95.8% | 92.7% |
 | Point within 0.05 (510 rows) | **65.5%** | 58.8% | 45.9% |
@@ -96,7 +98,7 @@ step 3750) is WebBrain's current visual fallback.
 | Six-section observation valid, VL-2 / VL-3 | 98.7% / **72.5%** | 99.4% / 65.0% | 97.5% / 38.8% |
 | Actionable inventory valid (first 20) | 25% | 30% | 10% |
-On intent → point/box the mean point distance is still worse than LFM's (0.190 vs 0.132; v1 0.248), although more
 answers land within 0.05: a minority of far misses. Long inventory lists often do not terminate, for every model.
 ### Text and tool calls: WebBrain `test/llm` routing benchmark (compact tier)
@@ -106,8 +108,8 @@ Tiny v2.1 run: strict = `ideal`, loose = `ideal` + `ideal_name`.
 | Model | First-turn structured calls | Strict exact action | Loose tool family | Discouraged actions |
 |---|---:|---:|---:|---:|
-| **v2** (0.8B language model) | 95/100 | 9/89 | **48/89** | 3 |
-| v1 | 97/100 | 10/89 | 38/89 | 5 |
 | MiniCPM-V-4.7-1B, untuned | 98/100 | 3/89 | 30/89 | 2 |
 | Compass Tiny v2.1 (2.6B, WebGPU q4f16) | 98/100 | **15/89** | 43/89 | 3 |
@@ -118,7 +120,7 @@ import torch
 from PIL import Image
 from transformers import AutoModelForImageTextToText, AutoProcessor
-repo = "webbrain-one/minicpmv47-webbrain-v2"
 processor = AutoProcessor.from_pretrained(repo)
 model = AutoModelForImageTextToText.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda").eval()
@@ -142,3 +144,5 @@ example the WebLINX-derived Compass subset), so the fine-tuned weights are relea
 only** under the WebBrain noncommercial research restrictions. Private hosting does not expand those rights.
 Model output is never proof that a browser action succeeded. Keep WebBrain's schema validation, grounding checks and
 confirmations in front of any action.

 - tool-calling
 - gui-grounding
 - decision-model
+- webbrain-compass
 ---
+# WebBrain Compass VLD Tiny v5
+`openbmb/MiniCPM-V-4.7-1B` fine-tuned in two rounds as one on-device model for WebBrain: Compass text and tool
+calls, VL (the visual fallback) and D (decisions). It covers three jobs:
 - **page-state decisions:** 120/126 on d1 with 0 false completions, 0.94 s per request in the browser
 - **the local visual fallback:** better grounding than LFM2.5-VL-450M on WebBrain's VL-3 validation
 This repository holds the **merged float32 weights**. Companion repositories:
+- [`webbrain-one/webbrain-compass-vld-tiny-v5-lora`](https://huggingface.co/webbrain-one/webbrain-compass-vld-tiny-v5-lora): the LoRA adapter and the merger weights.
+- [`webbrain-one/webbrain-compass-vld-tiny-v5-onnx`](https://huggingface.co/webbrain-one/webbrain-compass-vld-tiny-v5-onnx): the WebGPU ONNX bundle.
+Round 1 is kept as `webbrain-one/minicpmv47-webbrain-v1`.
 ## Training
 **Base:** `openbmb/MiniCPM-V-4.7-1B` (revision `379f120c2ff3588e6e06559ec6b9ec15cc05114e`): a SigLIP vision tower, a
 16x vision-to-language merger, and a Qwen3.5-0.8B-style language model (18 Gated DeltaNet + 6 attention layers).
+1. **Round 1** (`minicpmv47-webbrain-v1`): one pass over 344,139 samples: WebBrain-VL-4 (335,572 draws of VL-2/curated VL-3 observations,
    MolmoPoint GUI grounding, WebChain trajectories) plus the WebBrain Compass v3 maximal text release (8,583 rows).
    - Trainable: LoRA r = 64, alpha = 64 on all 186 language-model linear layers (43.3M parameters, fp32) and the
      full merger (26.0M, fp32). The vision tower is frozen.
    - AdamW with LoRA LR 1e-4 and merger LR 2e-5; 5,144 steps.
+2. **Round 2 (this model):** continues the round 1 adapter and merger with a fresh optimizer (LoRA LR 5e-5, merger LR 1e-5;
    3% warmup, cosine to 10%; effective batch 64; 784 steps; 51,811 samples). The data:
    - **Decision data:** the d1 balanced-V3 **train** split rendered in WebBrain's decision protocol, repeated 6x.
      It has 1,206 samples (918 verify Yes/No continuations and 288 option readouts), token-exact with the extension's
 The d1 validation split was used only for the eval loss and probe selection. The d1 final test split never fed
 training or selection.
+Final validation loss (full sets, round 1 → round 2): VL 0.295 → 0.288, Compass 1.138 → 1.116. On d1 validation decision
 samples the loss fell from 4.00 (verify) / 2.32 (readout) at the start of round 2 (first 96 samples) to
 0.066 / 0.067 at the end (all 594).
 | Model | Correct | Completions found | False completions | Unknown recognized | Time per request |
 |---|---:|---:|---:|---:|---:|
+| **v5, WebGPU q8 + probe** | **120/126** | **24/24** | **0/48** | **24/24** | 0.94 s |
+| **v5, WebGPU q4 (lite) + probe** | **120/126** | 24/24 | 0/48 | 24/24 | 0.94 s |
+| Round 1, WebGPU q8 + probe | 103/126 | 22/24 | 11/48 | 13/24 | 0.85 s |
 | MiniCPM-V-4.7-1B untuned, WebGPU q8 + probe | 104/126 | 23/24 | 8/48 | 16/24 | n/a |
 | Bonsai 2 2B Vision + probe | 110/126 | 21/24 | 4/48 | 20/24 | 2.2 s |
 | d1-omni-600M | 92/126 | n/a | 1/48 | n/a | 0.75 s |
+- The probe is no longer needed. Without it, v5's raw Yes − No verify margins score 121/126 in PyTorch and 120/126
   in WebGPU q8. In PyTorch the standard and conservative probes (fit on d1 train, chosen on validation) also score
   121/126, and the generated option readout alone scores 120/126.
+- A caution on scope: d1 is a synthetic benchmark. v5 was fine-tuned on d1's train families, which are disjoint from
   the test families but follow the same design. Bonsai's backbone was not trained on d1, only its probe was.
+  Validate v5 on real extension decisions before relying on these margins.
 ### Visual fallback: WebBrain VL-3 frozen validation, same 1,022 rows
 actionable inventory (first 20), the first 80 curated VL-3 rows and every 20th VL-2 row. LFM2.5-VL-450M (VL-3,
 step 3750) is WebBrain's current visual fallback.
+| Metric | v5 | Round 1 | LFM2.5-VL-450M |
 |---|---:|---:|---:|
 | Schema-valid outputs | **96.2%** | 95.8% | 92.7% |
 | Point within 0.05 (510 rows) | **65.5%** | 58.8% | 45.9% |
 | Six-section observation valid, VL-2 / VL-3 | 98.7% / **72.5%** | 99.4% / 65.0% | 97.5% / 38.8% |
 | Actionable inventory valid (first 20) | 25% | 30% | 10% |
+On intent → point/box the mean point distance is still worse than LFM's (0.190 vs 0.132; round 1 0.248), although more
 answers land within 0.05: a minority of far misses. Long inventory lists often do not terminate, for every model.
 ### Text and tool calls: WebBrain `test/llm` routing benchmark (compact tier)
 | Model | First-turn structured calls | Strict exact action | Loose tool family | Discouraged actions |
 |---|---:|---:|---:|---:|
+| **v5** (0.8B language model) | 95/100 | 9/89 | **48/89** | 3 |
+| Round 1 | 97/100 | 10/89 | 38/89 | 5 |
 | MiniCPM-V-4.7-1B, untuned | 98/100 | 3/89 | 30/89 | 2 |
 | Compass Tiny v2.1 (2.6B, WebGPU q4f16) | 98/100 | **15/89** | 43/89 | 3 |
 from PIL import Image
 from transformers import AutoModelForImageTextToText, AutoProcessor
+repo = "webbrain-one/webbrain-compass-vld-tiny-v5"
 processor = AutoProcessor.from_pretrained(repo)
 model = AutoModelForImageTextToText.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda").eval()
 only** under the WebBrain noncommercial research restrictions. Private hosting does not expand those rights.
 Model output is never proof that a browser action succeeded. Keep WebBrain's schema validation, grounding checks and
 confirmations in front of any action.
+Training data includes WebChain v2 (`webagentlab/webchain`, WebChain Academic Research License v1.0): Fan, Sicheng, Rui Wan, Yifei Leng, Gaoning Liang, Li Ling, Yanyi Shang, and Dehan Kong. "WebChain: A Large-Scale Human-Annotated Dataset of Real-World Web Interaction Traces." arXiv:2603.05295, 2026. It also includes WebLINX (McGill-NLP, CC BY-NC-SA 4.0) and MolmoPoint-GUISyn (Ai2, ODC-BY).