webbrain-one/webbrain-compass-vld-tiny-v5

WebBrain Compass VLD Tiny v5

openbmb/MiniCPM-V-4.7-1B fine-tuned in two rounds as one on-device model for WebBrain: Compass text and tool calls, VL (the visual fallback) and D (decisions). It covers three jobs:

• page-state decisions: 120/126 on d1 with 0 false completions, 0.94 s per request in the browser
• the local visual fallback: better grounding than LFM2.5-VL-450M on WebBrain's VL-3 validation
• offline text and tool calls: close to Compass Tiny v2.1 on the routing benchmark

This repository holds the merged float32 weights. Companion repositories:

• webbrain-one/webbrain-compass-vld-tiny-v5-lora: the LoRA adapter and the merger weights.
• webbrain-one/webbrain-compass-vld-tiny-v5-onnx: the WebGPU ONNX bundle.

Round 1 is kept as webbrain-one/minicpmv47-webbrain-v1.

Training

Two rounds on one RTX 5090 (about 20 hours in total). Base: openbmb/MiniCPM-V-4.7-1B (revision 379f120c2ff3588e6e06559ec6b9ec15cc05114e): a SigLIP vision tower, a 16x vision-to-language merger, and a Qwen3.5-0.8B-style language model (18 Gated DeltaNet + 6 attention layers).

1. Round 1 (minicpmv47-webbrain-v1): one pass over 344,139 samples: WebBrain-VL-4 (335,572 draws of VL-2/curated VL-3 observations, MolmoPoint GUI grounding, WebChain trajectories) plus the WebBrain Compass v3 maximal text release (8,583 rows).
   • Trainable: LoRA r = 64, alpha = 64 on all 186 language-model linear layers (43.3M parameters, fp32) and the full merger (26.0M, fp32). The vision tower is frozen.
   • AdamW with LoRA LR 1e-4 and merger LR 2e-5; 5,144 steps.
2. Round 2 (this model): continues the round 1 adapter and merger with a fresh optimizer (LoRA LR 5e-5, merger LR 1e-5; 3% warmup, cosine to 10%; effective batch 64; 784 steps; 51,811 samples). The data:
   • Decision data: the d1 balanced-V3 train split rendered in WebBrain's decision protocol, repeated 6x. It has 1,206 samples (918 verify Yes/No continuations and 288 option readouts), token-exact with the extension's prompts.
   • Text data: the Compass v3 release once more (17% of samples, up from 2.5%).
   • Replay data: a 36,000-draw VL-4 replay (WebChain 12k, VL-2 12k, curated VL-3 3k, MolmoPoint 9k).

Loss covers completion tokens only (per-sample mean), with thinking off and a 32K maximum length. The d1 validation split was used only for the eval loss and probe selection. The d1 final test split never fed training or selection.

Final validation loss (full sets, round 1 → round 2): VL 0.295 → 0.288, Compass 1.138 → 1.116. On d1 validation decision samples the loss fell from 4.00 (verify) / 2.32 (readout) at the start of round 2 (first 96 samples) to 0.066 / 0.067 at the end (all 594).

Results

Page-state decisions: d1 balanced-V3 final test (126 categorical questions)

Scored through WebBrain's decision protocol in the browser (WebGPU) on an RTX 5090. One request is a screenshot, its page state and its questions. The d1 final test split was never trained on.

Model	Correct	Completions found	False completions	Unknown recognized	Time per request
v5, WebGPU q8 + probe	120/126	24/24	0/48	24/24	0.94 s
v5, WebGPU q4 (lite) + probe	120/126	24/24	0/48	24/24	0.94 s
Round 1, WebGPU q8 + probe	103/126	22/24	11/48	13/24	0.85 s
MiniCPM-V-4.7-1B untuned, WebGPU q8 + probe	104/126	23/24	8/48	16/24	n/a
Bonsai 2 2B Vision + probe	110/126	21/24	4/48	20/24	2.2 s
d1-omni-600M	92/126	n/a	1/48	n/a	0.75 s

• The probe is no longer needed. Without it, v5's raw Yes − No verify margins score 121/126 in PyTorch and 120/126 in WebGPU q8. In PyTorch the standard and conservative probes (fit on d1 train, chosen on validation) also score 121/126, and the generated option readout alone scores 120/126.
• A caution on scope: d1 is a synthetic benchmark. v5 was fine-tuned on d1's train families, which are disjoint from the test families but follow the same design. Bonsai's backbone was not trained on d1, only its probe was. Validate v5 on real extension decisions before relying on these margins.

Visual fallback: WebBrain VL-3 frozen validation, same 1,022 rows

Greedy generation, scored with the VL-3 pipeline's own scorer. The rows are every MolmoPoint grounding row except actionable inventory (first 20), the first 80 curated VL-3 rows and every 20th VL-2 row. LFM2.5-VL-450M (VL-3, step 3750) is WebBrain's current visual fallback.

Metric	v5	Round 1	LFM2.5-VL-450M
Schema-valid outputs	96.2%	95.8%	92.7%
Point within 0.05 (510 rows)	65.5%	58.8%	45.9%
Mean point distance	0.156	0.194	0.161
Grounded interaction: point within 0.05 (255)	57.3%	51.0%	29.8%
Intent → point/box: point within 0.05 (255)	73.7%	66.7%	62.1%
Box IoU	0.175	0.113	0.050
Point inside the target box	33.3%	23.3%	10.4%
Six-section observation valid, VL-2 / VL-3	98.7% / 72.5%	99.4% / 65.0%	97.5% / 38.8%
Actionable inventory valid (first 20)	25%	30%	10%

On intent → point/box the mean point distance is still worse than LFM's (0.190 vs 0.132; round 1 0.248), although more answers land within 0.05: a minority of far misses. Long inventory lists often do not terminate, for every model.

Text and tool calls: WebBrain test/llm routing benchmark (compact tier)

Served from PyTorch BF16 with seed 3407 and thinking off. The table counts the 89 scenarios scored in the Compass Tiny v2.1 run: strict = ideal, loose = ideal + ideal_name.

Model	First-turn structured calls	Strict exact action	Loose tool family	Discouraged actions
v5 (0.8B language model)	95/100	9/89	48/89	3
Round 1	97/100	10/89	38/89	5
MiniCPM-V-4.7-1B, untuned	98/100	3/89	30/89	2
Compass Tiny v2.1 (2.6B, WebGPU q4f16)	98/100	15/89	43/89	3

Usage

import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

repo = "webbrain-one/webbrain-compass-vld-tiny-v5"
processor = AutoProcessor.from_pretrained(repo)
model = AutoModelForImageTextToText.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda").eval()

messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Describe this page."}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, tokenize=False)
inputs = processor(text=[prompt], images=[Image.open("screenshot.png").convert("RGB")], return_tensors="pt").to(model.device)
out = model.generate(**inputs, do_sample=False, max_new_tokens=512)
print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Tool calls use the chat template's native format: <tool_call>, then <function=NAME>, then one <parameter=KEY> block per argument. Pass OpenAI-style tools= to apply_chat_template, and parse the reply with a Qwen3.5 / Qwen3-coder style tool parser that types each argument from the tool's JSON schema.

For decisions, use WebBrain's verify protocol: one "Proposed answer: X ... Is the proposed answer correct? Answer Yes or No." continuation per option, compared by the Yes − No logit. See minicpmv-decisions.mjs in the ONNX repository.

License

The base model is Apache-2.0. This fine-tune was trained on data that includes noncommercial-licensed sources (for example the WebLINX-derived Compass subset), so the fine-tuned weights are released for noncommercial research only under the WebBrain noncommercial research restrictions. Private hosting does not expand those rights. Model output is never proof that a browser action succeeded. Keep WebBrain's schema validation, grounding checks and confirmations in front of any action.

Training data includes WebChain v2 (webagentlab/webchain, WebChain Academic Research License v1.0): Fan, Sicheng, Rui Wan, Yifei Leng, Gaoning Liang, Li Ling, Yanyi Shang, and Dehan Kong. "WebChain: A Large-Scale Human-Annotated Dataset of Real-World Web Interaction Traces." arXiv:2603.05295, 2026. It also includes WebLINX (McGill-NLP, CC BY-NC-SA 4.0) and MolmoPoint-GUISyn (Ai2, ODC-BY).