Model card

README.md

@@ -0,0 +1,128 @@
+---
+license: other
+license_name: webbrain-noncommercial-research-restrictions
+base_model: openbmb/MiniCPM-V-4.7-1B
+pipeline_tag: image-text-to-text
+library_name: transformers
+tags:
+- webbrain
+- minicpm-v
+- browser-agent
+- tool-calling
+- gui-grounding
+---
+
+# MiniCPM-V-4.7 WebBrain v1
+
+A WebBrain fine-tune of MiniCPM-V-4.7-1B, built to be one on-device model for three jobs in WebBrain:
+
+- page-state decisions (verify protocol plus probe)
+- the local visual fallback (screenshot observations and grounding)
+- the text and tool-calling model for offline mode
+
+This repository holds the **merged float32 weights**: the LoRA folded into the base, with the fine-tuned merger
+included. Companion repositories:
+
+- [`webbrain-one/minicpmv47-webbrain-v1-lora`](https://huggingface.co/webbrain-one/minicpmv47-webbrain-v1-lora): the LoRA adapter and the merger weights.
+- [`webbrain-one/minicpmv47-webbrain-v1-onnx`](https://huggingface.co/webbrain-one/minicpmv47-webbrain-v1-onnx): the WebGPU ONNX bundle (q8 text graphs, fp16 vision encoder, JS runtime and decision probes).
+
+## Training
+
+- **Base:** `openbmb/MiniCPM-V-4.7-1B` (revision `379f120c2ff3588e6e06559ec6b9ec15cc05114e`): SigLIP vision tower, a
+  16x vision-to-language merger, and a Qwen3.5-0.8B-style language model (18 Gated DeltaNet + 6 attention layers).
+- **Data:** one pass over 344,139 samples.
+  - WebBrain-VL-4 (the VL-3 draw schedule, 335,572 draws): VL-2 six-section page observations, curated VL-3
+    observations, MolmoPoint GUI grounding tasks (intent → point/box, point/box → element, grounded interaction,
+    actionable inventory) and WebChain trajectory steps.
+  - The WebBrain Compass v3 maximal text release (8,583 rows): ask, compact browser actions, WebLINX-derived compact
+    actions, escalation.
+- **Method:** LoRA r = 64, alpha = 64 on all 186 language-model linear layers (43.3M parameters, fp32 adapters) plus a
+  full fine-tune of the merger (26.0M, fp32). The vision tower is frozen.
+  - AdamW: LoRA 1e-4, merger 2e-5. 3% warmup, cosine decay to 10%, effective batch 64 (5,144 optimizer steps).
+  - Loss is the per-sample mean over completion tokens only, with a 32K maximum sequence length.
+  - Thinking is off: the model's chat template with `enable_thinking=False`.
+- **Compute:** one RTX 5090, about 15 hours.
+- **Final validation loss (full sets):** VL 0.295 (VL-2 0.216, curated VL-3 0.542, MolmoPoint 0.498); Compass 1.138.
+- **Held out:** d1's balanced-V3 final test split was never used for training or model selection.
+
+## Results
+
+### Visual fallback: WebBrain VL-3 frozen validation, same 1,344 rows
+
+Greedy generation, `max_new_tokens = max(128, expected + 96)`, scored with the VL-3 pipeline's own scorer
+(`run_webbrain_vl3_gpu_pipeline._score_generation`). The comparison is WebBrain's current visual fallback,
+LFM2.5-VL-450M fine-tuned on VL-3 (selected checkpoint, step 3750), on exactly the same rows: every MolmoPoint
+grounding row except actionable inventory (first 100 of 250), every curated VL-3 row, every 10th VL-2 row.
+
+| Metric | MiniCPM-V WebBrain v1 | LFM2.5-VL-450M (VL-3) |
+|---|---:|---:|
+| Schema-valid outputs | **90.1%** | 85.5% |
+| Point within 0.05 (510 rows) | **58.8%** | 45.9% |
+| Grounded interaction: point within 0.05 (255) | **51.0%** | 29.8% |
+| Grounded interaction: mean point distance | **0.139** | 0.189 |
+| Box IoU | **0.113** | 0.050 |
+| Point inside the target box | **23.3%** | 10.4% |
+| Six-section observation valid, VL-2 / VL-3 | **99.7% / 69.3%** | 97.1% / 44.6% |
+| Actionable inventory valid (100) | **19%** | 10% |
+
+On intent → point/box the mean point distance is worse (0.248 vs 0.132) although accuracy within 0.05 is higher
+(66.7% vs 62.1%): a minority of far misses. Long inventory lists often do not terminate for either model.
+
+### Text and tool calls: WebBrain `test/llm` routing benchmark (compact tier)
+
+Same suite and grader as the Compass Tiny cards (`run-llamacpp.mjs` first turns, `run-scenarios.mjs` scenarios),
+served from PyTorch BF16 with seed 3407 and thinking off. The 89 scenarios are the scored ones shared with the
+Compass Tiny v2.1 run; strict = `ideal`, loose = `ideal` + `ideal_name`.
+
+| Model | First-turn structured calls | Strict exact action | Loose tool family | Discouraged actions |
+|---|---:|---:|---:|---:|
+| **MiniCPM-V WebBrain v1** (0.8B language model) | 97/100 | 10/89 | 38/89 | 5 |
+| MiniCPM-V-4.7-1B, untuned | 98/100 | 3/89 | 30/89 | 2 |
+| Compass Tiny v2.1 (2.6B, WebGPU q4f16) | 98/100 | 15/89 | 43/89 | 3 |
+| Compass Tiny XS v3 (1.7B, BF16, from its card) | 97/100 | 9/89 | 39/89 | n/a |
+
+### Page-state decisions: d1 balanced-V3 final test (126 categorical questions)
+
+WebBrain's verify protocol (one Yes/No continuation per option on a shared prefill) plus a linear probe on the
+margin and `decision_hidden`, fit on d1's train split with its strength chosen on validation; the final test is
+never trained on. The conservative probe weighs a false "completed" four times in its fit.
+
+| Model | Correct | Completions found | False completions | Unknown recognized | Time per request |
+|---|---:|---:|---:|---:|---:|
+| v1, WebGPU q8, probe | 103/126 | 22/24 | 11/48 | 13/24 | 0.85 s (RTX 5090) |
+| v1, WebGPU q8, conservative probe | 101/126 | 14/24 | **4/48** | 20/24 | 0.85 s |
+| MiniCPM-V-4.7-1B untuned, WebGPU q8, probe | 104/126 | 23/24 | 8/48 | 16/24 | n/a |
+| Bonsai 2 2B Vision + probe (reference) | 110/126 | 21/24 | 4/48 | 20/24 | 2.2 s |
+| d1-omni-600M (reference) | 92/126 | n/a | 1/48 | n/a | 0.75 s |
+
+The fine-tune did not improve decisions: its data has no decision-format examples.
+
+## Usage
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoModelForImageTextToText, AutoProcessor
+
+repo = "webbrain-one/minicpmv47-webbrain-v1"
+processor = AutoProcessor.from_pretrained(repo)
+model = AutoModelForImageTextToText.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda").eval()
+
+messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Describe this page."}]}]
+prompt = processor.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, tokenize=False)
+inputs = processor(text=[prompt], images=[Image.open("screenshot.png").convert("RGB")], return_tensors="pt").to(model.device)
+out = model.generate(**inputs, do_sample=False, max_new_tokens=512)
+print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
+```
+
+Tool calls use the chat template's native format: `<tool_call>`, then `<function=NAME>`, then one `<parameter=KEY>`
+block per argument. Pass OpenAI-style `tools=` to `apply_chat_template`, and parse the reply with a Qwen3.5 /
+Qwen3-coder style tool parser that types each argument from the tool's JSON schema.
+
+## License
+
+The base model is Apache-2.0. This fine-tune was trained on data that includes noncommercial-licensed sources (for
+example the WebLINX-derived Compass subset), so the fine-tuned weights are released for **noncommercial research
+only** under the WebBrain noncommercial research restrictions. Private hosting does not expand those rights.
+Model output is never proof that a browser action succeeded. Keep WebBrain's schema validation, grounding checks and
+confirmations in front of any action.