Model card

README.md

@@ -0,0 +1,144 @@
+---
+license: other
+license_name: webbrain-noncommercial-research-restrictions
+base_model: openbmb/MiniCPM-V-4.7-1B
+pipeline_tag: image-text-to-text
+library_name: transformers
+tags:
+- webbrain
+- minicpm-v
+- browser-agent
+- tool-calling
+- gui-grounding
+- decision-model
+---
+
+# MiniCPM-V-4.7 WebBrain v2
+
+MiniCPM-V-4.7-1B fine-tuned in two rounds as one on-device model for WebBrain. It covers three jobs:
+
+- **page-state decisions:** 120/126 on d1 with 0 false completions, 0.94 s per request in the browser
+- **the local visual fallback:** better grounding than LFM2.5-VL-450M on WebBrain's VL-3 validation
+- **offline text and tool calls:** close to Compass Tiny v2.1 on the routing benchmark
+
+This repository holds the **merged float32 weights**. Companion repositories:
+
+- [`webbrain-one/minicpmv47-webbrain-v2-lora`](https://huggingface.co/webbrain-one/minicpmv47-webbrain-v2-lora): the LoRA adapter and the merger weights.
+- [`webbrain-one/minicpmv47-webbrain-v2-onnx`](https://huggingface.co/webbrain-one/minicpmv47-webbrain-v2-onnx): the WebGPU ONNX bundle.
+
+The previous round is kept as `webbrain-one/minicpmv47-webbrain-v1`.
+
+## Training
+
+Two rounds on one RTX 5090 (about 20 hours in total).
+**Base:** `openbmb/MiniCPM-V-4.7-1B` (revision `379f120c2ff3588e6e06559ec6b9ec15cc05114e`): a SigLIP vision tower, a
+16x vision-to-language merger, and a Qwen3.5-0.8B-style language model (18 Gated DeltaNet + 6 attention layers).
+
+1. **Round 1 (v1):** one pass over 344,139 samples: WebBrain-VL-4 (335,572 draws of VL-2/curated VL-3 observations,
+   MolmoPoint GUI grounding, WebChain trajectories) plus the WebBrain Compass v3 maximal text release (8,583 rows).
+   - Trainable: LoRA r = 64, alpha = 64 on all 186 language-model linear layers (43.3M parameters, fp32) and the
+     full merger (26.0M, fp32). The vision tower is frozen.
+   - AdamW with LoRA LR 1e-4 and merger LR 2e-5; 5,144 steps.
+2. **Round 2 (v2, this model):** continues the v1 adapter and merger with a fresh optimizer (LoRA LR 5e-5, merger LR 1e-5;
+   3% warmup, cosine to 10%; effective batch 64; 784 steps; 51,811 samples). The data:
+   - **Decision data:** the d1 balanced-V3 **train** split rendered in WebBrain's decision protocol, repeated 6x.
+     It has 1,206 samples (918 verify Yes/No continuations and 288 option readouts), token-exact with the extension's
+     prompts.
+   - **Text data:** the Compass v3 release once more (17% of samples, up from 2.5%).
+   - **Replay data:** a 36,000-draw VL-4 replay (WebChain 12k, VL-2 12k, curated VL-3 3k, MolmoPoint 9k).
+
+Loss covers completion tokens only (per-sample mean), with thinking off and a 32K maximum length.
+The d1 validation split was used only for the eval loss and probe selection. The d1 final test split never fed
+training or selection.
+
+Final validation loss (full sets, v1 → v2): VL 0.295 → 0.288, Compass 1.138 → 1.116. On d1 validation decision
+samples the loss fell from 4.00 (verify) / 2.32 (readout) at the start of round 2 (first 96 samples) to
+0.066 / 0.067 at the end (all 594).
+
+## Results
+
+### Page-state decisions: d1 balanced-V3 final test (126 categorical questions)
+
+Scored through WebBrain's decision protocol in the browser (WebGPU) on an RTX 5090. One request is a screenshot,
+its page state and its questions. The d1 final test split was never trained on.
+
+| Model | Correct | Completions found | False completions | Unknown recognized | Time per request |
+|---|---:|---:|---:|---:|---:|
+| **v2, WebGPU q8 + probe** | **120/126** | **24/24** | **0/48** | **24/24** | 0.94 s |
+| **v2, WebGPU q4 (lite) + probe** | **120/126** | 24/24 | 0/48 | 24/24 | 0.94 s |
+| v1, WebGPU q8 + probe | 103/126 | 22/24 | 11/48 | 13/24 | 0.85 s |
+| MiniCPM-V-4.7-1B untuned, WebGPU q8 + probe | 104/126 | 23/24 | 8/48 | 16/24 | n/a |
+| Bonsai 2 2B Vision + probe | 110/126 | 21/24 | 4/48 | 20/24 | 2.2 s |
+| d1-omni-600M | 92/126 | n/a | 1/48 | n/a | 0.75 s |
+
+- The probe is no longer needed. Without it, v2's raw Yes − No verify margins score 121/126 in PyTorch and 120/126
+  in WebGPU q8. In PyTorch the standard and conservative probes (fit on d1 train, chosen on validation) also score
+  121/126, and the generated option readout alone scores 120/126.
+- A caution on scope: d1 is a synthetic benchmark. v2 was fine-tuned on d1's train families, which are disjoint from
+  the test families but follow the same design. Bonsai's backbone was not trained on d1, only its probe was.
+  Validate v2 on real extension decisions before relying on these margins.
+
+### Visual fallback: WebBrain VL-3 frozen validation, same 1,022 rows
+
+Greedy generation, scored with the VL-3 pipeline's own scorer. The rows are every MolmoPoint grounding row except
+actionable inventory (first 20), the first 80 curated VL-3 rows and every 20th VL-2 row. LFM2.5-VL-450M (VL-3,
+step 3750) is WebBrain's current visual fallback.
+
+| Metric | v2 | v1 | LFM2.5-VL-450M |
+|---|---:|---:|---:|
+| Schema-valid outputs | **96.2%** | 95.8% | 92.7% |
+| Point within 0.05 (510 rows) | **65.5%** | 58.8% | 45.9% |
+| Mean point distance | **0.156** | 0.194 | 0.161 |
+| Grounded interaction: point within 0.05 (255) | **57.3%** | 51.0% | 29.8% |
+| Intent → point/box: point within 0.05 (255) | **73.7%** | 66.7% | 62.1% |
+| Box IoU | **0.175** | 0.113 | 0.050 |
+| Point inside the target box | **33.3%** | 23.3% | 10.4% |
+| Six-section observation valid, VL-2 / VL-3 | 98.7% / **72.5%** | 99.4% / 65.0% | 97.5% / 38.8% |
+| Actionable inventory valid (first 20) | 25% | 30% | 10% |
+
+On intent → point/box the mean point distance is still worse than LFM's (0.190 vs 0.132; v1 0.248), although more
+answers land within 0.05: a minority of far misses. Long inventory lists often do not terminate, for every model.
+
+### Text and tool calls: WebBrain `test/llm` routing benchmark (compact tier)
+
+Served from PyTorch BF16 with seed 3407 and thinking off. The table counts the 89 scenarios scored in the Compass
+Tiny v2.1 run: strict = `ideal`, loose = `ideal` + `ideal_name`.
+
+| Model | First-turn structured calls | Strict exact action | Loose tool family | Discouraged actions |
+|---|---:|---:|---:|---:|
+| **v2** (0.8B language model) | 95/100 | 9/89 | **48/89** | 3 |
+| v1 | 97/100 | 10/89 | 38/89 | 5 |
+| MiniCPM-V-4.7-1B, untuned | 98/100 | 3/89 | 30/89 | 2 |
+| Compass Tiny v2.1 (2.6B, WebGPU q4f16) | 98/100 | **15/89** | 43/89 | 3 |
+
+## Usage
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoModelForImageTextToText, AutoProcessor
+
+repo = "webbrain-one/minicpmv47-webbrain-v2"
+processor = AutoProcessor.from_pretrained(repo)
+model = AutoModelForImageTextToText.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda").eval()
+
+messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Describe this page."}]}]
+prompt = processor.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, tokenize=False)
+inputs = processor(text=[prompt], images=[Image.open("screenshot.png").convert("RGB")], return_tensors="pt").to(model.device)
+out = model.generate(**inputs, do_sample=False, max_new_tokens=512)
+print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
+```
+
+Tool calls use the chat template's native format: `<tool_call>`, then `<function=NAME>`, then one `<parameter=KEY>`
+block per argument. Pass OpenAI-style `tools=` to `apply_chat_template`, and parse the reply with a Qwen3.5 /
+Qwen3-coder style tool parser that types each argument from the tool's JSON schema.
+
+For decisions, use WebBrain's verify protocol: one "Proposed answer: X ... Is the proposed answer correct? Answer Yes or No." continuation per option, compared by the Yes − No logit. See `minicpmv-decisions.mjs` in the ONNX repository.
+
+## License
+
+The base model is Apache-2.0. This fine-tune was trained on data that includes noncommercial-licensed sources (for
+example the WebLINX-derived Compass subset), so the fine-tuned weights are released for **noncommercial research
+only** under the WebBrain noncommercial research restrictions. Private hosting does not expand those rights.
+Model output is never proof that a browser action succeeded. Keep WebBrain's schema validation, grounding checks and
+confirmations in front of any action.