Rename to WebBrain Compass VLD Tiny v5; attribution

1938a91 verified 8 days ago

10.1 kB

---
license: other
license_name: webbrain-noncommercial-research-restrictions
base_model: webbrain-one/webbrain-compass-vld-tiny-v5
pipeline_tag: image-text-to-text
library_name: onnxruntime
tags:
- webbrain
- minicpm-v
- onnx
- webgpu
- browser-agent
---
# WebBrain Compass VLD Tiny v5: WebGPU ONNX

This is the in-browser build of
[`webbrain-one/webbrain-compass-vld-tiny-v5`](https://huggingface.co/webbrain-one/webbrain-compass-vld-tiny-v5). One model
covers WebBrain's page-state decisions, the local visual fallback and offline text and tool calls, without loading
or unloading models between them. On d1's final test it answers 120 of 126 decision questions with no false
completions, in 0.94 s per request on an RTX 5090.

## Files

| File | Size | Role |
|---|---:|---|
| `onnx/vision_encoder_fp16.onnx` (+`_data`) | 1.10 GB | SigLIP and the fine-tuned merger. Input `pixel_values float32[num_crops,3,14h,14w]`; output `image_features float32[num_crops,(h/4)(w/4),1024]` |
| `onnx/decoder_model_merged_q8f16.onnx` (+`_data`) | 0.83 GB | Recommended language model: MatMulNBits 8-bit (block 32), fp16 activations and KV cache. Chunked Gated DeltaNet prefill. Outputs last-position `logits` and `decision_hidden` |
| `onnx/embed_tokens_q8f16.onnx` (+`_data`) | 0.28 GB | Token embeddings, 8-bit |
| `onnx/decoder_model_merged_q4f16.onnx` (+`_data`) | 0.44 GB | Lite: the same language model with 4-bit weights |
| `onnx/embed_tokens_q4f16.onnx` (+`_data`) | 0.15 GB | Lite: token embeddings, 4-bit |
| `runtime/minicpmv-runtime.mjs` | | Processor port (PIL bicubic, MiniCPM-V slicing, image placeholders), canvas M-RoPE positions, sessions, prefill/forward with cache reuse, greedy `generate` |
| `runtime/minicpmv-decisions.mjs` | | WebBrain's decision protocol (drop-in next to `src/chrome/src/offscreen/webgpu-decisions.js`) |
| `runtime/minicpmv-decision-probe-ft2.js`, `...-ft2-cs.js` | | Optional decision probes: standard, and conservative on false completions |
| `runtime/webgpu-check.mjs`, `runtime/decision-check.mjs` | | Parity and end-to-end decision checks |
| `validation/` | | Export report, training eval, decision, VL and routing results |

`config.json` names the graph sets under `webbrain_runtime`: `recommended` (q8) and `lite` (q4). The download is
about 2.2 GB with the q8 graphs, or 1.7 GB with the lite graphs.

- **Lite (q4) graphs:** Round 1's 4-bit build failed the quality bar (80/126 decisions, 25/48 false completions). This
  build's 4-bit graphs match q8 on d1 decisions: 120/126, 0/48 false completions. Text and visual generation were not
  validated on q4, so prefer q8 when the model generates.
- **Tested with:** Transformers.js 4.2.0 (tokenizer), onnxruntime-web 1.27.0 (WebGPU EP), Chromium, an RTX 5090.
- **Low-end GPUs are not a target:** on a Quadro T400 (4 GB), one decision request takes about 58 s, 45 s of it in
  the vision encoder.

## Usage

```javascript
import * as ort from './ort.webgpu.mjs';
import { AutoTokenizer } from './transformers.web.js';
import { MINICPMV, MiniCPMVRuntime, preprocessImage } from './runtime/minicpmv-runtime.mjs';
import { runMiniCPMVDecision } from './runtime/minicpmv-decisions.mjs';
const tokenizer = await AutoTokenizer.from_pretrained(BUNDLE_URL);
const tok = { encode: text => tokenizer.encode(text, { add_special_tokens: false }) };
const rt = await MiniCPMVRuntime.create(ort, tok, {      // sessions are created one at a time
  embed: `${BUNDLE_URL}/onnx/embed_tokens_q8f16.onnx`,      // lite: embed_tokens_q4f16.onnx
  vision: `${BUNDLE_URL}/onnx/vision_encoder_fp16.onnx`,
  decoder: `${BUNDLE_URL}/onnx/decoder_model_merged_q8f16.onnx`,   // lite: decoder_model_merged_q4f16.onnx
});
// decisions: images = [{ rgb: Uint8Array, width, height }], entries = [[id, d1-style question], ...]
// Without a probe each option is scored by its raw Yes − No margin. To use a probe, import minicpmvProbeScores
// from './runtime/minicpmv-decision-probe-ft2.js' and pass probe: { scores: minicpmvProbeScores }.
const { answers } = await runMiniCPMVDecision(rt, { state, images, entries });
// text or visual generation (greedy): a rendered chat prompt, one image placeholder per preprocessed image
const prompt = `<|im_start|>user\n${MINICPMV.imagePad}\nDescribe this page.<|im_end|>\n`
  + '<|im_start|>assistant\n<think>\n\n</think>\n\n';
const { ids } = await rt.generate(prompt, [preprocessImage(rgb, width, height)], { maxNewTokens: 256 });
const text = tokenizer.decode(ids, { skip_special_tokens: true });
```

## Validation

These checks use the d1 balanced-V3 final test and WebBrain's own prompt helpers.

- **Processor:** token ids and canvas positions are exact against the PyTorch processor. Pixels differ by at most 2/255
  on about 0.01% of values.
- **WebGPU q8 vs PyTorch** (144 questions):
  - Argmax agreement is 98.6% (2 flips).
  - The median Yes − No margin difference is 0.38 (mean 0.76).
  - The median `decision_hidden` cosine is 0.999.
  - The tails are wider than round 1's (worst margin difference 1.06, minimum cosine 0.990). 12 of 144 questions differ by
    more than 2 logits. One two-option auxiliary question flips outright: margins +4.0 / −7.4 in WebGPU against
    −7.4 / +5.5 in PyTorch, hidden cosine 0.85.
  - End to end, the q8 build scores 120/126 without a probe, against 121/126 in PyTorch.
- **End-to-end decisions** (RTX 5090, one request = a screenshot plus its questions):
  - q8 with the standard probe: 120/126 correct, 24/24 completions found, 0/48 false completions, 24/24 unknown
    recognized. The conservative probe and the q4 lite graphs score the same.
  - Time: 0.94 s per request on average (vision 0.28 s, prefill 0.12 s, questions 0.54 s). Model load takes 9.1 s
    (7.8 s for lite).
- **Generation smoke test** (q8, greedy): a 31-token page description in 1.7 s.
## Results

### Page-state decisions: d1 balanced-V3 final test (126 categorical questions)

Scored through WebBrain's decision protocol in the browser (WebGPU) on an RTX 5090. One request is a screenshot,
its page state and its questions. The d1 final test split was never trained on.

| Model | Correct | Completions found | False completions | Unknown recognized | Time per request |
|---|---:|---:|---:|---:|---:|
| **v5, WebGPU q8 + probe** | **120/126** | **24/24** | **0/48** | **24/24** | 0.94 s |
| **v5, WebGPU q4 (lite) + probe** | **120/126** | 24/24 | 0/48 | 24/24 | 0.94 s |
| Round 1, WebGPU q8 + probe | 103/126 | 22/24 | 11/48 | 13/24 | 0.85 s |
| MiniCPM-V-4.7-1B untuned, WebGPU q8 + probe | 104/126 | 23/24 | 8/48 | 16/24 | n/a |
| Bonsai 2 2B Vision + probe | 110/126 | 21/24 | 4/48 | 20/24 | 2.2 s |
| d1-omni-600M | 92/126 | n/a | 1/48 | n/a | 0.75 s |

- The probe is no longer needed. Without it, v5's raw Yes − No verify margins score 121/126 in PyTorch and 120/126
  in WebGPU q8. In PyTorch the standard and conservative probes (fit on d1 train, chosen on validation) also score
  121/126, and the generated option readout alone scores 120/126.
- A caution on scope: d1 is a synthetic benchmark. v5 was fine-tuned on d1's train families, which are disjoint from
  the test families but follow the same design. Bonsai's backbone was not trained on d1, only its probe was.
  Validate v5 on real extension decisions before relying on these margins.

### Visual fallback: WebBrain VL-3 frozen validation, same 1,022 rows

Greedy generation, scored with the VL-3 pipeline's own scorer. The rows are every MolmoPoint grounding row except
actionable inventory (first 20), the first 80 curated VL-3 rows and every 20th VL-2 row. LFM2.5-VL-450M (VL-3,
step 3750) is WebBrain's current visual fallback.

| Metric | v5 | Round 1 | LFM2.5-VL-450M |
|---|---:|---:|---:|
| Schema-valid outputs | **96.2%** | 95.8% | 92.7% |
| Point within 0.05 (510 rows) | **65.5%** | 58.8% | 45.9% |
| Mean point distance | **0.156** | 0.194 | 0.161 |
| Grounded interaction: point within 0.05 (255) | **57.3%** | 51.0% | 29.8% |
| Intent → point/box: point within 0.05 (255) | **73.7%** | 66.7% | 62.1% |
| Box IoU | **0.175** | 0.113 | 0.050 |
| Point inside the target box | **33.3%** | 23.3% | 10.4% |
| Six-section observation valid, VL-2 / VL-3 | 98.7% / **72.5%** | 99.4% / 65.0% | 97.5% / 38.8% |
| Actionable inventory valid (first 20) | 25% | 30% | 10% |

On intent → point/box the mean point distance is still worse than LFM's (0.190 vs 0.132; round 1 0.248), although more
answers land within 0.05: a minority of far misses. Long inventory lists often do not terminate, for every model.

### Text and tool calls: WebBrain `test/llm` routing benchmark (compact tier)

Served from PyTorch BF16 with seed 3407 and thinking off. The table counts the 89 scenarios scored in the Compass
Tiny v2.1 run: strict = `ideal`, loose = `ideal` + `ideal_name`.

| Model | First-turn structured calls | Strict exact action | Loose tool family | Discouraged actions |
|---|---:|---:|---:|---:|
| **v5** (0.8B language model) | 95/100 | 9/89 | **48/89** | 3 |
| Round 1 | 97/100 | 10/89 | 38/89 | 5 |
| MiniCPM-V-4.7-1B, untuned | 98/100 | 3/89 | 30/89 | 2 |
| Compass Tiny v2.1 (2.6B, WebGPU q4f16) | 98/100 | **15/89** | 43/89 | 3 |

## License

The base model is Apache-2.0. This fine-tune was trained on data that includes noncommercial-licensed sources (for
example the WebLINX-derived Compass subset), so the fine-tuned weights are released for **noncommercial research
only** under the WebBrain noncommercial research restrictions. Private hosting does not expand those rights.
Model output is never proof that a browser action succeeded. Keep WebBrain's schema validation, grounding checks and
confirmations in front of any action.

Training data includes WebChain v2 (`webagentlab/webchain`, WebChain Academic Research License v1.0): Fan, Sicheng, Rui Wan, Yifei Leng, Gaoning Liang, Li Ling, Yanyi Shang, and Dehan Kong. "WebChain: A Large-Scale Human-Annotated Dataset of Real-World Web Interaction Traces." arXiv:2603.05295, 2026. It also includes WebLINX (McGill-NLP, CC BY-NC-SA 4.0) and MolmoPoint-GUISyn (Ai2, ODC-BY).

	---
	license: other
	license_name: webbrain-noncommercial-research-restrictions
	base_model: webbrain-one/webbrain-compass-vld-tiny-v5
	pipeline_tag: image-text-to-text
	library_name: onnxruntime
	tags:
	- webbrain
	- minicpm-v
	- onnx
	- webgpu
	- browser-agent
	---

	# WebBrain Compass VLD Tiny v5: WebGPU ONNX

	This is the in-browser build of
	[`webbrain-one/webbrain-compass-vld-tiny-v5`](https://huggingface.co/webbrain-one/webbrain-compass-vld-tiny-v5). One model
	covers WebBrain's page-state decisions, the local visual fallback and offline text and tool calls, without loading
	or unloading models between them. On d1's final test it answers 120 of 126 decision questions with no false
	completions, in 0.94 s per request on an RTX 5090.

	## Files

	\| File \| Size \| Role \|
	\|---\|---:\|---\|
	\| `onnx/vision_encoder_fp16.onnx` (+`_data`) \| 1.10 GB \| SigLIP and the fine-tuned merger. Input `pixel_values float32[num_crops,3,14h,14w]`; output `image_features float32[num_crops,(h/4)(w/4),1024]` \|
	\| `onnx/decoder_model_merged_q8f16.onnx` (+`_data`) \| 0.83 GB \| Recommended language model: MatMulNBits 8-bit (block 32), fp16 activations and KV cache. Chunked Gated DeltaNet prefill. Outputs last-position `logits` and `decision_hidden` \|
	\| `onnx/embed_tokens_q8f16.onnx` (+`_data`) \| 0.28 GB \| Token embeddings, 8-bit \|
	\| `onnx/decoder_model_merged_q4f16.onnx` (+`_data`) \| 0.44 GB \| Lite: the same language model with 4-bit weights \|
	\| `onnx/embed_tokens_q4f16.onnx` (+`_data`) \| 0.15 GB \| Lite: token embeddings, 4-bit \|
	\| `runtime/minicpmv-runtime.mjs` \| \| Processor port (PIL bicubic, MiniCPM-V slicing, image placeholders), canvas M-RoPE positions, sessions, prefill/forward with cache reuse, greedy `generate` \|
	\| `runtime/minicpmv-decisions.mjs` \| \| WebBrain's decision protocol (drop-in next to `src/chrome/src/offscreen/webgpu-decisions.js`) \|
	\| `runtime/minicpmv-decision-probe-ft2.js`, `...-ft2-cs.js` \| \| Optional decision probes: standard, and conservative on false completions \|
	\| `runtime/webgpu-check.mjs`, `runtime/decision-check.mjs` \| \| Parity and end-to-end decision checks \|
	\| `validation/` \| \| Export report, training eval, decision, VL and routing results \|

	`config.json` names the graph sets under `webbrain_runtime`: `recommended` (q8) and `lite` (q4). The download is
	about 2.2 GB with the q8 graphs, or 1.7 GB with the lite graphs.

	- Lite (q4) graphs: Round 1's 4-bit build failed the quality bar (80/126 decisions, 25/48 false completions). This
	build's 4-bit graphs match q8 on d1 decisions: 120/126, 0/48 false completions. Text and visual generation were not
	validated on q4, so prefer q8 when the model generates.
	- Tested with: Transformers.js 4.2.0 (tokenizer), onnxruntime-web 1.27.0 (WebGPU EP), Chromium, an RTX 5090.
	- Low-end GPUs are not a target: on a Quadro T400 (4 GB), one decision request takes about 58 s, 45 s of it in
	the vision encoder.

	## Usage

	```javascript
	import * as ort from './ort.webgpu.mjs';
	import { AutoTokenizer } from './transformers.web.js';
	import { MINICPMV, MiniCPMVRuntime, preprocessImage } from './runtime/minicpmv-runtime.mjs';
	import { runMiniCPMVDecision } from './runtime/minicpmv-decisions.mjs';

	const tokenizer = await AutoTokenizer.from_pretrained(BUNDLE_URL);
	const tok = { encode: text => tokenizer.encode(text, { add_special_tokens: false }) };
	const rt = await MiniCPMVRuntime.create(ort, tok, { // sessions are created one at a time
	embed: `${BUNDLE_URL}/onnx/embed_tokens_q8f16.onnx`, // lite: embed_tokens_q4f16.onnx
	vision: `${BUNDLE_URL}/onnx/vision_encoder_fp16.onnx`,
	decoder: `${BUNDLE_URL}/onnx/decoder_model_merged_q8f16.onnx`, // lite: decoder_model_merged_q4f16.onnx
	});
	// decisions: images = [{ rgb: Uint8Array, width, height }], entries = [[id, d1-style question], ...]
	// Without a probe each option is scored by its raw Yes − No margin. To use a probe, import minicpmvProbeScores
	// from './runtime/minicpmv-decision-probe-ft2.js' and pass probe: { scores: minicpmvProbeScores }.
	const { answers } = await runMiniCPMVDecision(rt, { state, images, entries });
	// text or visual generation (greedy): a rendered chat prompt, one image placeholder per preprocessed image
	const prompt = `<\|im_start\|>user\n${MINICPMV.imagePad}\nDescribe this page.<\|im_end\|>\n`
	+ '<\|im_start\|>assistant\n<think>\n\n</think>\n\n';
	const { ids } = await rt.generate(prompt, [preprocessImage(rgb, width, height)], { maxNewTokens: 256 });
	const text = tokenizer.decode(ids, { skip_special_tokens: true });
	```

	## Validation

	These checks use the d1 balanced-V3 final test and WebBrain's own prompt helpers.

	- Processor: token ids and canvas positions are exact against the PyTorch processor. Pixels differ by at most 2/255
	on about 0.01% of values.
	- WebGPU q8 vs PyTorch (144 questions):
	- Argmax agreement is 98.6% (2 flips).
	- The median Yes − No margin difference is 0.38 (mean 0.76).
	- The median `decision_hidden` cosine is 0.999.
	- The tails are wider than round 1's (worst margin difference 1.06, minimum cosine 0.990). 12 of 144 questions differ by
	more than 2 logits. One two-option auxiliary question flips outright: margins +4.0 / −7.4 in WebGPU against
	−7.4 / +5.5 in PyTorch, hidden cosine 0.85.
	- End to end, the q8 build scores 120/126 without a probe, against 121/126 in PyTorch.
	- End-to-end decisions (RTX 5090, one request = a screenshot plus its questions):
	- q8 with the standard probe: 120/126 correct, 24/24 completions found, 0/48 false completions, 24/24 unknown
	recognized. The conservative probe and the q4 lite graphs score the same.
	- Time: 0.94 s per request on average (vision 0.28 s, prefill 0.12 s, questions 0.54 s). Model load takes 9.1 s
	(7.8 s for lite).
	- Generation smoke test (q8, greedy): a 31-token page description in 1.7 s.

	## Results

	### Page-state decisions: d1 balanced-V3 final test (126 categorical questions)

	Scored through WebBrain's decision protocol in the browser (WebGPU) on an RTX 5090. One request is a screenshot,
	its page state and its questions. The d1 final test split was never trained on.

	\| Model \| Correct \| Completions found \| False completions \| Unknown recognized \| Time per request \|
	\|---\|---:\|---:\|---:\|---:\|---:\|
	\| v5, WebGPU q8 + probe \| 120/126 \| 24/24 \| 0/48 \| 24/24 \| 0.94 s \|
	\| v5, WebGPU q4 (lite) + probe \| 120/126 \| 24/24 \| 0/48 \| 24/24 \| 0.94 s \|
	\| Round 1, WebGPU q8 + probe \| 103/126 \| 22/24 \| 11/48 \| 13/24 \| 0.85 s \|
	\| MiniCPM-V-4.7-1B untuned, WebGPU q8 + probe \| 104/126 \| 23/24 \| 8/48 \| 16/24 \| n/a \|
	\| Bonsai 2 2B Vision + probe \| 110/126 \| 21/24 \| 4/48 \| 20/24 \| 2.2 s \|
	\| d1-omni-600M \| 92/126 \| n/a \| 1/48 \| n/a \| 0.75 s \|

	- The probe is no longer needed. Without it, v5's raw Yes − No verify margins score 121/126 in PyTorch and 120/126
	in WebGPU q8. In PyTorch the standard and conservative probes (fit on d1 train, chosen on validation) also score
	121/126, and the generated option readout alone scores 120/126.
	- A caution on scope: d1 is a synthetic benchmark. v5 was fine-tuned on d1's train families, which are disjoint from
	the test families but follow the same design. Bonsai's backbone was not trained on d1, only its probe was.
	Validate v5 on real extension decisions before relying on these margins.

	### Visual fallback: WebBrain VL-3 frozen validation, same 1,022 rows

	Greedy generation, scored with the VL-3 pipeline's own scorer. The rows are every MolmoPoint grounding row except
	actionable inventory (first 20), the first 80 curated VL-3 rows and every 20th VL-2 row. LFM2.5-VL-450M (VL-3,
	step 3750) is WebBrain's current visual fallback.

	\| Metric \| v5 \| Round 1 \| LFM2.5-VL-450M \|
	\|---\|---:\|---:\|---:\|
	\| Schema-valid outputs \| 96.2% \| 95.8% \| 92.7% \|
	\| Point within 0.05 (510 rows) \| 65.5% \| 58.8% \| 45.9% \|
	\| Mean point distance \| 0.156 \| 0.194 \| 0.161 \|
	\| Grounded interaction: point within 0.05 (255) \| 57.3% \| 51.0% \| 29.8% \|
	\| Intent → point/box: point within 0.05 (255) \| 73.7% \| 66.7% \| 62.1% \|
	\| Box IoU \| 0.175 \| 0.113 \| 0.050 \|
	\| Point inside the target box \| 33.3% \| 23.3% \| 10.4% \|
	\| Six-section observation valid, VL-2 / VL-3 \| 98.7% / 72.5% \| 99.4% / 65.0% \| 97.5% / 38.8% \|
	\| Actionable inventory valid (first 20) \| 25% \| 30% \| 10% \|

	On intent → point/box the mean point distance is still worse than LFM's (0.190 vs 0.132; round 1 0.248), although more
	answers land within 0.05: a minority of far misses. Long inventory lists often do not terminate, for every model.

	### Text and tool calls: WebBrain `test/llm` routing benchmark (compact tier)

	Served from PyTorch BF16 with seed 3407 and thinking off. The table counts the 89 scenarios scored in the Compass
	Tiny v2.1 run: strict = `ideal`, loose = `ideal` + `ideal_name`.

	\| Model \| First-turn structured calls \| Strict exact action \| Loose tool family \| Discouraged actions \|
	\|---\|---:\|---:\|---:\|---:\|
	\| v5 (0.8B language model) \| 95/100 \| 9/89 \| 48/89 \| 3 \|
	\| Round 1 \| 97/100 \| 10/89 \| 38/89 \| 5 \|
	\| MiniCPM-V-4.7-1B, untuned \| 98/100 \| 3/89 \| 30/89 \| 2 \|
	\| Compass Tiny v2.1 (2.6B, WebGPU q4f16) \| 98/100 \| 15/89 \| 43/89 \| 3 \|

	## License

	The base model is Apache-2.0. This fine-tune was trained on data that includes noncommercial-licensed sources (for
	example the WebLINX-derived Compass subset), so the fine-tuned weights are released for **noncommercial research
	only** under the WebBrain noncommercial research restrictions. Private hosting does not expand those rights.
	Model output is never proof that a browser action succeeded. Keep WebBrain's schema validation, grounding checks and
	confirmations in front of any action.

	Training data includes WebChain v2 (`webagentlab/webchain`, WebChain Academic Research License v1.0): Fan, Sicheng, Rui Wan, Yifei Leng, Gaoning Liang, Li Ling, Yanyi Shang, and Dehan Kong. "WebChain: A Large-Scale Human-Annotated Dataset of Real-World Web Interaction Traces." arXiv:2603.05295, 2026. It also includes WebLINX (McGill-NLP, CC BY-NC-SA 4.0) and MolmoPoint-GUISyn (Ai2, ODC-BY).