webbrain-one commited on
Commit
03e3b28
·
verified ·
1 Parent(s): 3632834

Update README.md with enhanced intro, architecture diagram, and technical details

Browse files
Files changed (1) hide show
  1. README.md +78 -74
README.md CHANGED
@@ -13,95 +13,106 @@ tags:
13
  - tool-calling
14
  - experimental
15
  ---
 
16
  # WebBrain Compass Tiny v2.1
17
 
18
- **Private experimental q4f16/WebGPU release of the tested v24 candidate.**
19
- Based on WebBrain Compass Tiny v2, a fine-tune of **OpenBMB MiniCPM5-2B (~2.6B parameters)**.
20
- v2.1 is an **export/runtime revision**, not a new training run. The merged BF16 source remains
21
- `webbrain-one/webbrain-compass-tiny-v2` at `54bcab6731939137d0489ac01463c9404dff9da8`.
22
- The previous v23 release is separate and unchanged.
23
 
24
- ## What improved
25
 
26
- Only the ONNX graph's 211 `MatMulNBits` `accuracy_level` attributes changed from 4 to 2,
27
- disabling the dynamic INT8-activation path used in the pinned WebGPU runtime. Quantized
28
- weight shards, tokenizer, config and vendored runtime remain byte-identical to v23.
29
 
30
- | Check | v23 | v2.1 / v24 |
31
- |---|---:|---:|
32
- | Quantized PyTorch vs WebGPU numerical checks | 2/12 pass | **12/12 pass** |
33
- | Worst relative RMS logit difference; unchanged limit 2% | 6.5274% | **1.8334%** |
34
- | Minimum cosine; unchanged limit 0.999 | 0.998065500 | **0.999833960** |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
35
 
36
- Generation smoke (8 cases), bounded **16,384-total-token** context/cache checks, and a
37
- cold-browser restart with model downloads disabled passed. All three cold-cache native
38
- tool-call outputs were unchanged. Numerical tests cover three prompts, prefill plus
39
- three cached steps each, with identical token prefixes. They establish runtime parity
40
- in those probes—not BF16 equivalence, universal numerical accuracy, or a percentage
41
- loss in task accuracy. INT4 quality loss remains a separate issue; a Turkish smoke
42
- response remained awkward despite passing structural checks.
43
 
44
- ## Fixed next-response routing results
 
 
 
 
45
 
46
- | Metric | BF16 reference | v2.1 / v24 WebGPU |
 
 
 
 
 
 
47
  |---|---:|---:|
48
  | First-turn structured calls | 94/100 | **98/100** |
49
  | Strict exact action | 16/89 | **15/89** |
50
  | Loose tool-family match | 41/89 | **43/89** |
51
 
52
- This is **not end-to-end browser-agent task success or an Online-Mind2Web result**.
53
- One run: 100 first-turn cases, 89 scored scenarios and 11 predetermined skips;
54
- 194 raw responses including 5 smoke requests were retained locally. The 193 pinned
55
- input files match the BF16 reference, but precision, runtime and JS/Torch random
56
- generators differ. No output repair, model helper, fallback or result-based retry.
57
 
58
- **The full predeclared behavioral gate failed:** 3 explicitly discouraged actions
59
- versus a limit of 1. These were retrying an access-blocked URL, continuing a listing
60
- before reporting available results, and inventing a pagination URL. Other declared
61
- gates passed, with zero transport errors, zero native-decoder parser errors and zero
62
- length-limited responses. The owner authorized this separate release with these
63
- findings disclosed; it is not certified for unattended consequential actions.
64
 
65
- ## Exact package
 
 
66
 
67
  - GPTQ asymmetric INT4 weights, group size 32; FP16 scales/activations/KV cache.
68
- - Default graph: `onnx/model_q4f16.onnx`. **Both** external data files are required:
69
- `model_q4f16.onnx_data` and `model_q4f16.onnx_data_1` (1,868,992,512 bytes combined).
70
- - The inherited BF16 dtype and 131072 architecture field in `config.json` do not
71
- describe this quantized graph or establish a tested browser context limit.
72
- - Exact tested runtime under `runtime/`: Transformers.js **4.2.0**, ONNX Runtime Web
73
- **1.27.0**, WebBrain worker/parser and dependency licences. Tested with Chrome
74
- **150.0.7871.187**, NVIDIA **RTX 5090**. Other devices/runtimes are not verified here.
75
  - Generation-only, last-token logits; thinking disabled. No cloud or visual model.
76
- - Tests used a local model transport alias; this repository name is the release
77
- identifier. All 39 tested model/runtime files are mapped and hashed in provenance.
78
- - `validation/` contains results and compact audit summaries. Complete original
79
- local audit hashes are recorded; private training/calibration data, browser profiles,
80
- credentials and raw benchmark requests are not included.
81
 
82
  ## Integration
83
 
84
- 1. Download the **pinned commit returned with the release**, using an HF account with
85
- private-repository access:
86
 
87
- ```sh
88
  hf download webbrain-one/webbrain-compass-tiny-v2.1 --revision <PINNED_COMMIT> --local-dir ./compass-tiny-v2.1
89
  ```
90
 
91
- Keep `onnx/`, tokenizer/config files and both external shards together. Never embed
92
- a personal/write token in an extension, webpage, source file or log. Use a trusted
93
- authenticated asset proxy or a private local mirror pinned to that same commit.
94
- Anonymous HF downloads cannot access this private release.
95
 
96
- 2. In WebBrain, choose the custom model ID `webbrain-one/webbrain-compass-tiny-v2.1`,
97
- **device `webgpu`, dtype `q4f16`**, default filename stem `model`. Use the supplied
98
- runtime assets. The unchanged worker has no revision option: pin the asset proxy,
99
- or explicitly pass `revision` at pipeline initialization in your integration.
100
- The new model ID has a separate cache key; do not alias it onto v23's cached graph.
101
 
102
  Equivalent library initialization, with executable runtime assets served locally:
103
 
104
- ```js
105
  import { env, pipeline } from './runtime/vendor/transformers.web.js';
106
  import { parseMiniCpmToolCalls } from './integration/minicpm5-tool-parser.mjs';
107
 
@@ -133,21 +144,14 @@ findings disclosed; it is not certified for unattended consequential actions.
133
  }
134
  ```
135
 
136
- The native custom-model worker defaults to greedy generation and caps output at
137
- 256 tokens. The routing benchmark used its documented test hook (4096-token budget,
138
- temperature 0.15/0.3, seed 3407); do not claim those scores for the default UI settings.
139
- Budget input plus output within the tested 16K limit and start with short tasks.
140
 
141
- 3. **Parser integration is explicit, not silently changed.** The exact bundled parser
142
- under `runtime/` matches 162/194 recorded native-decoder responses. It can leave
143
- Python-style `False` and `['green', 'amber']` as strings. The optional, separately
144
- audited `integration/minicpm5-tool-parser.mjs` matches **194/194** recorded responses
145
- and passed its allowlist checks; it is not automatically wired into the worker.
146
- Pass the full tool schemas so string-valued JSON bodies remain strings. The helper
147
- does not execute literals, validate every schema constraint, or grant action approval.
148
- Preserve downstream validation, confirmations and raw-output/error logging.
149
 
150
  CPU-only helper checks: `node --test integration/parser.test.mjs`.
151
 
152
- This release remains subject to `ATTRIBUTIONS.md`, including the project's
153
- noncommercial-research restrictions. Private hosting does not expand usage rights.
 
 
 
 
13
  - tool-calling
14
  - experimental
15
  ---
16
+
17
  # WebBrain Compass Tiny v2.1
18
 
19
+ WebBrain Compass Tiny v2.1 is a compact language model optimized for low-latency decision making, structured tool use, and in-browser agentic execution inside the WebBrain runtime.
 
 
 
 
20
 
21
+ Fine-tuned from **OpenBMB MiniCPM5-2B (~2.6B parameters)** via `webbrain-one/webbrain-compass-tiny-v2`, v2.1 is an **export/runtime revision** delivering native in-browser execution through quantized asymmetric INT4 weights with FP16 activations (`q4f16`) on ONNX Runtime WebGPU.
22
 
23
+ It unifies three core WebBrain behavioral modes:
 
 
24
 
25
+ 1. **Ask & Clarify** — Answer directly or request missing information when execution is underspecified or unnecessary.
26
+ 2. **Direct Compact Tool Execution** — Select exact browser tools and generate grounded arguments for low-latency accessibility-tree actions.
27
+ 3. **Safe Escalation & Abstention** — Refuse, pause, or defer to higher execution tiers when actions lack adequate grounding or violate safety boundaries.
28
+
29
+ > **Note:** Compass Tiny v2.1 is an export/runtime optimization (v24 candidate), not a new pre-training or fine-tuning run. The merged BF16 base remains `webbrain-one/webbrain-compass-tiny-v2` (`54bcab6731939137d0489ac01463c9404dff9da8`). The previous v23 release is separate and unchanged.
30
+
31
+ ---
32
+
33
+ ## Role in WebBrain
34
+
35
+ ```text
36
+ User request
37
+ │
38
+ ▼
39
+ WebBrain observation & policy layer
40
+ │
41
+ ▼
42
+ WebBrain Compass Tiny v2.1 (WebGPU / q4f16)
43
+ ├─ Clarify or answer directly
44
+ ├─ Emit grounded browser tool call
45
+ └─ Abstain / safely escalate when execution is ungrounded
46
+ ```
47
+
48
+ The outer WebBrain runtime enforces tool-schema validation, evidence and parameter grounding, destination URL verification, browser-state freshness, and sandboxed security policies. Model output is never proof that an external action succeeded.
49
+
50
+ ---
51
+
52
+ ## Intended Capabilities
53
+
54
+ - **Browser Action Selection:** Grounded accessibility-tree interactions and structured function calling.
55
+ - **In-Browser WebGPU Inference:** Zero-cloud, client-side execution using Transformers.js and ONNX Runtime Web.
56
+ - **Unified Ask and Compact Modes:** Asking for missing information before acting, answering direct questions, or choosing tools.
57
+ - **Safe Refusal & Escalation:** Refusing unsupported actions or escalating when Compact execution lacks evidence, avoiding invented URLs or fabricated success states.
58
+
59
+ ---
60
+
61
+ ## What Improved in v2.1
62
 
63
+ In this revision, the ONNX graph's 211 `MatMulNBits` `accuracy_level` attributes were changed from 4 to 2, disabling the dynamic INT8-activation path used in the pinned WebGPU runtime. Quantized weight shards, tokenizer, config, and vendored runtime remain byte-identical to v23.
 
 
 
 
 
 
64
 
65
+ | Check | v23 | v2.1 / v24 | Status |
66
+ |---|---:|---:|:---:|
67
+ | Quantized PyTorch vs WebGPU numerical checks | 2/12 pass | **12/12 pass** | Passed |
68
+ | Worst relative RMS logit difference (limit: 2%) | 6.5274% | **1.8334%** | Passed |
69
+ | Minimum cosine similarity (limit: 0.999) | 0.998065500 | **0.999833960** | Passed |
70
 
71
+ Generation smoke (8 cases), bounded **16,384-total-token** context/cache checks, and a cold-browser restart with model downloads disabled passed. All three cold-cache native tool-call outputs were unchanged. Numerical tests cover three prompts, prefill plus three cached steps each, with identical token prefixes. They establish runtime parity in those probes—not BF16 equivalence, universal numerical accuracy, or a percentage loss in task accuracy. INT4 quality loss remains a separate issue; a Turkish smoke response remained awkward despite passing structural checks.
72
+
73
+ ---
74
+
75
+ ## Fixed Next-Response Routing Results
76
+
77
+ | Metric | BF16 Reference | v2.1 / v24 WebGPU |
78
  |---|---:|---:|
79
  | First-turn structured calls | 94/100 | **98/100** |
80
  | Strict exact action | 16/89 | **15/89** |
81
  | Loose tool-family match | 41/89 | **43/89** |
82
 
83
+ *Evaluated on 100 first-turn scenarios (89 scored, 11 predetermined skips) without output repair or result-based retries.*
 
 
 
 
84
 
85
+ > **Behavioral Gate Audit:** The full predeclared behavioral gate failed: 3 explicitly discouraged actions versus a limit of 1. These were retrying an access-blocked URL, continuing a listing before reporting available results, and inventing a pagination URL. Other declared gates passed, with zero transport errors, zero native-decoder parser errors, and zero length-limited responses. The owner authorized this separate release with these findings disclosed; it is not certified for unattended consequential actions.
 
 
 
 
 
86
 
87
+ ---
88
+
89
+ ## Exact Package
90
 
91
  - GPTQ asymmetric INT4 weights, group size 32; FP16 scales/activations/KV cache.
92
+ - Default graph: `onnx/model_q4f16.onnx`. **Both** external data files are required: `model_q4f16.onnx_data` and `model_q4f16.onnx_data_1` (1,868,992,512 bytes combined).
93
+ - The inherited BF16 dtype and 131072 architecture field in `config.json` do not describe this quantized graph or establish a tested browser context limit.
94
+ - Exact tested runtime under `runtime/`: Transformers.js **4.2.0**, ONNX Runtime Web **1.27.0**, WebBrain worker/parser and dependency licences. Tested with Chrome **150.0.7871.187**, NVIDIA **RTX 5090**. Other devices/runtimes are not verified here.
 
 
 
 
95
  - Generation-only, last-token logits; thinking disabled. No cloud or visual model.
96
+ - Tests used a local model transport alias; this repository name is the release identifier. All 39 tested model/runtime files are mapped and hashed in provenance.
97
+ - `validation/` contains results and compact audit summaries. Complete original local audit hashes are recorded; private training/calibration data, browser profiles, credentials, and raw benchmark requests are not included.
98
+
99
+ ---
 
100
 
101
  ## Integration
102
 
103
+ 1. Download the **pinned commit returned with the release**, using an HF account with private-repository access:
 
104
 
105
+ ```bash
106
  hf download webbrain-one/webbrain-compass-tiny-v2.1 --revision <PINNED_COMMIT> --local-dir ./compass-tiny-v2.1
107
  ```
108
 
109
+ Keep `onnx/`, tokenizer/config files, and both external shards together. Never embed a personal/write token in an extension, webpage, source file, or log. Use a trusted authenticated asset proxy or a private local mirror pinned to that same commit. Anonymous HF downloads cannot access this private release.
 
 
 
110
 
111
+ 2. In WebBrain, choose the custom model ID `webbrain-one/webbrain-compass-tiny-v2.1`, **device `webgpu`, dtype `q4f16`**, default filename stem `model`. Use the supplied runtime assets. The unchanged worker has no revision option: pin the asset proxy, or explicitly pass `revision` at pipeline initialization in your integration. The new model ID has a separate cache key; do not alias it onto v23's cached graph.
 
 
 
 
112
 
113
  Equivalent library initialization, with executable runtime assets served locally:
114
 
115
+ ```javascript
116
  import { env, pipeline } from './runtime/vendor/transformers.web.js';
117
  import { parseMiniCpmToolCalls } from './integration/minicpm5-tool-parser.mjs';
118
 
 
144
  }
145
  ```
146
 
147
+ The native custom-model worker defaults to greedy generation and caps output at 256 tokens. The routing benchmark used its documented test hook (4096-token budget, temperature 0.15/0.3, seed 3407); do not claim those scores for the default UI settings. Budget input plus output within the tested 16K limit and start with short tasks.
 
 
 
148
 
149
+ 3. **Parser integration is explicit, not silently changed.** The exact bundled parser under `runtime/` matches 162/194 recorded native-decoder responses. It can leave Python-style `False` and `['green', 'amber']` as strings. The optional, separately audited `integration/minicpm5-tool-parser.mjs` matches **194/194** recorded responses and passed its allowlist checks; it is not automatically wired into the worker. Pass the full tool schemas so string-valued JSON bodies remain strings. The helper does not execute literals, validate every schema constraint, or grant action approval. Preserve downstream validation, confirmations, and raw-output/error logging.
 
 
 
 
 
 
 
150
 
151
  CPU-only helper checks: `node --test integration/parser.test.mjs`.
152
 
153
+ ---
154
+
155
+ ## License & Attribution
156
+
157
+ This release remains subject to `ATTRIBUTIONS.md`, including the project's noncommercial-research restrictions. Private hosting does not expand usage rights.