Model card

.gitattributes

@@ -34,3 +34,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/bench_apps.png filter=lfs diff=lfs merge=lfs -text
+assets/bench_live.png filter=lfs diff=lfs merge=lfs -text
+assets/bench_live_synth.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,165 @@
+---
+license: apache-2.0
+license_link: https://huggingface.co/Qwen/Qwen3.5-4B/blob/main/LICENSE
+base_model:
+- Qwen/Qwen3.5-4B
+base_model_relation: finetune
+library_name: transformers
+pipeline_tag: image-text-to-text
+language:
+- en
+tags:
+- browser-agent
+- browser-use
+- computer-use
+- agent
+- tool-use
+- qwen3.5
+---
+
+# Nib 4B (preview)
+
+**Nib is a small browser agent.** It is [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) post-trained to work
+websites through the tools of common agent harnesses (Playwright MCP, browser-use, agent-browser-style CLIs, Chrome
+DevTools MCP, Aside, computer-use loops and coding agents that drive a browser). It reads page snapshots and
+screenshots, calls tools, and reports what it found.
+
+This is a **preview**: it is useful, but it still makes mistakes a bigger model would not, and it too often reports
+success on tasks it did not finish (see [Limitations](#limitations)). Check its answers before you rely on them.
+
+- **About twice as good as its base** on our browser tasks: 62% vs 29–30% on live web + synthetic tasks, and 80% vs 23%
+  on self-hosted web apps.
+- **Close to models many times its size**: on the combined set it is within noise of DeepSeek V4.1 Flash (high) and of
+  the untrained Qwen3.8-27B (tied with thinking off, 3–6 points behind with thinking on), and it leads both on
+  self-hosted apps.
+- **Fewer tokens**: 1,300–2,500 output tokens per task, 17–41% fewer than its base with thinking on.
+- **Four reasoning efforts**: thinking off, low, medium and xhigh, picked per request.
+
+## Benchmarks
+
+Every model ran the same tasks with the same harnesses (Aside, Playwright MCP and an agent-browser-style bash CLI).
+These are our own task sets, not public benchmarks:
+
+- **Live web (120 tasks):** real websites. Graded by one LLM judge with an evidence digest of the episode.
+- **Synthetic (108 tasks):** generated web apps with planted facts and forms. Graded by code.
+- **Self-hosted web apps (62 tasks):** fresh sandboxes of Gitea, WordPress, Odoo and similar apps. Graded by code against
+  the app's state.
+
+Charts plot success against mean output tokens per task; lines connect a model's reasoning efforts.
+
+![Live web + synthetic tasks](https://huggingface.co/ProCreations/nib-4b-preview/resolve/main/assets/bench_live_synth.png)
+
+![Live web tasks](https://huggingface.co/ProCreations/nib-4b-preview/resolve/main/assets/bench_live.png)
+
+![Self-hosted web-app tasks](https://huggingface.co/ProCreations/nib-4b-preview/resolve/main/assets/bench_apps.png)
+
+Success % on live web + synthetic / live web only / self-hosted apps, and mean output tokens per task (live + synthetic):
+
+| Model | Setting | Live + synthetic | Live web | Self-hosted apps | Tokens |
+|---|---|---:|---:|---:|---:|
+| **Nib 4B** | off | 56.6 | 52.5 | 80.6 | 1,293 |
+| **Nib 4B** | low | 61.8 | 53.3 | **82.3** | 1,783 |
+| **Nib 4B** | medium | 59.2 | 49.2 | 79.0 | 1,955 |
+| **Nib 4B** | xhigh | 62.3 | 55.8 | 80.6 | 2,537 |
+| Qwen3.5-4B (base, untrained) | off | 30.3 | 37.5 | 22.6 | 1,807 |
+| Qwen3.5-4B (base, untrained) | thinking on | 28.5 | 35.6 | 22.6 | 3,040 |
+| Qwen3.5-9B (untrained) | off | 41.2 | 49.2 | not run | 1,926 |
+| Qwen3.5-9B (untrained) | thinking on | 36.7 | 42.2 | not run | 2,983 |
+| Qwen3.8-27B (untrained) | off | 56.1 | 55.0 | 64.5 | 1,444 |
+| Qwen3.8-27B (untrained) | low | 65.8 | 65.0 | 75.8 | 2,727 |
+| Qwen3.8-27B (untrained) | medium | 64.9 | 66.7 | 64.5 | 2,188 |
+| Qwen3.8-27B (untrained) | xhigh | 65.8 | 66.7 | 77.4 | 3,771 |
+| Qwen3.8-Flash-Next | medium | 74.6 | 70.8 | not run | 3,179 |
+| Mistral Large 4 | high | 63.6 | 67.5 | 79.0 | 2,884 |
+| DeepSeek V4.1 Flash | high | 60.1 | 55.8 | 66.1 | 3,311 |
+| GLM 5.3 | high | 47.4 | 48.3 | 64.5 | 1,163 |
+
+- 95% confidence intervals are about ±6 points (live + synthetic), ±9 (live web) and ±10 (apps).
+- Qwen3.5-4B and Qwen3.5-9B have no effort levels (their chat template only turns thinking on or off); their
+  "thinking on" row averages three runs.
+- Nib trained on other tasks from the same synthetic generators and the same self-hosted apps (never the evaluation
+  tasks), so part of its lead there is in-distribution. Live web is the fairest comparison.
+- The other models ran through their providers' APIs at one setting, two days before Nib's runs. Live sites change;
+  synthetic and app tasks do not.
+- On a larger and harder live-web set (581 tasks across 8 harnesses), Nib scores about 45% against 56% for the
+  untrained Qwen3.8-27B, using about half the tokens.
+
+## Using Nib
+
+Nib uses the Qwen3.5 architecture (hybrid Gated DeltaNet + attention, with a vision encoder). We tested it with
+**SGLang 0.5.21**:
+
+```bash
+python -m sglang.launch_server \
+  --model-path ProCreations/nib-4b-preview --served-model-name nib-4b \
+  --context-length 131072 --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
+  --mamba-ssm-dtype bfloat16 --mamba-radix-cache-strategy extra_buffer_lazy --trust-remote-code
+```
+
+Pick the reasoning effort per request through the chat template:
+
+```python
+from openai import OpenAI
+client = OpenAI(base_url="http://127.0.0.1:30000/v1", api_key="none")
+resp = client.chat.completions.create(
+    model="nib-4b",
+    messages=[{"role": "user", "content": "Find the opening hours of the Louvre today."}],
+    tools=my_browser_tools,                     # your harness's tools
+    temperature=1.0, top_p=0.95,                # what all our evaluations used
+    extra_body={"chat_template_kwargs": {"enable_thinking": True, "reasoning_effort": "low"}},
+)
+```
+
+| Effort | `chat_template_kwargs` | When to use it |
+|---|---|---|
+| off | `{"enable_thinking": false}` | fastest and cheapest; fine for simple, well-specified tasks |
+| low | `{"enable_thinking": true, "reasoning_effort": "low"}` | best balance in our tests |
+| medium | `{"enable_thinking": true, "reasoning_effort": "medium"}` | |
+| xhigh | `{"enable_thinking": true}` (the default) | most careful, most tokens |
+
+Tips:
+- Nib works best inside an agent harness that gives it browser tools and page snapshots. It was trained with
+  Playwright MCP, browser-use, agent-browser-style CLIs, Chrome DevTools MCP, Aside, Qwen-style computer use, and coding
+  agents (Pi, OpenCode) driving a browser.
+- Screenshots help: it was trained with the images these harnesses send.
+- Most training episodes fit in 32k tokens. Longer sessions work, but quality on very long contexts is untested.
+- Not tested with vLLM, llama.cpp or MLX. They may work wherever Qwen3.5 is supported.
+
+## Quantized versions
+
+Every quant keeps the vision encoder, embeddings, norms, Gated DeltaNet gates and MTP weights in their original
+precision. Agreement is measured against these BF16 weights on 5,070 held-out assistant tokens from agent episodes.
+
+| Repo | Format | Size | Mean KL | Top-1 agreement | Browser tasks (581-task live set) |
+|---|---|---:|---:|---:|---|
+| this repo | BF16 | 9.3 GB | – | – | 45.3% |
+| [nib-4b-preview-FP8](https://huggingface.co/ProCreations/nib-4b-preview-FP8) | FP8 block 128×128 (SGLang) | 5.8 GB | 0.0028 | 98.9% | 42.5% (within run-to-run noise) |
+| [nib-4b-preview-NVFP4](https://huggingface.co/ProCreations/nib-4b-preview-NVFP4) | NVFP4 (SGLang, Blackwell) | 5.0 GB | 0.0147 | 97.0% | 45.6% |
+| [nib-4b-preview-EXL3](https://huggingface.co/ProCreations/nib-4b-preview-EXL3) | EXL3 3.05–6.05 bpw (exllamav3 / TabbyAPI) | 3.6–4.9 GB | 0.0257–0.0007 | 96.3–99.3% | not run |
+
+## How Nib was trained
+
+1. **Supervised fine-tuning** on browser-agent episodes that were verified as successful, from larger teacher models
+   (Qwen3.8-27B-class and Qwen3.8-Flash-Next), across about ten harnesses, with screenshots. It also included
+   demonstrations of reporting plainly when a task could not be done.
+2. **On-policy distillation**: Nib ran tasks itself, and larger teachers scored its own tokens, first a browser-tuned
+   Qwen3.8-27B, then Qwen3.8-Flash-Next. This preview is the third update of the final round, picked by a held-out live
+   evaluation. Later updates and an honesty-penalty round did not score higher.
+
+Rewards and filters came from code checkers (synthetic tasks, app databases) and an LLM judge (live websites). The
+evaluation tasks were held out from training. The training data is not released.
+
+## Limitations
+
+- **It often claims success when it failed.** In a held-out check, about 7 in 10 of its failed live-web tasks ended
+  with a confident (wrong) answer rather than a plain "I could not do this". Verify important results.
+- **Behind larger models on the open web.** On live websites it trails the untrained Qwen3.8-27B by 11–18 points
+  with thinking on, and by about 11 points on our harder 581-task live set.
+- **Small-model knowledge.** It can guess wrong URLs or facts that a bigger model would know, and should look things up
+  instead.
+- Training and evaluation tasks were almost all in English.
+- These are our own benchmarks; live-web results drift as sites change.
+
+## License
+
+Apache 2.0, the same as the base model [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B).