PEFT
Safetensors
English
web-agents
gui-agents
mind2web
lora
pointer-network
selective-prediction
calibration
decision-making
fast-decision-model
system-1
model-routing
gui-grounding
Instructions to use senpaicq/m2w-gemma4-e2b-pointer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use senpaicq/m2w-gemma4-e2b-pointer with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Update model card: decision-model tags, Kev credit, related work
Browse files
README.md
CHANGED
|
@@ -12,10 +12,13 @@ tags:
|
|
| 12 |
- pointer-network
|
| 13 |
- selective-prediction
|
| 14 |
- calibration
|
| 15 |
-
- decision
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
datasets:
|
| 17 |
- osunlp/Multimodal-Mind2Web
|
| 18 |
-
pipeline_tag: depth-estimation
|
| 19 |
---
|
| 20 |
|
| 21 |
# Gemma 4 E2B + LoRA pointer head for web element selection (Multimodal-Mind2Web)
|
|
@@ -29,6 +32,10 @@ Given a task, the previous actions, a full-page screenshot and a list of candida
|
|
| 29 |
- Speed: in the live form test below (short pages, one screenshot strip, about 33 candidates) a decision took about 0.2 s for the model plus 0.1 s for candidate extraction and screenshot on an A100. Inputs of Mind2Web size (several strips, up to 4600 tokens) are slower; we did not benchmark that separately.
|
| 30 |
- Honest summary of what it is good and bad at is in [Summary of results](#summary-of-results) and [Limitations](#limitations-and-what-this-model-is-not).
|
| 31 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
> **Status:** single training run, single seed. Confidence scores are **raw (not temperature-scaled)** for this checkpoint. Numbers are not directly comparable with published Mind2Web numbers (see [Evaluation protocol](#evaluation-protocol)).
|
| 33 |
|
| 34 |
## Summary of results
|
|
@@ -177,6 +184,12 @@ The scripts in `code/` reproduce the pipeline: `m2w_prep.py` (candidates, TF-IDF
|
|
| 177 |
- The model was trained on [Multimodal-Mind2Web](https://huggingface.co/datasets/osunlp/Multimodal-Mind2Web). **We did not verify the dataset's license for this card; check the dataset page and terms before any commercial use.** No dataset content is included in this repository.
|
| 178 |
- Screenshots and DOM snapshots in the dataset come from third-party websites.
|
| 179 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 180 |
## Citation
|
| 181 |
|
| 182 |
If you use this model, please also cite the benchmark and the multimodal extension:
|
|
@@ -194,4 +207,4 @@ If you use this model, please also cite the benchmark and the multimodal extensi
|
|
| 194 |
booktitle = {ICML},
|
| 195 |
year = {2024}
|
| 196 |
}
|
| 197 |
-
```
|
|
|
|
| 12 |
- pointer-network
|
| 13 |
- selective-prediction
|
| 14 |
- calibration
|
| 15 |
+
- decision-making
|
| 16 |
+
- fast-decision-model
|
| 17 |
+
- system-1
|
| 18 |
+
- model-routing
|
| 19 |
+
- gui-grounding
|
| 20 |
datasets:
|
| 21 |
- osunlp/Multimodal-Mind2Web
|
|
|
|
| 22 |
---
|
| 23 |
|
| 24 |
# Gemma 4 E2B + LoRA pointer head for web element selection (Multimodal-Mind2Web)
|
|
|
|
| 32 |
- Speed: in the live form test below (short pages, one screenshot strip, about 33 candidates) a decision took about 0.2 s for the model plus 0.1 s for candidate extraction and screenshot on an A100. Inputs of Mind2Web size (several strips, up to 4600 tokens) are slower; we did not benchmark that separately.
|
| 33 |
- Honest summary of what it is good and bad at is in [Summary of results](#summary-of-results) and [Limitations](#limitations-and-what-this-model-is-not).
|
| 34 |
|
| 35 |
+
**Role: a fast decision module ("System 1").** The model is meant as the quick, cheap first decision in a larger agent: it picks the next element or abstains with a confidence score, and low-confidence steps are escalated to a stronger model ("System 2", for example a large VLM) or a human. It is a decision/routing component, not a standalone agent.
|
| 36 |
+
|
| 37 |
+
**Approach: a calibrated decision head, "Kev-like".** Instead of generating the next action as text, the model makes a *decision over a closed set*: a query vector (from the task, history and screenshot) is scored against one key per candidate element plus a learned "none of these" key, and a softmax gives a probability for every option. This pointer-style, "Kev-like" decision head has three consequences we care about: (1) one forward pass picks the element, no token-by-token decoding; (2) the top-1 probability is a ready-made confidence score, and "none" is an explicit option, so the model can abstain; (3) because decisions and confidences are explicit numbers, they can be evaluated and calibrated with standard tools (ECE, Brier score, risk-coverage curves, temperature scaling) and used to route uncertain steps to a stronger model. In this release the decision quality and the confidence ranking are measured (see results); the confidences themselves are still raw and need temperature scaling on held-out data before being read as probabilities. The "Kev-like" design follows [Kev](https://github.com/jaredpalmer/kev) by Jared Palmer (open-source family of "System One" decision models: a small LoRA adapter plus a pointer head on a Qwen base, [weights on Hugging Face](https://huggingface.co/collections/jaredpalmer/kev), Apache-2.0), which in turn is described as Jev-like; see [Related work](#related-work-and-credit).
|
| 38 |
+
|
| 39 |
> **Status:** single training run, single seed. Confidence scores are **raw (not temperature-scaled)** for this checkpoint. Numbers are not directly comparable with published Mind2Web numbers (see [Evaluation protocol](#evaluation-protocol)).
|
| 40 |
|
| 41 |
## Summary of results
|
|
|
|
| 184 |
- The model was trained on [Multimodal-Mind2Web](https://huggingface.co/datasets/osunlp/Multimodal-Mind2Web). **We did not verify the dataset's license for this card; check the dataset page and terms before any commercial use.** No dataset content is included in this repository.
|
| 185 |
- Screenshots and DOM snapshots in the dataset come from third-party websites.
|
| 186 |
|
| 187 |
+
## Related work and credit
|
| 188 |
+
|
| 189 |
+
- **Kev** (Jared Palmer, 2026): open-source, Apache-2.0 family of small "System One" decision models that answer typed questions (choice, score, ...) with probabilities, built as a LoRA adapter plus a small pointer head on a Qwen base ([code](https://github.com/jaredpalmer/kev), [weights](https://huggingface.co/collections/jaredpalmer/kev)). The idea of a LoRA adapter with a pointer head over a closed set of options, and of reading calibrated probabilities from it, comes from this line of work. We are not affiliated with it, and this model is not a Kev checkpoint.
|
| 190 |
+
- **What is different here:** a different backbone (Gemma 4 E2B, a vision-language model), screenshots as input, a task-specific candidate set built from the DOM (hundreds of web elements per step), a learned "none" key and an operation head, LoRA on attention and MLP projections of layers 0-29 (rank 32), and an evaluation on Multimodal-Mind2Web with calibration and risk-coverage metrics. We did not compare against Kev checkpoints.
|
| 191 |
+
- **Benchmarks and data:** Mind2Web and Multimodal-Mind2Web (see Citation).
|
| 192 |
+
|
| 193 |
## Citation
|
| 194 |
|
| 195 |
If you use this model, please also cite the benchmark and the multimodal extension:
|
|
|
|
| 207 |
booktitle = {ICML},
|
| 208 |
year = {2024}
|
| 209 |
}
|
| 210 |
+
```
|