senpaicq commited on
Commit
13428d4
·
verified ·
1 Parent(s): 119e1b4

Update model card: decision-model tags, Kev credit, related work

Browse files
Files changed (1) hide show
  1. README.md +16 -3
README.md CHANGED
@@ -12,10 +12,13 @@ tags:
12
  - pointer-network
13
  - selective-prediction
14
  - calibration
15
- - decision
 
 
 
 
16
  datasets:
17
  - osunlp/Multimodal-Mind2Web
18
- pipeline_tag: depth-estimation
19
  ---
20
 
21
  # Gemma 4 E2B + LoRA pointer head for web element selection (Multimodal-Mind2Web)
@@ -29,6 +32,10 @@ Given a task, the previous actions, a full-page screenshot and a list of candida
29
  - Speed: in the live form test below (short pages, one screenshot strip, about 33 candidates) a decision took about 0.2 s for the model plus 0.1 s for candidate extraction and screenshot on an A100. Inputs of Mind2Web size (several strips, up to 4600 tokens) are slower; we did not benchmark that separately.
30
  - Honest summary of what it is good and bad at is in [Summary of results](#summary-of-results) and [Limitations](#limitations-and-what-this-model-is-not).
31
 
 
 
 
 
32
  > **Status:** single training run, single seed. Confidence scores are **raw (not temperature-scaled)** for this checkpoint. Numbers are not directly comparable with published Mind2Web numbers (see [Evaluation protocol](#evaluation-protocol)).
33
 
34
  ## Summary of results
@@ -177,6 +184,12 @@ The scripts in `code/` reproduce the pipeline: `m2w_prep.py` (candidates, TF-IDF
177
  - The model was trained on [Multimodal-Mind2Web](https://huggingface.co/datasets/osunlp/Multimodal-Mind2Web). **We did not verify the dataset's license for this card; check the dataset page and terms before any commercial use.** No dataset content is included in this repository.
178
  - Screenshots and DOM snapshots in the dataset come from third-party websites.
179
 
 
 
 
 
 
 
180
  ## Citation
181
 
182
  If you use this model, please also cite the benchmark and the multimodal extension:
@@ -194,4 +207,4 @@ If you use this model, please also cite the benchmark and the multimodal extensi
194
  booktitle = {ICML},
195
  year = {2024}
196
  }
197
- ```
 
12
  - pointer-network
13
  - selective-prediction
14
  - calibration
15
+ - decision-making
16
+ - fast-decision-model
17
+ - system-1
18
+ - model-routing
19
+ - gui-grounding
20
  datasets:
21
  - osunlp/Multimodal-Mind2Web
 
22
  ---
23
 
24
  # Gemma 4 E2B + LoRA pointer head for web element selection (Multimodal-Mind2Web)
 
32
  - Speed: in the live form test below (short pages, one screenshot strip, about 33 candidates) a decision took about 0.2 s for the model plus 0.1 s for candidate extraction and screenshot on an A100. Inputs of Mind2Web size (several strips, up to 4600 tokens) are slower; we did not benchmark that separately.
33
  - Honest summary of what it is good and bad at is in [Summary of results](#summary-of-results) and [Limitations](#limitations-and-what-this-model-is-not).
34
 
35
+ **Role: a fast decision module ("System 1").** The model is meant as the quick, cheap first decision in a larger agent: it picks the next element or abstains with a confidence score, and low-confidence steps are escalated to a stronger model ("System 2", for example a large VLM) or a human. It is a decision/routing component, not a standalone agent.
36
+
37
+ **Approach: a calibrated decision head, "Kev-like".** Instead of generating the next action as text, the model makes a *decision over a closed set*: a query vector (from the task, history and screenshot) is scored against one key per candidate element plus a learned "none of these" key, and a softmax gives a probability for every option. This pointer-style, "Kev-like" decision head has three consequences we care about: (1) one forward pass picks the element, no token-by-token decoding; (2) the top-1 probability is a ready-made confidence score, and "none" is an explicit option, so the model can abstain; (3) because decisions and confidences are explicit numbers, they can be evaluated and calibrated with standard tools (ECE, Brier score, risk-coverage curves, temperature scaling) and used to route uncertain steps to a stronger model. In this release the decision quality and the confidence ranking are measured (see results); the confidences themselves are still raw and need temperature scaling on held-out data before being read as probabilities. The "Kev-like" design follows [Kev](https://github.com/jaredpalmer/kev) by Jared Palmer (open-source family of "System One" decision models: a small LoRA adapter plus a pointer head on a Qwen base, [weights on Hugging Face](https://huggingface.co/collections/jaredpalmer/kev), Apache-2.0), which in turn is described as Jev-like; see [Related work](#related-work-and-credit).
38
+
39
  > **Status:** single training run, single seed. Confidence scores are **raw (not temperature-scaled)** for this checkpoint. Numbers are not directly comparable with published Mind2Web numbers (see [Evaluation protocol](#evaluation-protocol)).
40
 
41
  ## Summary of results
 
184
  - The model was trained on [Multimodal-Mind2Web](https://huggingface.co/datasets/osunlp/Multimodal-Mind2Web). **We did not verify the dataset's license for this card; check the dataset page and terms before any commercial use.** No dataset content is included in this repository.
185
  - Screenshots and DOM snapshots in the dataset come from third-party websites.
186
 
187
+ ## Related work and credit
188
+
189
+ - **Kev** (Jared Palmer, 2026): open-source, Apache-2.0 family of small "System One" decision models that answer typed questions (choice, score, ...) with probabilities, built as a LoRA adapter plus a small pointer head on a Qwen base ([code](https://github.com/jaredpalmer/kev), [weights](https://huggingface.co/collections/jaredpalmer/kev)). The idea of a LoRA adapter with a pointer head over a closed set of options, and of reading calibrated probabilities from it, comes from this line of work. We are not affiliated with it, and this model is not a Kev checkpoint.
190
+ - **What is different here:** a different backbone (Gemma 4 E2B, a vision-language model), screenshots as input, a task-specific candidate set built from the DOM (hundreds of web elements per step), a learned "none" key and an operation head, LoRA on attention and MLP projections of layers 0-29 (rank 32), and an evaluation on Multimodal-Mind2Web with calibration and risk-coverage metrics. We did not compare against Kev checkpoints.
191
+ - **Benchmarks and data:** Mind2Web and Multimodal-Mind2Web (see Citation).
192
+
193
  ## Citation
194
 
195
  If you use this model, please also cite the benchmark and the multimodal extension:
 
207
  booktitle = {ICML},
208
  year = {2024}
209
  }
210
+ ```